Overview¶

Important

Before executing any of the commands shown here, you should make sure that you activate the environment into which you installed super_hydro (see {ref}`installing). For example, you might need to do one of the following:

Conda:

conda activate super_hydro  # or super_hydro_gpu

Anaconda Project:

anaconda-project run shell  # or shell_gpu

Quick Start¶

Once the appropriate environment is activated, the fastest way to see what is here is to run the Flask client, then open your browse:

python3 bin/client

This will spawn a web-server at http://localhost:27372 (or a similar port, depending on your configuration). This will open the Flask client, which will display a list of models for you to explore.

Alternatively, you can run demons in a Jupyter notebook from the Demo folder:

jupyter notebook "Docs/Demonstrations/Contents (Start Here).md"

Components¶

We use Poetry to manage environments and dependencies, however, Conda can be useful, especially when trying to install GPU dependencies. For primary development work you should try to use Poetry, falling back to Conda if needed for speed or to get binary dependencies.

Install a virtual environment. This can either be pure pure python or with Conda:
- Pure Python:
```
poetry env use python3.9
```
- Conda: If you want to use Conda, first create an environment and activate it. We have an anaconda-project.yaml file which can be used to create appropriate Conda environments:
```
anaconda-project prepare --refresh
conda activate envs/super_hydro
```
  or
```
anaconda-project prepare --env-spec super_hydro_gpu   # If you have an NVIDIA GPU
conda activate envs/super_hydro_gpu
```
Use Poetry to install super_hydro with the tests and docs extras and optionally with the fftw or gpu extras if you have the FFTW libraries and/or NVIDIA Cuda libraries installed:
```
poetry install -E tests -E docs
poetry install -E tests -E docs -E fftw  # If you have the FFTW libraries
poetry install -E tests -E docs -E gpu   # If you have an NVIDIA GPU
```
Note: If you are using Conda, be sure to first conda activate the appropriate environment. I do not know a way to get poetry env use to do this by default.

Run a shell and use super_hydro:

poetry shell                     # If using pure poetry
conda activate envs/super_hydro  # If using Conda

Jupyter Kernel

If you run Jupyter outside of the virtual environment, then you should install the super_hydro kernel so it is available with your version of Jupyter. For example, the following will register a kernel named super_hydro in your personal Jupyter configuration folder.

jupyter kernelspec install $(poetry env info -p) --user --name=super_hydro

poetry run python3 -m ipykernel install --sys-prefix

If you need to add dependencies, use poetry add and then correct in pyproject.toml:

poetry add uncertainties
poetry add -D sphinx-autobuild
poetry update           # Update lock file
poetry install

You can then activate this with poetry shell for development work.

The development install process is specified in the Makefile, but needs a few tools installed:

Conda: We use this for installing binary packages (makes working with the GPU easier for example). I install a fairly minimal Miniconda base environment.

Poetry: The recommended approach for installing this is actually outside of the environment:

curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -

poetry2conda: This can then be installed with pip (optionally in your user folder with --user so it is available with multiple environments:
```
python3 -m pip --user install poetry2conda
```

. Currently this uses Poetry to specify dependencies etc. in pyproject.toml, then uses poetry2conda to generate an environment-*.yml file which can be used to create a Conda super_hydro environment:

make conda-env
make conda-env-gpu    # Add cupy

Conda¶

Some issues/questions related to the Conda super_hydro environment:

Where should we create this? Probably not in the user’s base installation (which might be write-only), so we should probably allow the --prefix ${CONDA_PREFIX} option to be set. (Maybe provide a configure script which allows this to be set?). To be able to activate this environment by name we need to run something like:
```
conda config --append envs_dirs ${CONDA_PREFIX}
```
Unfortunately, this update the global env_dirs path, which is the hard to remove (uninstall). We should probably just rely on setting CONDA_ENVS_PATH as part of the environment.

Environment Activation

In the remainder of the documentation, we shall assume that the appropriate environment is first activated with one of the following:

poetry shell                # If using poetry
conda activate super_hydro  # If using conda

Running the Server¶

There are several options for running the computation server.

Notebook Client¶

With the notebook interface, you can launch a server and view the results from a Jupyter notebook. Start a jupyter notebook server and visit Docs/Demonstrations/Contents%20(Start%20Here).ipynb:

conda activate super_hydro
jupyter notebook "Docs/Demonstrations/Contents (Start Here).ipynb"

Standalone Server¶

The computation server can also be started independently by running:

conda activate super_hydro
python bin/server

This will start a server listening on the port specified in super_hydro.conf (by default 27372). You can then connect to this port with any of the clients. By default, for security, the server only listens for connection from the same computer (localhost). To run a server on another machine, first setup an ssh tunnel to the server forwarding the port. For example:

<local > $ ssh <user@remote.machine.edu> -L 27372:27372
<remote> $ cd super_hydro
<remote> $ conda activate super_hydro # or poetry shell as needed
<remote> $ python bin/server

Flask Client¶

The simplest approach is to just run the Flask client:

poetry run python bin/client   # If using pure python
anaconda-project run client    # If using conda

This will start a Flask webserver listening on the specified port. Connecting a browser to this will allow you to select the various demonstrations from a menu, which will then launch a server, allowing you to run the simulation.

Details¶

These are my attempts to understand the structure of Kyle’s Flask code.

templates/

Conda Environment¶

Documentation¶

The main API documentation is in Docs/sphinx-source and can be made by running:

# If you need to, activate an environment:
poetry env use 3.8
poetry shell

# Now build the documentation.
cd Docs
make html

open _build/html/index.html

If you are working on documentation, you can have it auto-build when you save changes by running:

make auto

then visiting http://127.0.0.1:8000.

Sphinx/Jupyter Book¶

The documentation uses [Sphinx], following the approach of Jupyter Book without explicitly using the Jupyter Book tools as discussed in the section Jupyter Book is a distribution of Sphinx.

Read the Docs¶

The documentation nd should be maintained so that it can be published at Read the Docs. See Getting Started with Sphinx for details.

Structure¶

The documentation is defined in the Docs/sphinx-source folder with the following files:

conf.py: Configuration of Sphinx and the documentation in general.
index.md: Master file. This defines the landing page and the overall structure through {toctree} statements. The file is written in the MyST format which is an extension of [Markdown] with special syntax to support ReStructure text features used by [Sphinx].
README.md: Symlink to the top-level README file. This is currently directly included in index.md and so should be maintained so that it displays properly on GitHub and GitLab repos. (See this SO question for some suggestions on how to validate your page.)

We use the nbsphinx extension so that we can include Jupyter notebooks in the documentation.

Citations¶

Citations to academic references can be included in master.bib and included as described in the Jupyter{book} documentation:

{cite}`Saint-Jalm:2019` : results in []

References¶

numpydoc format: We use the numpydoc format for documenting functions, classes, etc. so that they can be automatically included in documentation.
Sphinx

Super_Hydro

Overview¶

Quick Start¶

Components¶

Conda¶

Running the Server¶

Notebook Client¶

Standalone Server¶

Flask Client¶

Details¶

Conda Environment¶

Documentation¶

Sphinx/Jupyter Book¶

Read the Docs¶

Structure¶

Citations¶

References¶

Architecture¶

Asynchronous Code¶

Threading¶

Server¶

Publication Goals¶

Deployment¶

To Do¶

Issues¶

Dependencies¶

To Do¶

Sat 9 Oct 2021¶

Thu 16 July 2020¶

Fri 25 Sept 2020¶

Tue 08 Dec 2020¶

Lobby Display¶

Issues¶