Developer Notes¶

Name: “super_hydro”¶

As of 9 September 2018, there are only 69 matches on Google for the search term “super_hydro” (with quotes) and most are squirreled away (email addresses, etc.) This seems like a safe choice for a project name. For comparison “SuperHydro” returned about 21k matches.

Environment variables will be prefixed with SUPER_HYDRO_ to minimize conflicts.
The main module is super_hydro.
Config files will use super_hydro.

Developer Install¶

We use Poetry to manage environments and dependencies, however, Conda can be useful, especially when trying to install GPU dependencies. For primary development work you should try to use Poetry, falling back to Conda if needed for speed or to get binary dependencies.

Install a virtual environment. This can either be pure pure python or with Conda:
- Pure Python:
```
poetry env use python3.9
```
- Conda: If you want to use Conda, first create an environment and activate it. We have an anaconda-project.yaml file which can be used to create appropriate Conda environments:
```
anaconda-project prepare --refresh
conda activate envs/super_hydro
```
  or
```
anaconda-project prepare --env-spec super_hydro_gpu   # If you have an NVIDIA GPU
conda activate envs/super_hydro_gpu
```
Use Poetry to install super_hydro with the tests and docs extras and optionally with the fftw or gpu extras if you have the FFTW libraries and/or NVIDIA Cuda libraries installed:
```
poetry install -E tests -E docs
poetry install -E tests -E docs -E fftw  # If you have the FFTW libraries
poetry install -E tests -E docs -E gpu   # If you have an NVIDIA GPU
```
Note: If you are using Conda, be sure to first conda activate the appropriate environment. I do not know a way to get poetry env use to do this by default.

Run a shell and use super_hydro:

poetry shell                     # If using pure poetry
conda activate envs/super_hydro  # If using Conda

Jupyter Kernel

If you run Jupyter outside of the virtual environment, then you should install the super_hydro kernel so it is available with your version of Jupyter. For example, the following will register a kernel named super_hydro in your personal Jupyter configuration folder.

jupyter kernelspec install $(poetry env info -p) --user --name=super_hydro

poetry run python3 -m ipykernel install --sys-prefix

If you need to add dependencies, use poetry add and then correct in pyproject.toml:

poetry add uncertainties
poetry add -D sphinx-autobuild
poetry update           # Update lock file
poetry install

You can then activate this with poetry shell for development work.

The development install process is specified in the Makefile, but needs a few tools installed:

Conda: We use this for installing binary packages (makes working with the GPU easier for example). I install a fairly minimal Miniconda base environment.

Poetry: The recommended approach for installing this is actually outside of the environment:

curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -

poetry2conda: This can then be installed with pip (optionally in your user folder with --user so it is available with multiple environments:
```
python3 -m pip --user install poetry2conda
```

. Currently this uses Poetry to specify dependencies etc. in pyproject.toml, then uses poetry2conda to generate an environment-*.yml file which can be used to create a Conda super_hydro environment:

make conda-env
make conda-env-gpu    # Add cupy

Conda¶

Some issues/questions related to the Conda super_hydro environment:

Where should we create this? Probably not in the user’s base installation (which might be write-only), so we should probably allow the --prefix ${CONDA_PREFIX} option to be set. (Maybe provide a configure script which allows this to be set?). To be able to activate this environment by name we need to run something like:
```
conda config --append envs_dirs ${CONDA_PREFIX}
```
Unfortunately, this update the global env_dirs path, which is the hard to remove (uninstall). We should probably just rely on setting CONDA_ENVS_PATH as part of the environment.

Environment Activation

In the remainder of the documentation, we shall assume that the appropriate environment is first activated with one of the following:

poetry shell                # If using poetry
conda activate super_hydro  # If using conda

Running the Server¶

There are several options for running the computation server.

Notebook Client¶

With the notebook interface, you can launch a server and view the results from a Jupyter notebook. Start a jupyter notebook server and visit Docs/Demonstrations/Contents%20(Start%20Here).ipynb:

conda activate super_hydro
jupyter notebook "Docs/Demonstrations/Contents (Start Here).ipynb"

Standalone Server¶

The computation server can also be started independently by running:

conda activate super_hydro
python bin/server

This will start a server listening on the port specified in super_hydro.conf (by default 27372). You can then connect to this port with any of the clients. By default, for security, the server only listens for connection from the same computer (localhost). To run a server on another machine, first setup an ssh tunnel to the server forwarding the port. For example:

<local > $ ssh <user@remote.machine.edu> -L 27372:27372
<remote> $ cd super_hydro
<remote> $ conda activate super_hydro # or poetry shell as needed
<remote> $ python bin/server

Flask Client¶

The simplest approach is to just run the Flask client:

poetry run python bin/client   # If using pure python
anaconda-project run client    # If using conda

This will start a Flask webserver listening on the specified port. Connecting a browser to this will allow you to select the various demonstrations from a menu, which will then launch a server, allowing you to run the simulation.

Details¶

These are my attempts to understand the structure of Kyle’s Flask code.

templates/

Documentation¶

The main API documentation is in Docs/sphinx-source and can be made by running:

anaconda-project run shell

# Now build the documentation.
cd Docs
make html

# Open the docs:
open _build/html/index.html

# Or run the following, which will continuously monitor your docs and
# update your page:
make auto

If you are working on documentation, you can have it auto-build when you save changes by running:

make dev-server

then visiting http://127.0.0.1:8000.

Sphinx/Jupyter Book¶

The documentation uses Sphinx, following the approach of Jupyter Book without explicitly using the Jupyter Book tools as discussed in the section Jupyter Book is a distribution of Sphinx.

Read the Docs¶

The documentation nd should be maintained so that it can be published at Read the Docs. See Getting Started with Sphinx for details.

Structure¶

The documentation is defined in the Docs/sphinx-source folder with the following files:

conf.py: Configuration of Sphinx and the documentation in general.
index.md: Master file. This defines the landing page and the overall structure through {toctree} statements. The file is written in the MyST format which is an extension of [Markdown] with special syntax to support ReStructure text features used by Sphinx.
README.md: Symlink to the top-level README file. This is currently directly included in index.md and so should be maintained so that it displays properly on GitHub and GitLab repos. (See this SO question for some suggestions on how to validate your page.)

We use the nbsphinx extension so that we can include Jupyter notebooks in the documentation. A couple of points about this:

Don’t include a Table of Contents in your notebooks: this does not look good.
Look closely at the headers: the structure of the documentation is dictated by these. Notebooks should all have a # Title followed by everything in appropriate subsection.
By default, empty notebooks will be executed so that the output gets rendered. This is good for example, but not so good for interactive sessions! Some options:
1. Store some output in the notebook: this will disable execution.
2. Added metadata {"nbsphinx": "hidden"} to the cell. (Use the View->Cell Toolbar->Edit Metadata option to get the Edit Metadata button.)
3. Hack in to nbconvert to control output. I have included code to add the TagRemovePreprocessor, but keep in mind that this runs after execution. This allows you to add remove_cell as a tag for example. (See this comment for details.)

Citations¶

Citations to academic references can be included in local.bib and included as described in the Jupyter{book} documentation:

{cite}`Saint-Jalm:2019` : results in [].

References¶

numpydoc format: We use the numpydoc format for documenting functions, classes, etc. so that they can be automatically included in documentation.
Sphinx

Super_Hydro

Developer Notes¶

Name: “super_hydro”¶

Developer Install¶

Conda¶

Running the Server¶

Notebook Client¶

Standalone Server¶

Flask Client¶

Details¶

Documentation¶

Sphinx/Jupyter Book¶

Read the Docs¶

Structure¶

Citations¶

References¶

Architecture¶

Asynchronous Code¶

Threading¶

Server¶

Publication Goals¶

Deployment¶

To Do¶

Issues¶

Dependencies¶

To Do¶

Lobby Display¶

Issues¶