diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 64b2eea849ba5ef8efca0990017b9da1a9d54880..a903aeb45fb4d354f19b3bc282604667c90353bd 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -32,6 +32,7 @@ repos: args: [--pipe, black] additional_dependencies: - black==23.1.0 # Matches black hook + exclude: "web/.*" - repo: https://github.com/codespell-project/codespell rev: v2.2.2 hooks: diff --git a/web/content/docs/devguide/advanced/wsl/index.md b/web/content/docs/devguide/advanced/wsl/index.md index 5c31f9de22eee1a3f644163ad0a67c89f88c9fc3..815de73eca4584c3d756673a553b35fbfa344c5f 100644 --- a/web/content/docs/devguide/advanced/wsl/index.md +++ b/web/content/docs/devguide/advanced/wsl/index.md @@ -13,7 +13,7 @@ The [Windows Subsystem for Linux](https://en.wikipedia.org/wiki/Windows_Subsyste ## Setup -- Install WSL2 by following [this guide](https://docs.microsoft.com/en-us/windows/wsl/install-win10). **Important:** Choose **Ubuntu 20.04 LTS** as the Linux distribution. Other distributions may not have a sufficient compiler. +- Install WSL2 by following [this guide](https://docs.microsoft.com/en-us/windows/wsl/install-win10). **Important:** Choose **Ubuntu {{< dataFile "versions.tested_version.ubuntu" >}} LTS** as the Linux distribution. Other distributions may not have a sufficient compiler. - Optional but recommended: Install the new [Windows Terminal](https://docs.microsoft.com/en-us/windows/terminal/get-started) - Follow the [developer guide for Linux]({{< ref "prerequisites.md" >}}) from now on. @@ -31,12 +31,12 @@ You can use the native Windows Visual Studio Code IDE (VS Code) for developing i ### Build and debug OGS -- Configure your project by following [this guide](https://vector-of-bool.github.io/docs/vscode-cmake-tools/getting_started.html#configuring-your-project). Select `GCC 9.3.0` as the [CMake kit](https://vector-of-bool.github.io/docs/vscode-cmake-tools/kits.html#kits). +- Configure your project by following [this guide](https://vector-of-bool.github.io/docs/vscode-cmake-tools/getting_started.html#configuring-your-project). Select `GCC {{< dataFile "versions.minimum_version.gcc" >}}` (or higher, depending on your installed version) as the [CMake kit](https://vector-of-bool.github.io/docs/vscode-cmake-tools/kits.html#kits). - Follow [this guide](https://vector-of-bool.github.io/docs/vscode-cmake-tools/debugging.html#selecting-a-launch-target) for debug a target. Select `ogs` as the debug target. ## Additional notes -The filesystem of the WSL is not inside your regular user directories. You can find it by running `explorer.exe .` inside the WSL shell. It should be something like `\\wsl$\Ubuntu-20.04\home\[username]\...`. +The filesystem of the WSL is not inside your regular user directories. You can find it by running `explorer.exe .` inside the WSL shell. It should be something like `\\wsl$\Ubuntu-{{< dataFile "versions.tested_version.ubuntu" >}}\home\[username]\...`. You can also run OGS inside the WSL with benchmarks located in your regular Windows directories. You regular filesystem can be accessed inside WSL with the `/mnt/c/`-prefix. E.g. to run an OGS benchmark: diff --git a/web/content/docs/devguide/development-workflows/jupyter-docs/index.md b/web/content/docs/devguide/development-workflows/jupyter-docs/index.md deleted file mode 100644 index dd3ffe1f49fd553c0493732ecea04184db278a46..0000000000000000000000000000000000000000 --- a/web/content/docs/devguide/development-workflows/jupyter-docs/index.md +++ /dev/null @@ -1,52 +0,0 @@ -+++ -date = "2022-06-28T15:28:13+01:00" -title = "Jupyter notebooks as documentation" -author = "Lars Bilke" -weight = 1025 - -[menu.devguide] -parent = "development-workflows" -+++ - -## The big picture - -Jupyter notebooks can be placed in `Tests/Data` along regular benchmarks. See [Notebook testing]({{< relref "test-data#notebook-testing" >}}) for details on how to setup ogs execution (especially input data path handling) inside the notebook. See also the [SimpleMechanics.ipynb](https://gitlab.opengeosys.org/ogs/ogs/-/blob/master/Tests/Data/Mechanics/Linear/SimpleMechanics.ipynb)-notebook as an example. - -## Add web meta information - -Similar to regular web documentation pages the notebook requires to have a frontmatter with some meta information as the first cell in the notebook: - -- Add a new cell and move it to the first position in the notebook -- Change the cell type to `raw`! -- Add meta information, conclude with a end-of-file marker (`<!--eofm-->`) e.g.: - - ```toml - title = "SimplePETSc" - date = "2021-11-09" - author = "Lars Bilke" - notebook = "Mechanics/Linear/SimpleMechanics.ipynb" - image = "optional_gallery_image.png" - web_subsection = "small-deformations" - <!--eofm--> - ``` - -Static images e.g. for the gallery image or to be used in Markdown cells have to located in either `images`- or `figures`-subdirectories beneath the Notebook file! Otherwise they will not show up in the web site. - -## General advice - -### Python cells - -- Do not use machine-specific or absolute paths! -- Assume that ogs and other tools are in the `PATH`. - -### Markdown cells - -- Do not use HTML inside Markdown blocks. -- For image captions add a title in quotation marks after the image path, e.g. - - ```md -  - ``` - -- Please use `images`- or `figures`-subdirectories! -- Please note that in merge request web previews images are not shown at all. diff --git a/web/content/docs/devguide/documentation/_index.md b/web/content/docs/devguide/documentation/_index.md new file mode 100644 index 0000000000000000000000000000000000000000..b9bef53bcf77c5980da332564a07af85cbfb8cfd --- /dev/null +++ b/web/content/docs/devguide/documentation/_index.md @@ -0,0 +1,4 @@ ++++ +title = "Documentation" +weight = 3 ++++ diff --git a/web/content/docs/devguide/documentation/jupyter-docs/binderhub-button.png b/web/content/docs/devguide/documentation/jupyter-docs/binderhub-button.png new file mode 100644 index 0000000000000000000000000000000000000000..191653032475f9e847e6e416fad1cd467a85c61b Binary files /dev/null and b/web/content/docs/devguide/documentation/jupyter-docs/binderhub-button.png differ diff --git a/web/content/docs/devguide/documentation/jupyter-docs/index.md b/web/content/docs/devguide/documentation/jupyter-docs/index.md new file mode 100644 index 0000000000000000000000000000000000000000..82ae9b0526689f9bc59d01c3f1d0433863131c73 --- /dev/null +++ b/web/content/docs/devguide/documentation/jupyter-docs/index.md @@ -0,0 +1,154 @@ ++++ +date = "2022-06-28T15:28:13+01:00" +title = "Jupyter notebooks as documentation and tests" +author = "Lars Bilke" +weight = 1025 + +[menu.devguide] +parent = "development-workflows" ++++ + +## The big picture + +[Jupyter notebooks](https://jupyter.org) are interactive computing environments where prose and code can be combined. In the OGS project notebooks can be used to define complex benchmark workflows and its results can be converted to be shown on the OGS web page (see an [example here](/docs/benchmarks/small-deformations/linear_disc_with_hole/)). + +## Create a new notebook + +Create a new notebook file in `Tests/Data` (either as regular `.ipynb`-files or as Markdown-based notebooks via [jupytext](https://jupytext.readthedocs.io/en/latest)). See examples: + +- [SimpleMechanics.ipynb](https://gitlab.opengeosys.org/ogs/ogs/-/blob/master/Tests/Data/Mechanics/Linear/SimpleMechanics.ipynb) (regular `.ipynb`-notebook) +- [Linear_Disc_with_hole.md](https://gitlab.opengeosys.org/ogs/ogs/-/blob/master/Tests/Data/Mechanics/Linear/DiscWithHole/Linear_Disc_with_hole.md) (jupytext-based notebook in Markdown) + +## Add web meta information + +If the notebook result should appear as a page on the web documentation a frontmatter with some meta information (similar to [regular web pages]({{< ref "web-docs.md" >}})) is required as the first cell in the notebook: + +- Add a new cell and move it to the first position in the notebook +- Change the cell type to `raw`! +- Add meta information, conclude with a end-of-file marker (`<!--eofm-->`) e.g.: + + ```md + title = "SimplePETSc" + date = "2021-11-09" + author = "Lars Bilke" + image = "optional_gallery_image.png" + web_subsection = "small-deformations" + <!--eofm--> + ``` + +<div class='note'> + +In jupytext-based notebooks you can add the frontmatter within the `<!-- #raw -->`- and `<!-- #endraw -->`-markers: + +```md +<!-- #raw --> +title = "Linear elasticity: disc with hole" +date = "2022-04-27" +author = "Linda Günther, Sophia Einspänner, Robert Habel, Christoph Lehmann and Thomas Nagel" +web_subsection = "small-deformations" +<!--eofm--> +<!-- #endraw --> +``` + +`web_subsection` needs to be set to a sub-folder in [web/content/docs/benchmarks](https://gitlab.opengeosys.org/ogs/ogs/-/tree/master/web/content/docs/benchmarks) (if not set the notebook page will not be linked from navigation bar / benchmark gallery on the web page). + +</div> + +## Notebook setup + +### Python cells + +- Do not use machine-specific or absolute paths! See the following example to set up notebook output paths: + + ```python + import os + + # On CI out_dir is set to the notebooks directory inside the build directory + # similar to regular benchmark tests. On local testing it will output to the + # notebooks source directory under a _out-subdirectory. + out_dir = os.environ.get('OGS_TESTRUNNER_OUT_DIR', '_out') + if not os.path.exists(out_dir): + os.makedirs(out_dir) + + # ... + # Run ogs; get input data from current directory; write to `out_dir` + ! ogs my_project.prj -o {out_dir} > {out_dir}/log.txt + + # Verify results; on failure assert with: + assert False + # or + raise SystemExit() + ``` + +- Do not write anything into the source directories. Use an `out_dir` as above. +- Assume that `ogs` and other tools are in the `PATH`. + +### Markdown cells + +- Do not use HTML inside Markdown blocks. +- Static images e.g. for the gallery image or to be used in Markdown cells have to be located in either `images`- or `figures`-subdirectories beneath the Notebook file! Otherwise they will not show up in the web site. + - For image captions add a title in quotation marks after the image path, e.g. + + ```md +  + ``` + + - Please note that in merge request web previews static images are not shown at all. + +## Execution environment + +In CI the notebooks are executed with all dependencies installed into a virtual environment in the build directory. The installed packages are defined in `Test/Data/requirements.txt`. The same setup can be enabled locally by setting the CMake option `OGS_USE_PIP=ON`. E.g. + +```bash +cmake --preset release -DOGS_USE_PIP=ON # Creates the virtual environment +source ../build/release/.venv/bin/activate # Activates the virtual environment +jupyter lab # Starts Jupyter Lab +``` + +## Register with CTest + +Add the notebook to CTest with e.g.: + +```cmake +if(NOT OGS_USE_PETSC) + NotebookTest(NOTEBOOKFILE Mechanics/Linear/SimpleMechanics.ipynb RUNTIME 10) +endif() +``` + +Then e.g. run all notebook test (`-R nb`) in parallel (`-j 4`) with: + +```bash +# cd into build directory +source .venv/bin/activate # Is created with OGS_USE_PIP=ON, see above note on environment. +ctest -R nb -j 4 --output-on-failure +``` + +## Advanced topics + +### jupytext usage + +If you use the [execution environment](#execution-environment) [jupytext](https://jupytext.readthedocs.io/en/latest) as already configured and its usage is transparent: + +- Double-click on a markdown file will open it as a Notebook +- Upon saving or executing a linked `.ipynb`-file is created in the background which stores outputs +- You still edit the Markdown file but don't notice the difference to regular notebooks in the Lab UI + +### Run a notebook in BinderHub + +On the web site or MR web previews on pages generated by a notebook there is a new banner: + + + +- Click the button to launch the notebook in BinderHub. +- The environment running in BinderHub is defined in [bilke/binder-ogs-requirements at GitHub](https://github.com/bilke/binder-ogs-requirements) +- When clicking the link it launches a Jupyter Lab instance pre-configured with ogs [via wheel](https://gitlab.opengeosys.org/ogs/ogs/-/blob/master/Tests/Data/requirements-ogs.txt#L2), clones the current ogs repo in it and opens the respective notebook ready to run. Please note that startup times may be several minutes and the computing resources are limited (1 core, 2GB RAM). For improved performance we would need to setup own infrastructure. Also currently only works for serial ogs configs. + +### PyVista notebooks on headless Linux systems + +PyVista (or VTK) requires a windowing environment for rendering. You can provide a virtual window with `xvfb-run`: + +```bash +sudo apt install libgl1-mesa-glx xvbf # install xvfb + +xvfb-run -a ctest [...] # provide a virtual window to the ctest-run +``` diff --git a/web/content/docs/devguide/development-workflows/web-docs/index.md b/web/content/docs/devguide/documentation/web-docs/index.md similarity index 100% rename from web/content/docs/devguide/development-workflows/web-docs/index.md rename to web/content/docs/devguide/documentation/web-docs/index.md diff --git a/web/content/docs/devguide/development-workflows/web-docs/preview-web.png b/web/content/docs/devguide/documentation/web-docs/preview-web.png similarity index 100% rename from web/content/docs/devguide/development-workflows/web-docs/preview-web.png rename to web/content/docs/devguide/documentation/web-docs/preview-web.png diff --git a/web/content/docs/devguide/packages/mfront/index.md b/web/content/docs/devguide/packages/mfront/index.md index b609669aef4b09272e4ae593b1ed4a540b5f684d..57c6bb104254b948b4384b122277af62038eaace 100644 --- a/web/content/docs/devguide/packages/mfront/index.md +++ b/web/content/docs/devguide/packages/mfront/index.md @@ -9,6 +9,12 @@ weight = 1043 parent = "packages" +++ +<div class='note'> + +Please note that MFront is not available on Windows! You can use [WSL]({{< ref "wsl.md" >}}) to compile ogs with MFront-support on Windows. + +</div> + MFront / TFEL support is enabled by the CMake-option `-DOGS_USE_MFRONT=ON` and can be installed - system-wide, diff --git a/web/content/docs/devguide/testing/test-data/index.md b/web/content/docs/devguide/testing/test-data/index.md index b115b206ea6ea66cbf31108d264a9a885503b767..ddf98eb1df0feafddc19894e23ac9fb17ffcdba8 100644 --- a/web/content/docs/devguide/testing/test-data/index.md +++ b/web/content/docs/devguide/testing/test-data/index.md @@ -61,63 +61,6 @@ In code `BaseLib::BuildInfo::data_path` (from `BuildInfo.h`) references the data For adding new data files simply commit the new files as usual. -## Notebook testing +## Notebook tests -Full Jupyter Notebooks based workflows can be tested too. Create the notebook in `Tests/Data`. Configure output directory and try to use it for all outputs: - -```python -import os - -# On CI out_dir is set to the notebooks directory inside the build directory -# similar to regular benchmark tests. On local testing it will output to the -# notebooks source directory under a _out-subdirectory. -out_dir = os.environ.get('OGS_TESTRUNNER_OUT_DIR', '_out') -if not os.path.exists(out_dir): - os.makedirs(out_dir) - -# ... -# Run ogs; get input data from current directory; write to `out_dir` -! ogs my_project.prj -o {out_dir} > {out_dir}/log.txt - -# Verify results; on failure assert with: -assert False -# or -raise SystemExit() -``` - -Add new Python dependencies to `Test/Data/requirements.txt`. - -### Register with CTest - -Add the benchmark to CTest with e.g.: - -```cmake -if(NOT OGS_USE_PETSC) - NotebookTest(NOTEBOOKFILE Mechanics/Linear/SimpleMechanics.ipynb RUNTIME 10) -endif() -``` - -By registering notebooks [are automatically added]({{< relref "jupyter-docs" >}}) to the benchmark documentation page. - -For local testing please note that you need to configure OGS with `OGS_USE_PIP=ON` (to automatically create a virtual environment in the build directory which is used by the notebook tests). - -Then e.g. run all notebook test (`-R nb`) in parallel with: - -```bash -source .venv/bin/activate # May need to be activated -ctest -R nb -j 4 --output-on-failure -``` - -<div class='note'> - -### PyVista notebooks on headless Linux systems - -PyVista (or VTK) requires a windowing environment for rendering. You can provide a virtual window with `xvfb-run`: - -```bash -sudo apt install libgl1-mesa-glx xvbf # install xvfb - -xvfb-run -a ctest [...] # provide a virtual window to the ctest-run -``` - -</div> +For notebook-based tests [see its dedicated page]({{< ref "jupyter-docs.md" >}}).