[CI] Introduce maintenance tests to prevent regular developers from encountering pipeline errors unrelated to their merge requests
Separate regression and maintenance testing. Introduce maintenance tests. Reworked regression tests now prevent regular developers from encountering pipeline errors unrelated to their merge requests Maintenance test signal as early as possible to maintainers if OGS is "future-proof".
From:
Regression test | |
---|---|
OGS | latest |
Environment | latest |
Trigger | Merge-request |
to:
Regression test | Maintenance test | |
---|---|---|
OGS | latest | stable (tip of master) |
Environment | stable (pinned) | latest |
Trigger | Merge-request | scheduled |
Role | Developer | Maintainer |
latest indicates the "object under test"
Environment may be:
- pip (from PyPI)
- CPM (FindPackage, GIT_TAG / VERSION) - not in this MR
- Compiler version - not in this MR
- ...
When to update a "stable" an environment:
- MUST: Change of underlying stack (e.g. Relevant update of envinf)
- MUST: An MR with a change in the request environment (e.g. a new lib or a new version of an already used lib is needed)
- OPTIONAL: On any other occasion (e.g. OGS release)
General idea was tested for several months in OGSTools successfully:
After a pyvista update to 0.46 the maintenance tests signaled succesfully a problem. But regulare developers are not disturbed (2 later pipelines are running)
Implemented with uv
. New pyproject.toml in Tests/Data
specifying notebook test dependencies (formerly defined in requirements*.txt
-files, see Tests/Data/README.md
for usage.
Typical Workflow:
- Trigger: Maintenance test fails
-
- Check latest updates of 3rd party packages and their influence
-
- MR: Quick fix: Restrict versions in (open) dependencies, add/open ToDo
-
- Maintenance test succeeds (this allows other maintenance issues to be found)
-
- Investigation and fix of own software / report to 3rd party
-
- MR: Fix and reopen dependency