Executable testing and test data integration
Test Wrapper Introductiion
As proposed by Dima an executable test can be run with several wrappers, e.g. with valgrind
or memcheck
. Each wrapper run can then be validated, e.g. with file-diff
s or -grep
s. This can be configured in CMake:
INCLUDE(test/AddTest)
AddTest(SimpleGroundWaterFlow ${ExternalData_SOURCE_ROOT}/GroundwaterFlow_Quad4/288Cells case TIME DIFF)
AddTest(SimpleGroundWaterFlow ${ExternalData_SOURCE_ROOT}/GroundwaterFlow_Quad4/288Cells case MEMCHECK)
AddTest(SimpleGroundWaterFlow ${ExternalData_SOURCE_ROOT}/GroundwaterFlow_Quad4/288Cells case CALLGRIND)
Where arguments are:
- executable
- test data directory
- test case name
- wrapper name
- checker (validator) name, optional
Tests are then run with
make ctest
If the checker has some errors they are displayed.
The functionality is very flexible and more wrappers and checker can be added later on. e.g. for running some statistics on output files and comparing them with statistics from reference files.
The testing functionality has not been fully adapted to the test data integration because of a missing OGS simulator executable.
Test Data Introduction
For managing test data files Dima and I evaluated some options but in my the functionality provided by CMake (in the module ExternalData) is really nice:
- In the source tree there placeholder files representing the test data (see https://github.com/bilke/ogs/commit/62235db731dafdefb1b603168cb170f919742921). The files have the same name but an additional
.md5
at the end. The content of a file consists of a MD5-hash of the original file. - The files can then be referenced in CMake when tests are added (see
scripts/cmake/test/Data.cmake
in https://github.com/bilke/ogs/commit/4e470970c68ddeba8c7a6238ea265510fb70facb, a real example will be added once the OGS simulator executable is in the official repository) - The original data files are automatically retrieved by CMake when tests are run
- The files are renamed based on their MD5 hash
- The files are automatically downloaded from a web server (at the moment this is
http://www.opengeosys.org/images/dev/MD5/<hash>
) - Retrieved files are stored in
ogs-source-dir/../ogs6-data/MD5/<hash>
This approach has some really nice benefits:
- Large files e.g. a meshes which is used by several tests exists in the source tree several times but gets downloaded only once and is also stored locally only once (in the build tree several symlinks get created which point to the actual file)
- The same is true for files which differ in their filename but not in their content
- Everything is nicely integrated into CMake
- Adding new test files is relatively easy (see below)
- Test data can be shared between several build configurations and source trees
Adding new test files
- Just put the new files into the source tree under
Tests/Data/
- Reference these files in some tests (in CMake)
- Run CMake
- This automatically generates the
*.md5
reference files
- This automatically generates the
- Build the
move-data
-target, e.g.make move-data
- This moves the original files into your
ogs6-data
-directory and renames them based on their hash - Upload these files to the webserver (how to do that has to be discussed)
- This moves the original files into your
- Add the new (
*.md5
reference) files inTests/Data/
to the git repository
Requirements
- CMake 2.8.11 (on older versions testing is disabled)
Once merged I will move this documentation over to http://docs.opengeosys.org/docs/devguide/testing/test-data.