Skip to content

Executable testing and test data integration

Lars Bilke requested to merge github/fork/bilke/external-data into master

Test Wrapper Introductiion

As proposed by Dima an executable test can be run with several wrappers, e.g. with valgrind or memcheck. Each wrapper run can then be validated, e.g. with file-diffs or -greps. This can be configured in CMake:

INCLUDE(test/AddTest)
AddTest(SimpleGroundWaterFlow ${ExternalData_SOURCE_ROOT}/GroundwaterFlow_Quad4/288Cells case TIME DIFF)
AddTest(SimpleGroundWaterFlow ${ExternalData_SOURCE_ROOT}/GroundwaterFlow_Quad4/288Cells case MEMCHECK)
AddTest(SimpleGroundWaterFlow ${ExternalData_SOURCE_ROOT}/GroundwaterFlow_Quad4/288Cells case CALLGRIND)

Where arguments are:

  1. executable
  2. test data directory
  3. test case name
  4. wrapper name
  5. checker (validator) name, optional

Tests are then run with

make ctest

If the checker has some errors they are displayed.

The functionality is very flexible and more wrappers and checker can be added later on. e.g. for running some statistics on output files and comparing them with statistics from reference files.

The testing functionality has not been fully adapted to the test data integration because of a missing OGS simulator executable.

Test Data Introduction

For managing test data files Dima and I evaluated some options but in my the functionality provided by CMake (in the module ExternalData) is really nice:

  • In the source tree there placeholder files representing the test data (see https://github.com/bilke/ogs/commit/62235db731dafdefb1b603168cb170f919742921). The files have the same name but an additional .md5 at the end. The content of a file consists of a MD5-hash of the original file.
  • The files can then be referenced in CMake when tests are added (see scripts/cmake/test/Data.cmake in https://github.com/bilke/ogs/commit/4e470970c68ddeba8c7a6238ea265510fb70facb, a real example will be added once the OGS simulator executable is in the official repository)
  • The original data files are automatically retrieved by CMake when tests are run
    • The files are renamed based on their MD5 hash
    • The files are automatically downloaded from a web server (at the moment this is http://www.opengeosys.org/images/dev/MD5/<hash>)
    • Retrieved files are stored in ogs-source-dir/../ogs6-data/MD5/<hash>

This approach has some really nice benefits:

  • Large files e.g. a meshes which is used by several tests exists in the source tree several times but gets downloaded only once and is also stored locally only once (in the build tree several symlinks get created which point to the actual file)
  • The same is true for files which differ in their filename but not in their content
  • Everything is nicely integrated into CMake
  • Adding new test files is relatively easy (see below)
  • Test data can be shared between several build configurations and source trees

Adding new test files

  • Just put the new files into the source tree under Tests/Data/
  • Reference these files in some tests (in CMake)
  • Run CMake
    • This automatically generates the *.md5 reference files
  • Build the move-data-target, e.g. make move-data
    • This moves the original files into your ogs6-data-directory and renames them based on their hash
    • Upload these files to the webserver (how to do that has to be discussed)
  • Add the new (*.md5 reference) files in Tests/Data/ to the git repository

Requirements

  • CMake 2.8.11 (on older versions testing is disabled)

Once merged I will move this documentation over to http://docs.opengeosys.org/docs/devguide/testing/test-data.

Merge request reports