Skip to content

[MeL/IO] Fast time access and better usage of OGS-HDF5 output file

Tobias Meisel requested to merge TobiasMeisel/ogs:unlimited_time_dimension into master

HDF5 file structure now allows easy access (also with time dimension) to data with h5py and allows parallel post-processing

Example post-processing with python

Install H5Py e.g. with pip:
pip install h5py

Example file: cube_1e3.h5

import h5py
def demo():
    with h5py.File('cube_1e3.h5') as myfile:
        # all data will always be in 'data' section
        data = myfile['data']
        # inspect data
        data.keys()
        # example: <KeysViewHDF5 ['D1_left_front_N1_right', 'Linear_1_to_minus1', 'MaterialIDs', 'geometry', 'pressure', 'topology', 'v']>

        # choose any key , here in example 'pressure', it is only a reference (no copy of data into memory)
        pressure = data['pressure']

        # get dimensions
        pressure.shape
        # example: (2, 1895)
        # first dimension is time, here with length 2

        # load data into memory (only selected data is loaded into memory)
        # example: load all timesteps and choose all data points 100:200
        p = pressure [:,100:200]

        # p can now be processed e.g. with numpy
        # import numpy
        # y = numpy.array(p)

For more details on selecting data there is

File structures

Structure of HDF5 file

Output of all time steps is written into a single hdf5 - file. All data is written into group "data".

Example (and cut example) from h5dump -H cube_1e3.h5 cube_1e3.h5

HDF5 "cube_1e3.h5" {
GROUP "/" {
   GROUP "data" {
      DATASET "MaterialIDs" {
         DATATYPE  H5T_STD_I32LE
         DATASPACE  SIMPLE { ( 1, 1233 ) / ( H5S_UNLIMITED, 1233 ) }
      }
      DATASET "geometry" {
         DATATYPE  H5T_IEEE_F64LE
         DATASPACE  SIMPLE { ( 1, 1895, 3 ) / ( H5S_UNLIMITED, 1895, 3 ) }
      }
...

Static property

DATASPACE SIMPLE { ( 1, 1233 ) All properties have an additional dimension, representing the time steps. It is first dimension (for performance reasons) Subsequent dimension represent the data dimensions, here 1D with length 1233 for MaterialIDs. If the length of time steps is 1 - it is a constant property/attribute. Geometry and Topology are treated as constant properties/attributes.

Dynamic property

      
      DATASET "pressure" {
         DATATYPE  H5T_IEEE_F64LE
         DATASPACE  SIMPLE { ( 2, 1895 ) / ( H5S_UNLIMITED, 1895 ) }
      }
      
      DATASET "v" {
         DATATYPE  H5T_IEEE_F64LE
         DATASPACE  SIMPLE { ( 2, 1895, 3 ) / ( H5S_UNLIMITED, 1895, 3 ) }
      }
   }
}
}

pressure and v are dynamic properties. Here the simulation run for 2 time steps. The length of first dimension is equal to the number of time steps.

Structure of XDMF file

Important!
Never do any post-processing based on XDMF-format! OGS-XDMF is only for visualization in Paraview. Load .hdf file and use h5py instead!

<?xml version="1.0" encoding="utf-8"?>
<Xdmf xmlns:xi="http://www.w3.org/2001/XInclude" Version="3.0">
  <Domain>
    <Information Name="OGS_VERSION" Value="6.4.0-269-g0177d3b3a.dirty"/>
    <Grid CollectionType="Temporal" GridType="Collection" Name="Collection">
      <Grid Name="Grid">
        <Time Value="0"/>
        <Geometry Origin="" Type="XYZ">
          <DataItem DataType="Float" Dimensions="1895 3" Format="HDF" Precision="8">cube_1e3.h5:data/geometry</DataItem>
        </Geometry>
        <Topology Dimensions="11097" Type="Mixed">
          <DataItem DataType="Int" Dimensions="11097" Format="HDF" Precision="4">cube_1e3.h5:data/topology</DataItem>
        </Topology>
        <Attribute Center="Cell" ElementCell="" ElementDegree="0" ElementFamily="" ItemType="" Name="MaterialIDs" Type="None">
          <DataItem DataType="Int" Dimensions="1 1233" Format="HDF" Precision="4">cube_1e3.h5:data/MaterialIDs</DataItem>
        </Attribute>
        <Attribute Center="Node" ElementCell="" ElementDegree="0" ElementFamily="" ItemType="" Name="pressure" Type="None">
          <DataItem DataType="Float" Dimensions="1 1895" Format="HDF" Precision="8">cube_1e3.h5:data/pressure</DataItem>
        </Attribute>
        <Attribute Center="Node" ElementCell="" ElementDegree="0" ElementFamily="" ItemType="" Name="v" Type="None">
          <DataItem DataType="Float" Dimensions="1 1895 3" Format="HDF" Precision="8">cube_1e3.h5:data/v</DataItem>
        </Attribute>
      </Grid>
      <Grid Name="Grid">
        <Time Value="0.1"/>
        <xi:include xpointer="element(/1/1/2/1/4)"/>
        <xi:include xpointer="element(/1/1/2/1/5)"/>
        <xi:include xpointer="element(/1/1/2/1/6)"/>
        <Attribute Center="Node" ElementCell="" ElementDegree="0" ElementFamily="" ItemType="" Name="pressure" Type="None">
          <DataItem DataType="Float" Dimensions="1 1895" Format="HDF" Precision="8">cube_1e3.h5:data/pressure|1 0:1 1:1 1895:2 1895</DataItem>
        </Attribute>
        <Attribute Center="Node" ElementCell="" ElementDegree="0" ElementFamily="" ItemType="" Name="v" Type="None">
          <DataItem DataType="Float" Dimensions="1 1895 3" Format="HDF" Precision="8">cube_1e3.h5:data/v|1 0 0:1 1 1:1 1895 3:2 1895 3</DataItem>
        </Attribute>
      </Grid>
    </Grid>
  </Domain>
</Xdmf>

cube_1e3.xdmf All time steps are in temporal grid collection. cube_1e3.h5:data/v|1 0 0:1 1 1:1 1895 3:2 1895 3
Is a hyperslab selection with

  • start: 1 0 0
  • strides: 1 1 1
  • local dim: 1 1895 3
  • global dim:2 1895 3

Static property

  1. Feature description was added to the changelog
  2. Tests covering your feature were added?
  3. Any new feature or behavior change was documented?
Edited by Tobias Meisel

Merge request reports