Working with HDF data (examples h5py)

HDF5 file structure now allows easy access (also with time dimension) to data with h5py and allows parallel post-processing

Example post-processing with python

Install H5Py e.g. with pip:
pip install h5py

square_5x5_tris_32.h5

import h5py
def demo():
    with h5py.File('square_5x5_tris_32.h5') as myfile:
        # inspect what is inside the root of the file:
        # Each HDF will contain a group 'meshes' (with 1 or more meshes) and a group 'times' 
        # with time series
        myfile.keys()
        # Output: <KeysViewHDF5 ['meshes', 'times']>
        
        # Select the group 'meshes' and look what is inside and select a mesh
        meshes = myfile['meshes']
        meshes.keys()
        # < KeysViewHDF5['square_5x5_tris_32', 'square_5x5_tris_32_left_boundary', 'square_5x5_tris_32_right_boundary'] >

        # Select a mesh and check which attributes it contains
        mesh = meshes['square_5x5_tris_32']
        # Each mesh will contain 'geometry' and 'topology'. Optionally other attributes
        # < KeysViewHDF5['HydraulicFlow', 'MaterialIDs', 'geometry', 'pressure', 'topology', 'v']>    
        # Select an attribute
        # choose any key , here in example 'pressure', it is only a reference (no copy of data into memory)
        pressure = mesh['pressure']

        # get dimensions
        pressure.shape
        # example: (8, 25)
        # first dimension is time, here with length 8 - pressure is a variable attribute

        # e.g. for MateralIDs  we would get as number of time steps 1 - it is a constant attribute
        MaterialIDs = mesh['MaterialIDs']
        MaterialIDs.shape
        # (1, 25)
      
        # load data into memory (only selected data is loaded into memory)
        # example: load all timesteps and choose all data points 10:20
        p = pressure [:,10:20]

        # p can now be processed e.g. with numpy
        # import numpy
        # y = numpy.array(p)


        # In the example of pressure attribute we have 8 time steps for the variable attributes like pressure
        times = f['times']
        # list all time values
        times[:]
        # array([0. , 0.3, 0.6, 0.9, 1.2, 1.5, 1.8, 2. ])
        # To relate the values of pressure of timestep 2 (beginning with 0) to an actual time value we define:
        time_step = 2
        pressure_t2 = pressure[time_step,:]
        t2 = times[time_step]
        t2 
        # 0.6

For more details on selecting data there is

File structures

Structure of HDF5 file

Output of all time steps is written into a single hdf5 - file. All data is written into group "meshes" and its mesh name.

Example #1:

h5dump -H square_5x5_tris_32.h5 square_5x5_tris_32.h5

HDF5 "square_5x5_tris_32.h5" {
GROUP "/" {
   GROUP "meshes" {
      GROUP "square_5x5_tris_32" {
         DATASET "HydraulicFlow" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 1, 25 ) / ( H5S_UNLIMITED, 25 ) }
         }
         DATASET "MaterialIDs" {
            DATATYPE  H5T_STD_I32LE
            DATASPACE  SIMPLE { ( 1, 25 ) / ( H5S_UNLIMITED, 25 ) }
         }
         DATASET "geometry" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 1, 25, 3 ) / ( H5S_UNLIMITED, 25, 3 ) }
         }
         DATASET "pressure" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 8, 25 ) / ( H5S_UNLIMITED, 25 ) }
         }
         DATASET "topology" {
            DATATYPE  H5T_STD_I32LE
            DATASPACE  SIMPLE { ( 1, 128 ) / ( H5S_UNLIMITED, 128 ) }
         }
         DATASET "v" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 8, 25, 2 ) / ( H5S_UNLIMITED, 25, 2 ) }
         }
      }
      GROUP "square_5x5_tris_32_left_boundary" {
         DATASET "bulk_element_ids" {
            DATATYPE  H5T_STD_U64LE
            DATASPACE  SIMPLE { ( 1, 4 ) / ( H5S_UNLIMITED, 4 ) }
         }
         DATASET "bulk_node_ids" {
            DATATYPE  H5T_STD_U64LE
            DATASPACE  SIMPLE { ( 1, 5 ) / ( H5S_UNLIMITED, 5 ) }
         }
         DATASET "geometry" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 1, 5, 3 ) / ( H5S_UNLIMITED, 5, 3 ) }
         }
         DATASET "pressure" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 8, 5 ) / ( H5S_UNLIMITED, 5 ) }
         }
         DATASET "topology" {
            DATATYPE  H5T_STD_I32LE
            DATASPACE  SIMPLE { ( 1, 16 ) / ( H5S_UNLIMITED, 16 ) }
         }
      }
      GROUP "square_5x5_tris_32_right_boundary" {
         DATASET "bulk_element_ids" {
            DATATYPE  H5T_STD_U64LE
            DATASPACE  SIMPLE { ( 1, 4 ) / ( H5S_UNLIMITED, 4 ) }
         }
         DATASET "bulk_node_ids" {
            DATATYPE  H5T_STD_U64LE
            DATASPACE  SIMPLE { ( 1, 5 ) / ( H5S_UNLIMITED, 5 ) }
         }
         DATASET "geometry" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 1, 5, 3 ) / ( H5S_UNLIMITED, 5, 3 ) }
         }
         DATASET "pressure" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 8, 5 ) / ( H5S_UNLIMITED, 5 ) }
         }
         DATASET "topology" {
            DATATYPE  H5T_STD_I32LE
            DATASPACE  SIMPLE { ( 1, 16 ) / ( H5S_UNLIMITED, 16 ) }
         }
      }
   }
   DATASET "times" {
      DATATYPE  H5T_IEEE_F64LE
      DATASPACE  SIMPLE { ( 8 ) / ( 8 ) }
   }
}
}

Example (and cut example) from h5dump -H cube_1e3.h5 cube_1e3.h5

HDF5 "cube_1e3.h5" {
GROUP "/" {
   GROUP "data" {
      DATASET "MaterialIDs" {
         DATATYPE  H5T_STD_I32LE
         DATASPACE  SIMPLE { ( 1, 1233 ) / ( H5S_UNLIMITED, 1233 ) }
      }
      DATASET "geometry" {
         DATATYPE  H5T_IEEE_F64LE
         DATASPACE  SIMPLE { ( 1, 1895, 3 ) / ( H5S_UNLIMITED, 1895, 3 ) }
      }
...

Static property

DATASPACE SIMPLE { ( 1, 1233 ) All properties have an additional dimension, representing the time steps. It is first dimension (for performance reasons) Subsequent dimension represent the data dimensions, here 1D with length 1233 for MaterialIDs. If the length of time steps is 1 - it is a constant property/attribute. Geometry and Topology are treated as constant properties/attributes.

Dynamic property

      
      DATASET "pressure" {
         DATATYPE  H5T_IEEE_F64LE
         DATASPACE  SIMPLE { ( 2, 1895 ) / ( H5S_UNLIMITED, 1895 ) }
      }
      
      DATASET "v" {
         DATATYPE  H5T_IEEE_F64LE
         DATASPACE  SIMPLE { ( 2, 1895, 3 ) / ( H5S_UNLIMITED, 1895, 3 ) }
      }
   }
}
}

pressure and v are dynamic properties. Here the simulation run for 2 time steps. The length of first dimension is equal to the number of time steps.

Structure of XDMF file

Important!
Never do any post-processing based on XDMF-format! OGS-XDMF is only for visualization in Paraview. Load .hdf file and use h5py instead!

<?xml version="1.0" encoding="utf-8"?>
<Xdmf xmlns:xi="http://www.w3.org/2001/XInclude" Version="3.0">
  <Domain>
    <Information Name="OGS_VERSION" Value="6.4.0-269-g0177d3b3a.dirty"/>
    <Grid CollectionType="Temporal" GridType="Collection" Name="Collection">
      <Grid Name="Grid">
        <Time Value="0"/>
        <Geometry Origin="" Type="XYZ">
          <DataItem DataType="Float" Dimensions="1895 3" Format="HDF" Precision="8">cube_1e3.h5:data/geometry</DataItem>
        </Geometry>
        <Topology Dimensions="11097" Type="Mixed">
          <DataItem DataType="Int" Dimensions="11097" Format="HDF" Precision="4">cube_1e3.h5:data/topology</DataItem>
        </Topology>
        <Attribute Center="Cell" ElementCell="" ElementDegree="0" ElementFamily="" ItemType="" Name="MaterialIDs" Type="None">
          <DataItem DataType="Int" Dimensions="1 1233" Format="HDF" Precision="4">cube_1e3.h5:data/MaterialIDs</DataItem>
        </Attribute>
        <Attribute Center="Node" ElementCell="" ElementDegree="0" ElementFamily="" ItemType="" Name="pressure" Type="None">
          <DataItem DataType="Float" Dimensions="1 1895" Format="HDF" Precision="8">cube_1e3.h5:data/pressure</DataItem>
        </Attribute>
        <Attribute Center="Node" ElementCell="" ElementDegree="0" ElementFamily="" ItemType="" Name="v" Type="None">
          <DataItem DataType="Float" Dimensions="1 1895 3" Format="HDF" Precision="8">cube_1e3.h5:data/v</DataItem>
        </Attribute>
      </Grid>
      <Grid Name="Grid">
        <Time Value="0.1"/>
        <xi:include xpointer="element(/1/1/2/1/4)"/>
        <xi:include xpointer="element(/1/1/2/1/5)"/>
        <xi:include xpointer="element(/1/1/2/1/6)"/>
        <Attribute Center="Node" ElementCell="" ElementDegree="0" ElementFamily="" ItemType="" Name="pressure" Type="None">
          <DataItem DataType="Float" Dimensions="1 1895" Format="HDF" Precision="8">cube_1e3.h5:data/pressure|1 0:1 1:1 1895:2 1895</DataItem>
        </Attribute>
        <Attribute Center="Node" ElementCell="" ElementDegree="0" ElementFamily="" ItemType="" Name="v" Type="None">
          <DataItem DataType="Float" Dimensions="1 1895 3" Format="HDF" Precision="8">cube_1e3.h5:data/v|1 0 0:1 1 1:1 1895 3:2 1895 3</DataItem>
        </Attribute>
      </Grid>
    </Grid>
  </Domain>
</Xdmf>

cube_1e3.xdmf All time steps are in temporal grid collection. cube_1e3.h5:data/v|1 0 0:1 1 1:1 1895 3:2 1895 3
Is a hyperslab selection with

start: 1 0 0
strides: 1 1 1
local dim: 1 1895 3
global dim:2 1895 3