[MeL/IO] Fast time access and better usage of OGS-HDF5 output file
HDF5 file structure now allows easy access (also with time dimension) to data with h5py and allows parallel post-processing
Example post-processing with python
Install H5Py e.g. with pip:
pip install h5py
Example file: cube_1e3.h5
import h5py
def demo():
with h5py.File('cube_1e3.h5') as myfile:
# all data will always be in 'data' section
data = myfile['data']
# inspect data
data.keys()
# example: <KeysViewHDF5 ['D1_left_front_N1_right', 'Linear_1_to_minus1', 'MaterialIDs', 'geometry', 'pressure', 'topology', 'v']>
# choose any key , here in example 'pressure', it is only a reference (no copy of data into memory)
pressure = data['pressure']
# get dimensions
pressure.shape
# example: (2, 1895)
# first dimension is time, here with length 2
# load data into memory (only selected data is loaded into memory)
# example: load all timesteps and choose all data points 100:200
p = pressure [:,100:200]
# p can now be processed e.g. with numpy
# import numpy
# y = numpy.array(p)
For more details on selecting data there is
File structures
Structure of HDF5 file
Output of all time steps is written into a single hdf5 - file. All data is written into group "data".
Example (and cut example) from h5dump -H cube_1e3.h5
cube_1e3.h5
HDF5 "cube_1e3.h5" {
GROUP "/" {
GROUP "data" {
DATASET "MaterialIDs" {
DATATYPE H5T_STD_I32LE
DATASPACE SIMPLE { ( 1, 1233 ) / ( H5S_UNLIMITED, 1233 ) }
}
DATASET "geometry" {
DATATYPE H5T_IEEE_F64LE
DATASPACE SIMPLE { ( 1, 1895, 3 ) / ( H5S_UNLIMITED, 1895, 3 ) }
}
...
Static property
DATASPACE SIMPLE { ( 1, 1233 )
All properties have an additional dimension, representing the time steps.
It is first dimension (for performance reasons)
Subsequent dimension represent the data dimensions, here 1D with length 1233 for MaterialIDs.
If the length of time steps is 1 - it is a constant property/attribute. Geometry and Topology are treated as constant properties/attributes.
Dynamic property
DATASET "pressure" {
DATATYPE H5T_IEEE_F64LE
DATASPACE SIMPLE { ( 2, 1895 ) / ( H5S_UNLIMITED, 1895 ) }
}
DATASET "v" {
DATATYPE H5T_IEEE_F64LE
DATASPACE SIMPLE { ( 2, 1895, 3 ) / ( H5S_UNLIMITED, 1895, 3 ) }
}
}
}
}
pressure and v are dynamic properties. Here the simulation run for 2 time steps. The length of first dimension is equal to the number of time steps.
Structure of XDMF file
Important!
Never do any post-processing based on XDMF-format! OGS-XDMF is only for visualization in Paraview.
Load .hdf file and use h5py instead!
<?xml version="1.0" encoding="utf-8"?>
<Xdmf xmlns:xi="http://www.w3.org/2001/XInclude" Version="3.0">
<Domain>
<Information Name="OGS_VERSION" Value="6.4.0-269-g0177d3b3a.dirty"/>
<Grid CollectionType="Temporal" GridType="Collection" Name="Collection">
<Grid Name="Grid">
<Time Value="0"/>
<Geometry Origin="" Type="XYZ">
<DataItem DataType="Float" Dimensions="1895 3" Format="HDF" Precision="8">cube_1e3.h5:data/geometry</DataItem>
</Geometry>
<Topology Dimensions="11097" Type="Mixed">
<DataItem DataType="Int" Dimensions="11097" Format="HDF" Precision="4">cube_1e3.h5:data/topology</DataItem>
</Topology>
<Attribute Center="Cell" ElementCell="" ElementDegree="0" ElementFamily="" ItemType="" Name="MaterialIDs" Type="None">
<DataItem DataType="Int" Dimensions="1 1233" Format="HDF" Precision="4">cube_1e3.h5:data/MaterialIDs</DataItem>
</Attribute>
<Attribute Center="Node" ElementCell="" ElementDegree="0" ElementFamily="" ItemType="" Name="pressure" Type="None">
<DataItem DataType="Float" Dimensions="1 1895" Format="HDF" Precision="8">cube_1e3.h5:data/pressure</DataItem>
</Attribute>
<Attribute Center="Node" ElementCell="" ElementDegree="0" ElementFamily="" ItemType="" Name="v" Type="None">
<DataItem DataType="Float" Dimensions="1 1895 3" Format="HDF" Precision="8">cube_1e3.h5:data/v</DataItem>
</Attribute>
</Grid>
<Grid Name="Grid">
<Time Value="0.1"/>
<xi:include xpointer="element(/1/1/2/1/4)"/>
<xi:include xpointer="element(/1/1/2/1/5)"/>
<xi:include xpointer="element(/1/1/2/1/6)"/>
<Attribute Center="Node" ElementCell="" ElementDegree="0" ElementFamily="" ItemType="" Name="pressure" Type="None">
<DataItem DataType="Float" Dimensions="1 1895" Format="HDF" Precision="8">cube_1e3.h5:data/pressure|1 0:1 1:1 1895:2 1895</DataItem>
</Attribute>
<Attribute Center="Node" ElementCell="" ElementDegree="0" ElementFamily="" ItemType="" Name="v" Type="None">
<DataItem DataType="Float" Dimensions="1 1895 3" Format="HDF" Precision="8">cube_1e3.h5:data/v|1 0 0:1 1 1:1 1895 3:2 1895 3</DataItem>
</Attribute>
</Grid>
</Grid>
</Domain>
</Xdmf>
cube_1e3.xdmf
All time steps are in temporal grid collection.
cube_1e3.h5:data/v|1 0 0:1 1 1:1 1895 3:2 1895 3
Is a hyperslab selection with
- start: 1 0 0
- strides: 1 1 1
- local dim: 1 1895 3
- global dim:2 1895 3
Static property
-
Feature description was added to the changelog -
Tests covering your feature were added? -
Any new feature or behavior change was documented?