|
|
HDF5 file structure now allows easy access (also with time dimension) to data with h5py and allows [parallel post-processing](https://docs.h5py.org/en/stable/mpi.html)
|
|
|
|
|
|
# Example post-processing with python
|
|
|
|
|
|
Install [H5Py](https://docs.h5py.org) e.g. with pip: \
|
|
|
`pip install h5py`
|
|
|
|
|
|
Example file: [cube_1e3.h5](/uploads/b6771c94baefbc4602e84ff0f4ca876a/cube_1e3.h5)
|
|
|
|
|
|
```python
|
|
|
import h5py
|
|
|
def demo():
|
|
|
with h5py.File('cube_1e3.h5') as myfile:
|
|
|
# all data will always be in 'data' section
|
|
|
data = myfile['data']
|
|
|
# inspect data
|
|
|
data.keys()
|
|
|
# example: <KeysViewHDF5 ['D1_left_front_N1_right', 'Linear_1_to_minus1', 'MaterialIDs', 'geometry', 'pressure', 'topology', 'v']>
|
|
|
|
|
|
# choose any key , here in example 'pressure', it is only a reference (no copy of data into memory)
|
|
|
pressure = data['pressure']
|
|
|
|
|
|
# get dimensions
|
|
|
pressure.shape
|
|
|
# example: (2, 1895)
|
|
|
# first dimension is time, here with length 2
|
|
|
|
|
|
# load data into memory (only selected data is loaded into memory)
|
|
|
# example: load all timesteps and choose all data points 100:200
|
|
|
p = pressure [:,100:200]
|
|
|
|
|
|
# p can now be processed e.g. with numpy
|
|
|
# import numpy
|
|
|
# y = numpy.array(p)
|
|
|
```
|
|
|
|
|
|
For more details on selecting data there is
|
|
|
|
|
|
* [H5Py](https://docs.h5py.org/en/stable/high/dataset.html)
|
|
|
* [Python slicing](https://www.geeksforgeeks.org/python-list-slicing)
|
|
|
* [Numpy slicing](https://numpy.org/doc/stable/reference/arrays.indexing.html)
|
|
|
|
|
|
# File structures
|
|
|
|
|
|
## Structure of HDF5 file
|
|
|
|
|
|
Output of all time steps is written into a single hdf5 - file. All data is written into group "data".
|
|
|
|
|
|
Example (and cut example) from `h5dump -H cube_1e3.h5` [cube_1e3.h5](/uploads/b6771c94baefbc4602e84ff0f4ca876a/cube_1e3.h5)
|
|
|
|
|
|
```plaintext
|
|
|
HDF5 "cube_1e3.h5" {
|
|
|
GROUP "/" {
|
|
|
GROUP "data" {
|
|
|
DATASET "MaterialIDs" {
|
|
|
DATATYPE H5T_STD_I32LE
|
|
|
DATASPACE SIMPLE { ( 1, 1233 ) / ( H5S_UNLIMITED, 1233 ) }
|
|
|
}
|
|
|
DATASET "geometry" {
|
|
|
DATATYPE H5T_IEEE_F64LE
|
|
|
DATASPACE SIMPLE { ( 1, 1895, 3 ) / ( H5S_UNLIMITED, 1895, 3 ) }
|
|
|
}
|
|
|
...
|
|
|
```
|
|
|
|
|
|
### Static property
|
|
|
|
|
|
`DATASPACE SIMPLE { ( 1, 1233 )` All properties have an additional dimension, representing the time steps. It is first dimension (for performance reasons) Subsequent dimension represent the data dimensions, here 1D with length 1233 for MaterialIDs. If the length of time steps is 1 - it is a constant property/attribute. Geometry and Topology are treated as constant properties/attributes.
|
|
|
|
|
|
### Dynamic property
|
|
|
|
|
|
```xml
|
|
|
|
|
|
DATASET "pressure" {
|
|
|
DATATYPE H5T_IEEE_F64LE
|
|
|
DATASPACE SIMPLE { ( 2, 1895 ) / ( H5S_UNLIMITED, 1895 ) }
|
|
|
}
|
|
|
|
|
|
DATASET "v" {
|
|
|
DATATYPE H5T_IEEE_F64LE
|
|
|
DATASPACE SIMPLE { ( 2, 1895, 3 ) / ( H5S_UNLIMITED, 1895, 3 ) }
|
|
|
}
|
|
|
}
|
|
|
}
|
|
|
}
|
|
|
```
|
|
|
|
|
|
pressure and v are dynamic properties. Here the simulation run for 2 time steps. The length of first dimension is equal to the number of time steps.
|
|
|
|
|
|
## Structure of XDMF file
|
|
|
|
|
|
**Important!** \
|
|
|
Never do any post-processing based on XDMF-format! OGS-XDMF is only for visualization in Paraview. Load .hdf file and use h5py instead!
|
|
|
|
|
|
```plaintext
|
|
|
<?xml version="1.0" encoding="utf-8"?>
|
|
|
<Xdmf xmlns:xi="http://www.w3.org/2001/XInclude" Version="3.0">
|
|
|
<Domain>
|
|
|
<Information Name="OGS_VERSION" Value="6.4.0-269-g0177d3b3a.dirty"/>
|
|
|
<Grid CollectionType="Temporal" GridType="Collection" Name="Collection">
|
|
|
<Grid Name="Grid">
|
|
|
<Time Value="0"/>
|
|
|
<Geometry Origin="" Type="XYZ">
|
|
|
<DataItem DataType="Float" Dimensions="1895 3" Format="HDF" Precision="8">cube_1e3.h5:data/geometry</DataItem>
|
|
|
</Geometry>
|
|
|
<Topology Dimensions="11097" Type="Mixed">
|
|
|
<DataItem DataType="Int" Dimensions="11097" Format="HDF" Precision="4">cube_1e3.h5:data/topology</DataItem>
|
|
|
</Topology>
|
|
|
<Attribute Center="Cell" ElementCell="" ElementDegree="0" ElementFamily="" ItemType="" Name="MaterialIDs" Type="None">
|
|
|
<DataItem DataType="Int" Dimensions="1 1233" Format="HDF" Precision="4">cube_1e3.h5:data/MaterialIDs</DataItem>
|
|
|
</Attribute>
|
|
|
<Attribute Center="Node" ElementCell="" ElementDegree="0" ElementFamily="" ItemType="" Name="pressure" Type="None">
|
|
|
<DataItem DataType="Float" Dimensions="1 1895" Format="HDF" Precision="8">cube_1e3.h5:data/pressure</DataItem>
|
|
|
</Attribute>
|
|
|
<Attribute Center="Node" ElementCell="" ElementDegree="0" ElementFamily="" ItemType="" Name="v" Type="None">
|
|
|
<DataItem DataType="Float" Dimensions="1 1895 3" Format="HDF" Precision="8">cube_1e3.h5:data/v</DataItem>
|
|
|
</Attribute>
|
|
|
</Grid>
|
|
|
<Grid Name="Grid">
|
|
|
<Time Value="0.1"/>
|
|
|
<xi:include xpointer="element(/1/1/2/1/4)"/>
|
|
|
<xi:include xpointer="element(/1/1/2/1/5)"/>
|
|
|
<xi:include xpointer="element(/1/1/2/1/6)"/>
|
|
|
<Attribute Center="Node" ElementCell="" ElementDegree="0" ElementFamily="" ItemType="" Name="pressure" Type="None">
|
|
|
<DataItem DataType="Float" Dimensions="1 1895" Format="HDF" Precision="8">cube_1e3.h5:data/pressure|1 0:1 1:1 1895:2 1895</DataItem>
|
|
|
</Attribute>
|
|
|
<Attribute Center="Node" ElementCell="" ElementDegree="0" ElementFamily="" ItemType="" Name="v" Type="None">
|
|
|
<DataItem DataType="Float" Dimensions="1 1895 3" Format="HDF" Precision="8">cube_1e3.h5:data/v|1 0 0:1 1 1:1 1895 3:2 1895 3</DataItem>
|
|
|
</Attribute>
|
|
|
</Grid>
|
|
|
</Grid>
|
|
|
</Domain>
|
|
|
</Xdmf>
|
|
|
```
|
|
|
|
|
|
[cube_1e3.xdmf](/uploads/4ffdde382cb6cfe79fc52f554050c2b3/cube_1e3.xdmf) All time steps are in temporal grid collection. `cube_1e3.h5:data/v|1 0 0:1 1 1:1 1895 3:2 1895 3`\
|
|
|
Is a hyperslab selection with
|
|
|
|
|
|
* start: 1 0 0
|
|
|
* strides: 1 1 1
|
|
|
* local dim: 1 1895 3
|
|
|
* global dim:2 1895 3 |
|
|
\ No newline at end of file |