[MeL/IO] HDF5 - Types of Application I/O to Parallel File Systems

Overview

In 1 different types of IO to parallel file systems are shown.

Until now there is:

• VTU as File-per-processor
• HDF5 as Shared file

With this MR

• HDF5 as File-per-group is added

Open point is:

• Collective buffering

I/O groups

A group may be:

• a single processor -> File-per-processor
• multiple processor -> Shared-file
• a single node (see notes File-per-node)

Num_of_files

The groups are defined by the new introduced parameter number_of_files in the output tag. It is active only for type XDMF

 <output>
            <type>XDMF</type>
            <prefix>cube_1e3_np3</prefix>
            <variables>
                <variable> pressure </variable>
                <variable> v      </variable>
            </variables>
            <suffix>_ts_{:timestep}_t_{:time}</suffix>
            <hdf>
                <number_of_files>2</number_of_files>
            </hdf>
        </output>

File per node

1 File per node is achieved when number_of_files is set to the number of nodes.

e.g.

• 48 (processor per node on JUWELS)
• example with 768 partitions -> 768 processors
• number of files must be set to 768/48 = 16

1. Feature description was added to the changelog
2. Tests covering your feature were added?
3. Any new feature or behavior change was documented?

1