Skip to content
GitLab
  • Menu
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in / Register
  • O ogs
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 74
    • Issues 74
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 26
    • Merge requests 26
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages & Registries
    • Packages & Registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Wiki
    • Wiki
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar

Announcement: OGS Community Meeting 2022 in September

  • ogs
  • ogs
  • Merge requests
  • !3307

[MeL/IO] Parallel writer based on HDF5 and MPI IO

  • Review changes

  • Download
  • Email patches
  • Plain diff
Merged Tobias Meisel requested to merge TobiasMeisel/ogs:parallelHDF5Writer into master Dec 08, 2020
  • Overview 93
  • Commits 11
  • Pipelines 30
  • Changes 33
  • First parallel implementation of HDF5 writer. (activated when using XDMF as output format)
  • synchronous: computation->IO->computation->IO ..
  • The collective MPI IO is used as middleware.
  • Each process is writing its own data. This includes all ghost cells and nodes which can be viewed as a multi-block composite via xdmf. The inclusion of ghost cells will enable a simulation restart at already computer time step (not implemented yet).
  • The writing algorithm is not tuned for performance yet. Performance can be compared with VTK option, with this restriction: either use real world data or disable compression for both, see explanation in !3292 (merged)
  • all partition are written into a single hyperslab. Compression is off
  • views to HDF5 file via XDMF file. Default xdmf generated is view to first block (first partition)
  • parallel writing algorithm supports all features introduced for serial writing in !3246 (merged)
  1. Feature description was added to the changelog
  2. Tests covering your feature were added?
  3. Any new feature or behavior change was documented?

Next steps:

  • multi-write (for each attribute) to single-write
  • alternative to hyperslab to allow compression for each partition
  • analysis of bottleneck (darshan)
  • chunk-size
  • one process per node is writing data
  • async io
Edited Jan 05, 2021 by Tobias Meisel
Assignee
Assign to
Reviewer
Request review from
Time tracking
Source branch: parallelHDF5Writer