Skip to content

support setting HDF5 collective metadata operations #2199

@prudhomm

Description

@prudhomm

The objective is to enhance the HDF5 class to allow dynamic enabling or disabling of collective metadata operations (H5Pset_all_coll_metadata_ops and H5Pset_coll_metadata_write). This flexibility is crucial for optimizing performance in different parallel computing environments and use cases, especially when using the Lustre filesystem.

Background: Collective metadata operations can significantly improve I/O performance in applications involving parallel file access in high-performance computing (HPC). However, their impact varies depending on the specific parallel file system (e.g., Lustre) and the application's access patterns. Currently, our HDF5 class lacks the flexibility to control these settings, leading to a one-size-fits-all approach that may not be optimal in all scenarios.

here is an interesting presentation on the subject: https://www.hdfgroup.org/wp-content/uploads/2020/06/2020-06-26-Parallel-HDF5-Performance-Tuning.pdf

Metadata

Metadata

Assignees

Type

No type

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions