-
Notifications
You must be signed in to change notification settings - Fork 67
Description
The objective is to enhance the HDF5 class to allow dynamic enabling or disabling of collective metadata operations (H5Pset_all_coll_metadata_ops and H5Pset_coll_metadata_write). This flexibility is crucial for optimizing performance in different parallel computing environments and use cases, especially when using the Lustre filesystem.
Background: Collective metadata operations can significantly improve I/O performance in applications involving parallel file access in high-performance computing (HPC). However, their impact varies depending on the specific parallel file system (e.g., Lustre) and the application's access patterns. Currently, our HDF5 class lacks the flexibility to control these settings, leading to a one-size-fits-all approach that may not be optimal in all scenarios.
here is an interesting presentation on the subject: https://www.hdfgroup.org/wp-content/uploads/2020/06/2020-06-26-Parallel-HDF5-Performance-Tuning.pdf
Metadata
Metadata
Assignees
Type
Projects
Status