There is already the https://github.com/pangeo-data/storage-benchmarks repository, which we can build on (possibly move into this repo). I think that these benchmarks should consider different formats:
And I think we need to compare these to their "idealized" use cases, which are independent I/O (i.e., each process reads/writes from/to its own file) for zarr and MPI-IO (each process reads/writes from/to the same file) for hdf5 and netcdf.
These benchmarks should be run on different platforms and storage systems (HPC with GPFS or Lustre, AWS S3, GCS, etc.).
What all do we need for this?
There is already the https://github.com/pangeo-data/storage-benchmarks repository, which we can build on (possibly move into this repo). I think that these benchmarks should consider different formats:
And I think we need to compare these to their "idealized" use cases, which are independent I/O (i.e., each process reads/writes from/to its own file) for zarr and MPI-IO (each process reads/writes from/to the same file) for hdf5 and netcdf.
These benchmarks should be run on different platforms and storage systems (HPC with GPFS or Lustre, AWS S3, GCS, etc.).
What all do we need for this?