Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider adding support for different drivers in HDF5IO #75

Open
oruebel opened this issue Aug 20, 2024 · 0 comments
Open

Consider adding support for different drivers in HDF5IO #75

oruebel opened this issue Aug 20, 2024 · 0 comments
Labels
category: proposal proposed enhancements or new features priority: low alternative solution already working and/or relevant to only specific user(s)
Milestone

Comments

@oruebel
Copy link
Contributor

oruebel commented Aug 20, 2024

HDF5 supports a range of different "drivers" to store a "file" in different ways on disk https://docs.hdfgroup.org/hdf5/v1_14/_h5_f__u_g.html#title11 For AqNWB, drivers to split data across files could potentially be of interest, e.g., family, split and multi dirver

family : Split into sub-files based on size threshold

For very large recordings, the family driver in HDF5 could be useful. The driver automatically splits a file into subfiles based on user-defined size threshold. E.g.,:

Write:

    // File access property list
    hid_t fapl_id = H5Pcreate(H5P_FILE_ACCESS);
    H5Pset_fapl_family(fapl_id, 1024 * 1024 * 1024, H5P_DEFAULT); // 1GB per file

    // Create a new file using the family driver
    file_id = H5Fcreate(FILE_NAME, H5F_ACC_TRUNC, H5P_DEFAULT, fapl_id);

Read:

    // File access property list
    hid_t fapl_id = H5Pcreate(H5P_FILE_ACCESS);
    H5Pset_fapl_family(fapl_id, 1024 * 1024 * 1024, H5P_DEFAULT); // 1GB per file

    // Open the existing file using the family driver
    file_id = H5Fopen(FILE_NAME, H5F_ACC_RDONLY, fapl_id);
@oruebel oruebel added category: proposal proposed enhancements or new features priority: low alternative solution already working and/or relevant to only specific user(s) labels Aug 20, 2024
@oruebel oruebel added this to the Future milestone Aug 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category: proposal proposed enhancements or new features priority: low alternative solution already working and/or relevant to only specific user(s)
Projects
None yet
Development

No branches or pull requests

1 participant