You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
neural-flows uses a type of matfile that allows for partial i/o operations. With current SSD w/r speeds it shouldn't be a problem. This choice was made to keep memory usage to minimum. The disadvantage is that this feature may bring an HPC down if the storage directory is on a network drive rather than scratch space.
Describe the solution you'd like
For small datasets, use a struct() with equivalent hierarchy and organisation as the current file_object. Keep everything in memory and write at the end.
Describe alternatives you've considered
Memory mapped files, but they don't work very well with parallelised loops.
Pick any two: minimise bookkeping complexity, enable parallel computations, minimise memory usage.
The text was updated successfully, but these errors were encountered:
neural-flows
uses a type of matfile that allows for partial i/o operations. With current SSD w/r speeds it shouldn't be a problem. This choice was made to keep memory usage to minimum. The disadvantage is that this feature may bring an HPC down if the storage directory is on a network drive rather than scratch space.Describe the solution you'd like
For small datasets, use a
struct()
with equivalent hierarchy and organisation as the currentfile_object
. Keep everything in memory and write at the end.Describe alternatives you've considered
Memory mapped files, but they don't work very well with parallelised loops.
Pick any two: minimise bookkeping complexity, enable parallel computations, minimise memory usage.
The text was updated successfully, but these errors were encountered: