-
Notifications
You must be signed in to change notification settings - Fork 11
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Replace memmap usage with direct file I/O (#82)
* Replace memmap usage with direct file I/O Previously, the library made extensive usage of memory-map to copy the contents of inputs & outputs to & from binary files for compatibility with the SNAPHU executable. Memory-mapping is convenient for copying 2-D blocks of data to ensure that we don't run out of memory while copying entire large datasets. However, there seems to be no interface in NumPy or Python's `mmap` library to ensure that the memory-mapped file is safely closed (at least prior to Python 3.13 -- see python/cpython#78502). This may cause issues when unwrapping a large series of interferograms in a single process, potentially leading to the number of open file descriptors exceeding the system's resource limits. In this update, we replace usage of `numpy.memmap` with direct file I/O. In order to avoid undue complexity (as well as potential performance issues due to buffered file access) we now read/write batches of data that span all columns of the input/output 2-D datasets, rather than operating on 2-D sub-blocks of data. This new approach gives full control over when files are opened and closed, in order to avoid leaking resources. * Close files opened by `tempfile.mkstemp` Add a wrapper for `tempfile.mkstemp` that ensures that open files are properly closed. Replace usage of `tempfile.mkstemp` with the wrapper function.
- Loading branch information
Showing
4 changed files
with
324 additions
and
299 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.