-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
New data format #502
base: develop
Are you sure you want to change the base?
New data format #502
Commits on Aug 16, 2024
-
First pass at new format for data files.
* Store static data as top-level metadata of the file
Configuration menu - View commit details
-
Copy full SHA for 77888a4 - Browse repository at this point
Copy the full SHA 77888a4View commit details
Commits on Aug 19, 2024
-
Moved SliceData to its own file.
* `data_write.SliceData` -> `utils.file_formats.SliceData` * Refactored the general HDF5 file format: - Top level "metadata" group - All data stored as `Dataset`, with `"description"` metadata attached - File-level metadata is hard-linked within each record group to the top-level "metadata" group entry
Configuration menu - View commit details
-
Copy full SHA for 0243ab1 - Browse repository at this point
Copy the full SHA 0243ab1View commit details
Commits on Aug 29, 2024
-
Major refactor of data_write.py and file formats.
* Added labels for dimensions of vector fields in the data files, to aid in data interpretation and usage. * Created HDF5Writer class to handle turning SliceData dataclass into the correct types for writing to Borealis HDF5 files. * Removed ability to write JSON files * Started refactoring DataWrite.output_data() to remove the internal functions * DataWrite.__init__() now instantiates sockets internally (avoids passing sockets to threads, which is explicitly recommended against by zmq documentation) * Functionality to send rawacf record to realtime is now internal to write_correlations() function * Added `rx_main_phases` and `rx_intf_phases` fields * Removed `--file-type` option to data_write.py script (only hdf5 supported) * Replaced useless `assert` statement (ignored when script run with `-O` flag, which `steamed_hams.py` uses for release mode)
Configuration menu - View commit details
-
Copy full SHA for 82a491f - Browse repository at this point
Copy the full SHA 82a491fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 64e174e - Browse repository at this point
Copy the full SHA 64e174eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 5d3b919 - Browse repository at this point
Copy the full SHA 5d3b919View commit details
Commits on Aug 30, 2024
-
Continued development of new data format.
* Added support for writing rawacf files directly as DMAP * Added dimension scales to certain fields for HDF5 files. These are datasets associated with a dimension of another dataset, e.g. associating the `sqn_timestamps` with the "sequence" dimension of `data`. * Added units metadata for fields in HDF5 files. * Added "rawacf_format" field to config files, specifying the default format to use when writing rawacf files. * Added support to overwrite the rawacf format files are written with the "--rawacf-format" argument to steamed_hams.py * Added `darn-dmap` as a dependency, and fixed numpy and pydarnio versions.
Configuration menu - View commit details
-
Copy full SHA for bc7dd9b - Browse repository at this point
Copy the full SHA bc7dd9bView commit details
Commits on Sep 6, 2024
-
Refactored config file format to specify antenna locations.
* New `antennas` field which holds all antenna information - `main_locations`: {index: [x, y, z]} for each main antenna - `intf_locations`: {index: [x, y, z]} for each intf antenna - `main_antenna_count`: number of main array antennas - `intf_antenna_count`: number of intf array antennas - `main_antenna_spacing`: uniform spacing between main-array antennas - `intf_antenna_spacing`: uniform spacing between intf-array antennas - `standard_positions`: flag indicating whether array antennas follow the standard linear configuration. If so, verifies that positions are parallel to x-axis and equally spaced by [main|intf]_antenna_spacing * Added tests for the new fields * Updated config files for each site
Configuration menu - View commit details
-
Copy full SHA for 75a5127 - Browse repository at this point
Copy the full SHA 75a5127View commit details
Commits on Sep 7, 2024
-
Updates to fields stored in files.
* Each type (antennas_iq, bfiq, etc.) has its own data field (antennas_iq_data, bfiq_data, etc.) * `antenna_locations` field added containing [x, y, z] locations of each antenna * `antenna_arrays` field added for bfiq files containing descriptors for the array dimension of the data (e.g. ["main", "intf"] * `required` added to metadata, indicating whether it is an error for a field to be missing or not. * `data` field removed * `[main|intf]_antenna_count` fields removed * `lags` field renamed to `lag_pulses` * `num_ranges` and `num_samps` fields removed * `range_gates` field added, simply an array of the range gates for the file (e.g. 0-74) * `rx_antennas`, ``rx_main_antennas`, `rx_intf_antennas`, and `tx_antennas` fields added, giving indices into `antenna_locations` of the antennas used for the experiment * `station_location` field added, giving lat, lon, altitude of the radar * Refactored `get_phase_shift()` in `signals.py` to use the antenna positions and interferometer array offsets for beamforming
Configuration menu - View commit details
-
Copy full SHA for 5ea03a4 - Browse repository at this point
Copy the full SHA 5ea03a4View commit details
Commits on Sep 9, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 590dd20 - Browse repository at this point
Copy the full SHA 590dd20View commit details -
Configuration menu - View commit details
-
Copy full SHA for 23c9d1d - Browse repository at this point
Copy the full SHA 23c9d1dView commit details -
Configuration menu - View commit details
-
Copy full SHA for da6b06e - Browse repository at this point
Copy the full SHA da6b06eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4763ee8 - Browse repository at this point
Copy the full SHA 4763ee8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3a506ce - Browse repository at this point
Copy the full SHA 3a506ceView commit details -
Fix bug with trying to write missing optional scale fields
* also refactored some variable names for simplicity
Configuration menu - View commit details
-
Copy full SHA for f141659 - Browse repository at this point
Copy the full SHA f141659View commit details -
Configuration menu - View commit details
-
Copy full SHA for a9f9ac8 - Browse repository at this point
Copy the full SHA a9f9ac8View commit details -
Configuration menu - View commit details
-
Copy full SHA for a7aba6d - Browse repository at this point
Copy the full SHA a7aba6dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0f66792 - Browse repository at this point
Copy the full SHA 0f66792View commit details -
Configuration menu - View commit details
-
Copy full SHA for b5761b1 - Browse repository at this point
Copy the full SHA b5761b1View commit details -
Configuration menu - View commit details
-
Copy full SHA for e422b2f - Browse repository at this point
Copy the full SHA e422b2fView commit details -
Configuration menu - View commit details
-
Copy full SHA for d67997b - Browse repository at this point
Copy the full SHA d67997bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 09075bf - Browse repository at this point
Copy the full SHA 09075bfView commit details
Commits on Sep 10, 2024
-
Updating documentation of file types in ReadTheDocs.
* Created script file_docs_builder.py to generate .rst files for each file type * Changed file name when writing DMAP files directly (ensuring slice_id is written as a letter instead of a number)
Configuration menu - View commit details
-
Copy full SHA for 25fec2a - Browse repository at this point
Copy the full SHA 25fec2aView commit details
Commits on Sep 11, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 40c644e - Browse repository at this point
Copy the full SHA 40c644eView commit details -
Added
sample_time
field that is a Dimension Scale for antennas_iq_d……ata, bfiq_data, rawrf_data final dimension. * Given as an array of ints, representing the time of measurement relative to the first pulse in the sequence. Microseconds.
Configuration menu - View commit details
-
Copy full SHA for ee57608 - Browse repository at this point
Copy the full SHA ee57608View commit details -
Created simulator to test software without N200s
* `tests/simulators/steamed_sham.py` will call a simulator instead of usrp_driver.cpp * `tests/simulators/driver_sim.py` mocks usrp_driver.cpp, generating noise instead of data (and not currently adding the pulse data to the noise)
Configuration menu - View commit details
-
Copy full SHA for dcc0147 - Browse repository at this point
Copy the full SHA dcc0147View commit details
Commits on Sep 12, 2024
-
Bugfixes and testing with a
steamed_sham.py
* Updated record name format to use hyphens, like `YYYYMMDD-HHMM-SS.fffff` * Removed `dim_labels` from `lag_pulses` field * Correctly format SliceData object as DMAP * Use new pydarnio functions for converting rawacf records * Fix dmap filename convention * Serve data for all slices to realtime from data_write * Replaced test data for realtime sim with single-record dmap
Configuration menu - View commit details
-
Copy full SHA for dea7fc4 - Browse repository at this point
Copy the full SHA dea7fc4View commit details
Commits on Sep 13, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 8c65992 - Browse repository at this point
Copy the full SHA 8c65992View commit details -
Adding fields to make array-structured representation compatible with…
… xarray * pyDARNio implementation of array-structured fields being added in parallel to this branch. This implementation will only allow in-memory array-structuring, and will not support writing of array-structured data files.
Configuration menu - View commit details
-
Copy full SHA for 5f2b8f1 - Browse repository at this point
Copy the full SHA 5f2b8f1View commit details
Commits on Sep 18, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 4575290 - Browse repository at this point
Copy the full SHA 4575290View commit details
Commits on Sep 23, 2024
-
Merge branch 'new_data_format' of github.com:SuperDARNCanada/borealis…
… into new_data_format * Pulling in the merge conflict commit Draven added
Configuration menu - View commit details
-
Copy full SHA for f3183dc - Browse repository at this point
Copy the full SHA f3183dcView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0469e2d - Browse repository at this point
Copy the full SHA 0469e2dView commit details