Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New data format #502

Open
wants to merge 31 commits into
base: develop
Choose a base branch
from
Open

New data format #502

wants to merge 31 commits into from

Commits on Aug 16, 2024

  1. First pass at new format for data files.

    * Store static data as top-level metadata of the file
    RemingtonRohel committed Aug 16, 2024
    Configuration menu
    Copy the full SHA
    77888a4 View commit details
    Browse the repository at this point in the history

Commits on Aug 19, 2024

  1. Moved SliceData to its own file.

    * `data_write.SliceData` -> `utils.file_formats.SliceData`
    * Refactored the general HDF5 file format:
       - Top level "metadata" group
       - All data stored as `Dataset`, with `"description"` metadata attached
       - File-level metadata is hard-linked within each record group to the top-level "metadata" group entry
    RemingtonRohel committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    0243ab1 View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2024

  1. Major refactor of data_write.py and file formats.

    * Added labels for dimensions of vector fields in the data files, to aid in data interpretation and usage.
    * Created HDF5Writer class to handle turning SliceData dataclass into the correct types for writing to Borealis HDF5 files.
    * Removed ability to write JSON files
    * Started refactoring DataWrite.output_data() to remove the internal functions
    * DataWrite.__init__() now instantiates sockets internally (avoids passing sockets to threads, which is explicitly recommended against by zmq documentation)
    * Functionality to send rawacf record to realtime is now internal to write_correlations() function
    * Added `rx_main_phases` and `rx_intf_phases` fields
    * Removed `--file-type` option to data_write.py script (only hdf5 supported)
    * Replaced useless `assert` statement (ignored when script run with `-O` flag, which `steamed_hams.py` uses for release mode)
    RemingtonRohel committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    82a491f View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    64e174e View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    5d3b919 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2024

  1. Continued development of new data format.

    * Added support for writing rawacf files directly as DMAP
    * Added dimension scales to certain fields for HDF5 files. These are datasets associated with a dimension of another dataset, e.g. associating the `sqn_timestamps` with the "sequence" dimension of `data`.
    * Added units metadata for fields in HDF5 files.
    * Added "rawacf_format" field to config files, specifying the default format to use when writing rawacf files.
    * Added support to overwrite the rawacf format files are written with the "--rawacf-format" argument to steamed_hams.py
    * Added `darn-dmap` as a dependency, and fixed numpy and pydarnio versions.
    RemingtonRohel committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    bc7dd9b View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2024

  1. Refactored config file format to specify antenna locations.

    * New `antennas` field which holds all antenna information
       - `main_locations`: {index: [x, y, z]} for each main antenna
       - `intf_locations`: {index: [x, y, z]} for each intf antenna
       - `main_antenna_count`: number of main array antennas
       - `intf_antenna_count`: number of intf array antennas
       - `main_antenna_spacing`: uniform spacing between main-array antennas
       - `intf_antenna_spacing`: uniform spacing between intf-array antennas
       - `standard_positions`: flag indicating whether array antennas follow the standard linear configuration. If so, verifies that positions are parallel to x-axis and equally spaced by [main|intf]_antenna_spacing
    * Added tests for the new fields
    * Updated config files for each site
    RemingtonRohel committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    75a5127 View commit details
    Browse the repository at this point in the history

Commits on Sep 7, 2024

  1. Updates to fields stored in files.

    * Each type (antennas_iq, bfiq, etc.) has its own data field (antennas_iq_data, bfiq_data, etc.)
    * `antenna_locations` field added containing [x, y, z] locations of each antenna
    * `antenna_arrays` field added for bfiq files containing descriptors for the array dimension of the data (e.g. ["main", "intf"]
    * `required` added to metadata, indicating whether it is an error for a field to be missing or not.
    * `data` field removed
    * `[main|intf]_antenna_count` fields removed
    * `lags` field renamed to `lag_pulses`
    * `num_ranges` and `num_samps` fields removed
    * `range_gates` field added, simply an array of the range gates for the file (e.g. 0-74)
    * `rx_antennas`, ``rx_main_antennas`, `rx_intf_antennas`, and `tx_antennas` fields added, giving indices into `antenna_locations` of the antennas used for the experiment
    * `station_location` field added, giving lat, lon, altitude of the radar
    * Refactored `get_phase_shift()` in `signals.py` to use the antenna positions and interferometer array offsets for beamforming
    RemingtonRohel committed Sep 7, 2024
    Configuration menu
    Copy the full SHA
    5ea03a4 View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2024

  1. Configuration menu
    Copy the full SHA
    590dd20 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    23c9d1d View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    da6b06e View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    4763ee8 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    3a506ce View commit details
    Browse the repository at this point in the history
  6. Fix bug with trying to write missing optional scale fields

    * also refactored some variable names for simplicity
    RemingtonRohel committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    f141659 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    a9f9ac8 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    a7aba6d View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    0f66792 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    b5761b1 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    e422b2f View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    d67997b View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    09075bf View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2024

  1. Updating documentation of file types in ReadTheDocs.

    * Created script file_docs_builder.py to generate .rst files for each file type
    * Changed file name when writing DMAP files directly (ensuring slice_id is written as a letter instead of a number)
    RemingtonRohel committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    25fec2a View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2024

  1. Configuration menu
    Copy the full SHA
    40c644e View commit details
    Browse the repository at this point in the history
  2. Added sample_time field that is a Dimension Scale for antennas_iq_d…

    …ata, bfiq_data, rawrf_data final dimension.
    
    * Given as an array of ints, representing the time of measurement relative to the first pulse in the sequence. Microseconds.
    RemingtonRohel committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    ee57608 View commit details
    Browse the repository at this point in the history
  3. Created simulator to test software without N200s

    * `tests/simulators/steamed_sham.py` will call a simulator instead of usrp_driver.cpp
    * `tests/simulators/driver_sim.py` mocks usrp_driver.cpp, generating
      noise instead of data (and not currently adding the pulse data to the
    noise)
    RemingtonRohel committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    dcc0147 View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2024

  1. Bugfixes and testing with a steamed_sham.py

    * Updated record name format to use hyphens, like `YYYYMMDD-HHMM-SS.fffff`
    * Removed `dim_labels` from `lag_pulses` field
    * Correctly format SliceData object as DMAP
    * Use new pydarnio functions for converting rawacf records
    * Fix dmap filename convention
    * Serve data for all slices to realtime from data_write
    * Replaced test data for realtime sim with single-record dmap
    RemingtonRohel committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    dea7fc4 View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2024

  1. Configuration menu
    Copy the full SHA
    8c65992 View commit details
    Browse the repository at this point in the history
  2. Adding fields to make array-structured representation compatible with…

    … xarray
    
    * pyDARNio implementation of array-structured fields being added in parallel to this branch. This implementation will only allow in-memory array-structuring, and will not support writing of array-structured data files.
    RemingtonRohel committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    5f2b8f1 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2024

  1. Configuration menu
    Copy the full SHA
    4575290 View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2024

  1. Merge branch 'new_data_format' of github.com:SuperDARNCanada/borealis…

    … into new_data_format
    
    * Pulling in the merge conflict commit Draven added
    RemingtonRohel committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    f3183dc View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    0469e2d View commit details
    Browse the repository at this point in the history