Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New data format #502

Open
wants to merge 31 commits into
base: develop
Choose a base branch
from
Open

New data format #502

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
77888a4
First pass at new format for data files.
RemingtonRohel Aug 16, 2024
0243ab1
Moved SliceData to its own file.
RemingtonRohel Aug 19, 2024
82a491f
Major refactor of data_write.py and file formats.
RemingtonRohel Aug 29, 2024
64e174e
Ran `ruff check` and `ruff format` over the files modified by the pre…
RemingtonRohel Aug 29, 2024
5d3b919
Starting to add support for writing rawacf files as DMAP
RemingtonRohel Aug 29, 2024
bc7dd9b
Continued development of new data format.
RemingtonRohel Aug 30, 2024
75a5127
Refactored config file format to specify antenna locations.
RemingtonRohel Sep 6, 2024
5ea03a4
Updates to fields stored in files.
RemingtonRohel Sep 7, 2024
590dd20
Fixing requirements.txt in docs/
RemingtonRohel Sep 9, 2024
23c9d1d
Fixing relative imports
RemingtonRohel Sep 9, 2024
da6b06e
Updating C++ parsing of config file
RemingtonRohel Sep 9, 2024
4763ee8
Fixing bugs from lab testing.
RemingtonRohel Sep 9, 2024
3a506ce
Fixing bad dim_scale `lags`
RemingtonRohel Sep 9, 2024
f141659
Fix bug with trying to write missing optional scale fields
RemingtonRohel Sep 9, 2024
a9f9ac8
Another attempt to fix bug with trying to write missing optional scal…
RemingtonRohel Sep 9, 2024
a7aba6d
Fixing bug with attempting to compress scalar datasets.
RemingtonRohel Sep 9, 2024
0f66792
Continuing with bugfixes from testing on lab.
RemingtonRohel Sep 9, 2024
b5761b1
Fixed array comparison for bytes types
RemingtonRohel Sep 9, 2024
e422b2f
Update docs/ requirements.txt to include darn-dmap
RemingtonRohel Sep 9, 2024
d67997b
Corrected CFS fields with input from Draven
RemingtonRohel Sep 9, 2024
09075bf
Fixed bugs with writing bfiq files.
RemingtonRohel Sep 9, 2024
25fec2a
Updating documentation of file types in ReadTheDocs.
RemingtonRohel Sep 10, 2024
40c644e
Fixing bugs with loading record using xarray
RemingtonRohel Sep 11, 2024
ee57608
Added `sample_time` field that is a Dimension Scale for antennas_iq_d…
RemingtonRohel Sep 11, 2024
dcc0147
Created simulator to test software without N200s
RemingtonRohel Sep 11, 2024
dea7fc4
Bugfixes and testing with a `steamed_sham.py`
RemingtonRohel Sep 12, 2024
8c65992
Fixing lint errors in driver_sim.py
RemingtonRohel Sep 13, 2024
5f2b8f1
Adding fields to make array-structured representation compatible with…
RemingtonRohel Sep 13, 2024
4575290
Merge branch 'develop' into new_data_format
Doreban Sep 18, 2024
f3183dc
Merge branch 'new_data_format' of github.com:SuperDARNCanada/borealis…
RemingtonRohel Sep 23, 2024
0469e2d
Added docstring per code review.
RemingtonRohel Sep 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 32 additions & 4 deletions config/cly/cly_config.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,6 @@
"site_id" : "cly",
"gps_octoclock_addr" : "addr=192.168.10.131",
"device_options" : "recv_frame_size=4000",
"main_antenna_count" : "16",
"intf_antenna_count" : "4",
"n200s" : [
{
"addr" : "192.168.10.100",
Expand Down Expand Up @@ -108,8 +106,37 @@
"tx_channel_0" : ""
}
],
"main_antenna_spacing" : "15.24",
"intf_antenna_spacing" : "15.24",
"antennas" : {
"main_locations": {
"0" : [-114.3, 0.0, 0.0],
"1" : [-99.06, 0.0, 0.0],
"2" : [-83.82, 0.0, 0.0],
"3" : [-68.58, 0.0, 0.0],
"4" : [-53.34, 0.0, 0.0],
"5" : [-38.10, 0.0, 0.0],
"6" : [-22.86, 0.0, 0.0],
"7" : [-7.62, 0.0, 0.0],
"8" : [7.62, 0.0, 0.0],
"9" : [22.86, 0.0, 0.0],
"10" : [38.10, 0.0, 0.0],
"11" : [53.34, 0.0, 0.0],
"12" : [68.58, 0.0, 0.0],
"13" : [83.82, 0.0, 0.0],
"14" : [99.06, 0.0, 0.0],
"15" : [114.3, 0.0, 0.0]
},
"intf_locations": {
"0" : [-22.86, 100.0, 0.0],
"1" : [-7.62, 100.0, 0.0],
"2" : [7.62, 100.0, 0.0],
"3" : [22.86, 100.0, 0.0]
},
"main_antenna_count" : "16",
"intf_antenna_count" : "4",
"main_antenna_spacing": "15.24",
"intf_antenna_spacing": "15.24",
"standard_positions": true
},
"min_freq" : "8.0e6",
"max_freq" : "20.0e6",
"min_pulse_length" : "100",
Expand Down Expand Up @@ -146,6 +173,7 @@
"ringbuffer_name": "data_ringbuffer",
"ringbuffer_size_bytes" : "200e6",
"data_directory" : "/data/borealis_data",
"rawacf_format" : "dmap",
"log_handlers" : {
"console" : {
"enable" : true,
Expand Down
36 changes: 32 additions & 4 deletions config/inv/inv_config.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,6 @@
"site_id" : "inv",
"gps_octoclock_addr" : "addr=192.168.10.131",
"device_options" : "recv_frame_size=4000",
"main_antenna_count" : "16",
"intf_antenna_count" : "4",
"n200s" : [
{
"addr" : "192.168.10.100",
Expand Down Expand Up @@ -108,8 +106,37 @@
"tx_channel_0" : ""
}
],
"main_antenna_spacing" : "15.24",
"intf_antenna_spacing" : "15.24",
"antennas" : {
"main_locations": {
"0" : [-114.3, 0.0, 0.0],
"1" : [-99.06, 0.0, 0.0],
"2" : [-83.82, 0.0, 0.0],
"3" : [-68.58, 0.0, 0.0],
"4" : [-53.34, 0.0, 0.0],
"5" : [-38.10, 0.0, 0.0],
"6" : [-22.86, 0.0, 0.0],
"7" : [-7.62, 0.0, 0.0],
"8" : [7.62, 0.0, 0.0],
"9" : [22.86, 0.0, 0.0],
"10" : [38.10, 0.0, 0.0],
"11" : [53.34, 0.0, 0.0],
"12" : [68.58, 0.0, 0.0],
"13" : [83.82, 0.0, 0.0],
"14" : [99.06, 0.0, 0.0],
"15" : [114.3, 0.0, 0.0]
},
"intf_locations": {
"0" : [-21.36, 100.0, 0.0],
"1" : [-6.12, 100.0, 0.0],
"2" : [9.12, 100.0, 0.0],
"3" : [24.36, 100.0, 0.0]
},
"main_antenna_count" : "16",
"intf_antenna_count" : "4",
"main_antenna_spacing": "15.24",
"intf_antenna_spacing": "15.24",
"standard_positions": true
},
"min_freq" : "8.0e6",
"max_freq" : "20.0e6",
"min_pulse_length" : "100",
Expand Down Expand Up @@ -146,6 +173,7 @@
"ringbuffer_name": "data_ringbuffer",
"ringbuffer_size_bytes" : "200e6",
"data_directory" : "/data/borealis_data",
"rawacf_format" : "dmap",
"log_handlers" : {
"console" : {
"enable" : true,
Expand Down
36 changes: 32 additions & 4 deletions config/lab/lab_config.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,6 @@
"site_id" : "lab",
"gps_octoclock_addr" : "addr=192.168.10.3",
"device_options" : "recv_frame_size=4000",
"main_antenna_count" : "16",
"intf_antenna_count" : "4",
"n200s" : [
{
"addr" : "192.168.10.100",
Expand Down Expand Up @@ -108,8 +106,37 @@
"tx_channel_0" : ""
}
],
"main_antenna_spacing" : "15.24",
"intf_antenna_spacing" : "15.24",
"antennas" : {
"main_locations": {
"0" : [-114.3, 0.0, 0.0],
"1" : [-99.06, 0.0, 0.0],
"2" : [-83.82, 0.0, 0.0],
"3" : [-68.58, 0.0, 0.0],
"4" : [-53.34, 0.0, 0.0],
"5" : [-38.10, 0.0, 0.0],
"6" : [-22.86, 0.0, 0.0],
"7" : [-7.62, 0.0, 0.0],
"8" : [7.62, 0.0, 0.0],
"9" : [22.86, 0.0, 0.0],
"10" : [38.10, 0.0, 0.0],
"11" : [53.34, 0.0, 0.0],
"12" : [68.58, 0.0, 0.0],
"13" : [83.82, 0.0, 0.0],
"14" : [99.06, 0.0, 0.0],
"15" : [114.3, 0.0, 0.0]
},
"intf_locations": {
"0" : [-22.86, -100.0, 0.0],
"1" : [-7.62, -100.0, 0.0],
"2" : [7.62, -100.0, 0.0],
"3" : [22.86, -100.0, 0.0]
},
"main_antenna_count" : "16",
"intf_antenna_count" : "4",
"main_antenna_spacing": "15.24",
"intf_antenna_spacing": "15.24",
"standard_positions": true
},
"min_freq" : "8.0e6",
"max_freq" : "20.0e6",
"min_pulse_length" : "100",
Expand Down Expand Up @@ -146,6 +173,7 @@
"ringbuffer_name": "data_ringbuffer",
"ringbuffer_size_bytes" : "200e6",
"data_directory" : "/data/borealis_data",
"rawacf_format" : "hdf5",
"log_handlers" : {
"console" : {
"enable" : true,
Expand Down
36 changes: 32 additions & 4 deletions config/pgr/pgr_config.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,6 @@
"site_id" : "pgr",
"gps_octoclock_addr" : "addr=192.168.10.131",
"device_options" : "recv_frame_size=4000",
"main_antenna_count" : "16",
"intf_antenna_count" : "4",
"n200s" : [
{
"addr" : "192.168.10.100",
Expand Down Expand Up @@ -108,8 +106,37 @@
"tx_channel_0" : ""
}
],
"main_antenna_spacing" : "15.24",
"intf_antenna_spacing" : "15.24",
"antennas" : {
"main_locations": {
"0" : [-114.3, 0.0, 0.0],
"1" : [-99.06, 0.0, 0.0],
"2" : [-83.82, 0.0, 0.0],
"3" : [-68.58, 0.0, 0.0],
"4" : [-53.34, 0.0, 0.0],
"5" : [-38.10, 0.0, 0.0],
"6" : [-22.86, 0.0, 0.0],
"7" : [-7.62, 0.0, 0.0],
"8" : [7.62, 0.0, 0.0],
"9" : [22.86, 0.0, 0.0],
"10" : [38.10, 0.0, 0.0],
"11" : [53.34, 0.0, 0.0],
"12" : [68.58, 0.0, 0.0],
"13" : [83.82, 0.0, 0.0],
"14" : [99.06, 0.0, 0.0],
"15" : [114.3, 0.0, 0.0]
},
"intf_locations": {
"0" : [-22.86, -100.0, 0.0],
"1" : [-7.62, -100.0, 0.0],
"2" : [7.62, -100.0, 0.0],
"3" : [22.86, -100.0, 0.0]
},
"main_antenna_count" : "16",
"intf_antenna_count" : "4",
"main_antenna_spacing": "15.24",
"intf_antenna_spacing": "15.24",
"standard_positions": true
},
"min_freq" : "8.0e6",
"max_freq" : "20.0e6",
"min_pulse_length" : "100",
Expand Down Expand Up @@ -146,6 +173,7 @@
"ringbuffer_name": "data_ringbuffer",
"ringbuffer_size_bytes" : "200e6",
"data_directory" : "/data/borealis_data",
"rawacf_format" : "dmap",
"log_handlers" : {
"console" : {
"enable" : true,
Expand Down
36 changes: 32 additions & 4 deletions config/rkn/rkn_config.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,6 @@
"site_id" : "rkn",
"gps_octoclock_addr" : "addr=192.168.10.131",
"device_options" : "recv_frame_size=4000",
"main_antenna_count" : "16",
"intf_antenna_count" : "4",
"n200s" : [
{
"addr" : "192.168.10.100",
Expand Down Expand Up @@ -108,8 +106,37 @@
"tx_channel_0" : ""
}
],
"main_antenna_spacing" : "15.24",
"intf_antenna_spacing" : "15.24",
"antennas" : {
"main_locations": {
"0" : [-114.3, 0.0, 0.0],
"1" : [-99.06, 0.0, 0.0],
"2" : [-83.82, 0.0, 0.0],
"3" : [-68.58, 0.0, 0.0],
"4" : [-53.34, 0.0, 0.0],
"5" : [-38.10, 0.0, 0.0],
"6" : [-22.86, 0.0, 0.0],
"7" : [-7.62, 0.0, 0.0],
"8" : [7.62, 0.0, 0.0],
"9" : [22.86, 0.0, 0.0],
"10" : [38.10, 0.0, 0.0],
"11" : [53.34, 0.0, 0.0],
"12" : [68.58, 0.0, 0.0],
"13" : [83.82, 0.0, 0.0],
"14" : [99.06, 0.0, 0.0],
"15" : [114.3, 0.0, 0.0]
},
"intf_locations": {
"0" : [-22.86, -100.0, 0.0],
"1" : [-7.62, -100.0, 0.0],
"2" : [7.62, -100.0, 0.0],
"3" : [22.86, -100.0, 0.0]
},
"main_antenna_count" : "16",
"intf_antenna_count" : "4",
"main_antenna_spacing": "15.24",
"intf_antenna_spacing": "15.24",
"standard_positions": true
},
"min_freq" : "8.0e6",
"max_freq" : "20.0e6",
"min_pulse_length" : "100",
Expand Down Expand Up @@ -146,6 +173,7 @@
"ringbuffer_name": "data_ringbuffer",
"ringbuffer_size_bytes" : "200e6",
"data_directory" : "/data/borealis_data",
"rawacf_format" : "dmap",
"log_handlers" : {
"console" : {
"enable" : true,
Expand Down
36 changes: 32 additions & 4 deletions config/sas/sas_config.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,6 @@
"site_id" : "sas",
"gps_octoclock_addr" : "addr=192.168.10.131",
"device_options" : "recv_frame_size=4000",
"main_antenna_count" : "16",
"intf_antenna_count" : "4",
"n200s" : [
{
"addr" : "192.168.10.100",
Expand Down Expand Up @@ -108,8 +106,37 @@
"tx_channel_0" : ""
}
],
"main_antenna_spacing" : "15.24",
"intf_antenna_spacing" : "15.24",
"antennas" : {
"main_locations": {
"0" : [-114.3, 0.0, 0.0],
"1" : [-99.06, 0.0, 0.0],
"2" : [-83.82, 0.0, 0.0],
"3" : [-68.58, 0.0, 0.0],
"4" : [-53.34, 0.0, 0.0],
"5" : [-38.10, 0.0, 0.0],
"6" : [-22.86, 0.0, 0.0],
"7" : [-7.62, 0.0, 0.0],
"8" : [7.62, 0.0, 0.0],
"9" : [22.86, 0.0, 0.0],
"10" : [38.10, 0.0, 0.0],
"11" : [53.34, 0.0, 0.0],
"12" : [68.58, 0.0, 0.0],
"13" : [83.82, 0.0, 0.0],
"14" : [99.06, 0.0, 0.0],
"15" : [114.3, 0.0, 0.0]
},
"intf_locations": {
"0" : [-22.86, -100.0, 0.0],
"1" : [-7.62, -100.0, 0.0],
"2" : [7.62, -100.0, 0.0],
"3" : [22.86, -100.0, 0.0]
},
"main_antenna_count" : "16",
"intf_antenna_count" : "4",
"main_antenna_spacing": "15.24",
"intf_antenna_spacing": "15.24",
"standard_positions": true
},
"min_freq" : "8.0e6",
"max_freq" : "20.0e6",
"min_pulse_length" : "100",
Expand Down Expand Up @@ -146,6 +173,7 @@
"ringbuffer_name": "data_ringbuffer",
"ringbuffer_size_bytes" : "200e6",
"data_directory" : "/data/borealis_data",
"rawacf_format" : "dmap",
"log_handlers" : {
"console" : {
"enable" : true,
Expand Down
3 changes: 3 additions & 0 deletions docs/conf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,9 @@
# Update the experiment subrepo so experiment files can be read into documentation
run(["git", "submodule", "update", "--init"])

# Generate documentation on the Borealis file types
run(["python3", "file_docs_builder.py"])

on_rtd = os.environ.get("READTHEDOCS", None) == "True"
if on_rtd: # Only run following changes on ReadTheDocs
RADAR_ID = "sas"
Expand Down
Loading