Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rethinking external assets 2 #272

Merged
merged 24 commits into from
Oct 11, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
f188425
Implemented external asset structure
evalott100 Jun 15, 2023
463743d
Added Partial typeddicts for use with bluesky
evalott100 Jun 15, 2023
cd601d7
Added Partial typeddicts for use with bluesky
evalott100 Jun 15, 2023
2910dfe
Changed the event counters to increment by seq_num, not data. Changed…
evalott100 Jun 20, 2023
e5ec76c
Changed ComposeStreamResource to only have one ComposeDatum. Changed …
evalott100 Jul 3, 2023
78d95b4
Specifying pydantic<2 in pyproject.toml, also changed stream_resource…
evalott100 Jul 3, 2023
19e6744
Removed content from stream_datum, added a Range Typeddict
evalott100 Jul 3, 2023
6e8a865
Allowed external="STREAM:" keys to not be represented in data in Comp…
evalott100 Jul 4, 2023
959bc59
Fixed typo
evalott100 Sep 14, 2023
79daf00
Fixed imports to avoid pydantic if the version is >=2
evalott100 Jul 4, 2023
1ed861d
We changed != to == when checking datakeys
evalott100 Jul 12, 2023
e1e8065
Made corrections to changes, still need to add descriptor reference t…
evalott100 Jul 13, 2023
60848ee
Added a reference to event-descriptor uid in StreamDatum
evalott100 Jul 20, 2023
7f53948
Made run_start and descriptor required in Resource and StreamDatum re…
evalott100 Jul 20, 2023
f579225
removed block_idx from StreamDatum
evalott100 Aug 14, 2023
9ca7ca7
Made seq_nums optional
evalott100 Sep 11, 2023
a7e80da
Made data_keys a str data_key and moved it to stream resource
evalott100 Sep 14, 2023
0432aeb
Supported empty event pages
evalott100 Sep 28, 2023
6d3d8f2
Ran black (minor change)
evalott100 Sep 29, 2023
8a23344
Made stream_resource::run_start optional and fixed incorrect description
evalott100 Oct 3, 2023
ba9fbcf
Fix typo
coretl Oct 10, 2023
bd90c47
Merge remote-tracking branch 'eva/typo_correction' into rethinking_ex…
coretl Oct 10, 2023
a389141
Missed from the last merge
coretl Oct 10, 2023
e57e53b
Changed doc examples, made run_start NotRequired in resource/stream_r…
evalott100 Oct 11, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 10 additions & 39 deletions docs/user/explanations/data-model.rst
Original file line number Diff line number Diff line change
Expand Up @@ -477,31 +477,17 @@ See :doc:`external` for details on the role Stream Resource documents play in
referencing external assets that are natively ragged, such as single-photon detectors,
or assets where there are many relatively small data sets (e.g. scanned fluorescence data).

Minimal nontrivial valid example:

.. code-block:: python

# 'Stream Resource' document
{'path_semantics': 'posix',
'resource_kwargs': {},
'resource_path': '/local/path/subdirectory/data_file',
'root': '/local/path/',
'run_start': '10bf6945-4afd-43ca-af36-6ad8f3540bcd',
'spec': 'SOME_SPEC',
'stream_names': ['point_det'],
'uid': '272132cf-564f-428f-bf6b-149ee4287024'}

Typical example:

.. code-block:: python

# resource
{'spec': 'AD_HDF5',
# 'Stream Resource' document
{'data_key': 'detector_1',
'spec': 'AD_HDF5',
'root': '/GPFS/DATA/Andor/',
'resource_path': '2020/01/03/8ff08ff9-a2bf-48c3-8ff3-dcac0f309d7d.h5',
'resource_kwargs': {'frame_per_point': 1},
'path_semantics': 'posix',
'stream_names': ['point_det'],
'uid': '3b300e6f-b431-4750-a635-5630d15c81a8',
'run_start': '10bf6945-4afd-43ca-af36-6ad8f3540bcd'}

Expand All @@ -518,32 +504,17 @@ See :doc:`external` for details on the role Stream Datum documents play in refer
external assets that are natively ragged, such as single-photon detectors,
or assets where there are many relatively small data sets (e.g. scanned fluorescence data).

Minimal nontrivial valid example:

.. code-block:: python

# 'datum' document
{'resource': '272132cf-564f-428f-bf6b-149ee4287024', # foreign key
'datum_kwargs': {}, # format-specific parameters
'datum_id': '272132cf-564f-428f-bf6b-149ee4287024/1',
'block_idx': 0,
'event_count': 1
}

Typical example:

.. code-block:: python

# datum
{'resource': '3b300e6f-b431-4750-a635-5630d15c81a8',
'datum_kwargs': {'index': 3},
'datum_id': '3b300e6f-b431-4750-a635-5630d15c81a8/3',
'block_idx': 0,
'event_count': 5,
'event_offset': 14}

It is an implementation detail that ``datum_id`` is often formatted as
``{resource}/{counter}`` but this should not be considered part of the schema.
# 'Stream Datum' document
{'uid': '86340942-9865-47f9-9a8d-bdaaab1bfce2',
'descriptor': '8c70b8c2-df32-40e3-9f50-29cda8142fa0',
'stream_resource': '272132cf-564f-428f-bf6b-149ee4287024', # foreign key
'indices': {'start': 0, 'stop': 1},
'seq_nums': {'start': 1, 'stop': 2},
}

Formal schema:

Expand Down
Loading