Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement II] Compatible processed types #17

Merged
merged 43 commits into from
Jul 19, 2024
Merged
Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
97bfa20
add multichannel volume
CodyCBakerPhD Jun 24, 2024
91db41b
swap to datainterface
CodyCBakerPhD Jun 24, 2024
200eb9b
fix import
CodyCBakerPhD Jun 24, 2024
b84aa0b
fix test name
CodyCBakerPhD Jun 24, 2024
c27778f
Merge branch 'avoid_name_collision' into both_prs_together
CodyCBakerPhD Jun 26, 2024
c8b5dfd
fix intercompatability
CodyCBakerPhD Jun 26, 2024
ad6b5e7
fix light sources
CodyCBakerPhD Jun 26, 2024
8b69db4
fix
CodyCBakerPhD Jun 26, 2024
00d2270
add plane segmentation stuff
CodyCBakerPhD Jun 26, 2024
c6b529c
fix autogenerated arg
CodyCBakerPhD Jun 26, 2024
22836ed
variable depth volume
CodyCBakerPhD Jun 26, 2024
13be731
Merge branch 'main' into add_multichannel_volume
CodyCBakerPhD Jun 27, 2024
da5a1b2
Merge branch 'add_multichannel_volume' into both_prs_together
CodyCBakerPhD Jun 27, 2024
f9c93a2
Merge pull request #16 from catalystneuro/both_prs_together
CodyCBakerPhD Jun 27, 2024
a00fa42
Merge branch 'add_multichannel_volume' into all_previous_prs_and_vari…
CodyCBakerPhD Jun 27, 2024
0423071
ryans suggestion
CodyCBakerPhD Jul 11, 2024
dd9ecb5
adjust to use full list
CodyCBakerPhD Jul 11, 2024
dbdad41
adjust constructor test
CodyCBakerPhD Jul 11, 2024
963e894
reorder kwargs in mock
CodyCBakerPhD Jul 11, 2024
0962561
adjust kwargs order in mock
CodyCBakerPhD Jul 11, 2024
7a7ae66
Implement lists of object references with tests
Jul 11, 2024
4a6444a
Adjust constructor test to match
Jul 11, 2024
4912507
fix outer spec to match altered one
Jul 11, 2024
523d972
Merge branch 'add_multichannel_volume' into all_previous_prs_and_vari…
CodyCBakerPhD Jul 12, 2024
5b645de
Update spec/ndx-microscopy.extensions.yaml
CodyCBakerPhD Jul 12, 2024
eb9ee22
Update spec/ndx-microscopy.extensions.yaml
CodyCBakerPhD Jul 12, 2024
5f12ff7
resolve conflicts
CodyCBakerPhD Jul 17, 2024
bb380a4
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 17, 2024
94ee228
alessandras comments and tests
CodyCBakerPhD Jul 17, 2024
0501249
Merge branch 'all_previous_prs_and_variable_depth_static' of https://…
CodyCBakerPhD Jul 17, 2024
e583e39
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 17, 2024
b581087
fix import
CodyCBakerPhD Jul 17, 2024
4a5e9c5
Merge branch 'all_previous_prs_and_variable_depth_static' of https://…
CodyCBakerPhD Jul 17, 2024
c4e8f3a
fix import
CodyCBakerPhD Jul 17, 2024
93a72a8
fix import
CodyCBakerPhD Jul 17, 2024
a300f9d
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 17, 2024
335c81d
debug
CodyCBakerPhD Jul 17, 2024
70fa119
debugs
CodyCBakerPhD Jul 17, 2024
5ac7eef
debugs
CodyCBakerPhD Jul 17, 2024
7effd8e
debugs
CodyCBakerPhD Jul 17, 2024
1d6fede
debugs
CodyCBakerPhD Jul 17, 2024
1d68a4a
debugs
CodyCBakerPhD Jul 17, 2024
5400462
debugs
CodyCBakerPhD Jul 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
277 changes: 254 additions & 23 deletions spec/ndx-microscopy.extensions.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ groups:
doc: General estimate of location in the brain being subset by this space.
Specify the area, layer, etc.
Use standard atlas names for anatomical regions when possible.
Specify 'whole-brain' if the entire brain is strictly contained within the space.
Specify 'whole brain' if the entire brain is strictly contained within the space.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also sneaking this change in

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From @alessandratrapani, has not seen the version with the dash, but intuitively wanted 'Whole Brain'

required: false
links:
- name: microscope
Expand All @@ -106,22 +106,14 @@ groups:
neurodata_type_inc: ImagingSpace
doc: Metadata about the 2-dimensional slice of physical space that imaging data was recorded from.
datasets:
- name: grid_spacing
- name: grid_spacing_in_um
alessandratrapani marked this conversation as resolved.
Show resolved Hide resolved
dtype: float64
dims:
- - x, y
shape:
- - 2
doc: Amount of space between pixels in the specified unit.
Specify 'z' only when imaging volume is a regular grid; otherwise only specify 'x' and 'y'.
See origin_coordinates to determine where the grid begins.
doc: Amount of space between pixels in micrometers.
quantity: '?'
# TODO: deal with grid_spacing units
# attributes:
# - name: unit
# dtype: text
# default_value: micrometers
# doc: Measurement units for grid spacing. The default value is 'micrometers'.
attributes:
- name: reference_frame
dtype: text
Expand All @@ -146,22 +138,14 @@ groups:
neurodata_type_inc: ImagingSpace
doc: Metadata about the 3-dimensional region of physical space that imaging data was recorded from.
datasets:
- name: grid_spacing
doc: Amount of space between pixels in (x, y) or voxels in (x, y, z), in the specified unit.
Specify 'z' only when imaging volume is a regular grid; otherwise only specify 'x' and 'y'.
See origin_coordinates to determine where the grid begins.
- name: grid_spacing_in_um
doc: Amount of space between voxels in micrometers.
dtype: float64
dims:
- - x, y, z
shape:
- - 3
quantity: '?'
# TODO: deal with grid_spacing units
# attributes:
# - name: unit
# dtype: text
# default_value: micrometers
# doc: Measurement units for grid spacing. The default value is 'micrometers'.
attributes:
- name: reference_frame
doc: Describes the reference frame of origin_coordinates and grid_spacing.
Expand All @@ -182,6 +166,106 @@ groups:
dtype: text
required: false


# These are needed to allow linkage of processed data to the new objects, until this is merged to core
# Technically the RoiResponseSeries shouldn't need to be modified since it just takes a DynamicTableRegion and
# does not care about the target
- neurodata_type_def: MicroscopyImageSegmentation
CodyCBakerPhD marked this conversation as resolved.
Show resolved Hide resolved
neurodata_type_inc: NWBDataInterface
default_name: MicroscopyImageSegmentation
doc: Stores pixels in an image that represent different regions of interest (ROIs)
or masks. All segmentation for a given imaging plane is stored together, with
storage for multiple imaging planes (masks) supported. Each ROI is stored in its
own subgroup, with the ROI group containing both a 2D mask and a list of pixels
that make up this mask. Segments can also be used for masking neuropil. If segmentation
is allowed to change with time, a new imaging plane (or module) is required and
ROI names should remain consistent between them.
groups:
- neurodata_type_inc: MicroscopyPlaneSegmentation
doc: Results from image segmentation of a specific imaging plane.
quantity: '+'
CodyCBakerPhD marked this conversation as resolved.
Show resolved Hide resolved


- neurodata_type_def: MicroscopyPlaneSegmentation
neurodata_type_inc: DynamicTable
doc: Results from image segmentation of a specific imaging plane.
datasets:
- name: image_mask
neurodata_type_inc: VectorData
dims:
- - num_roi
- num_x
- num_y
- - num_roi
- num_x
- num_y
- num_z
shape:
- - null
- null
- null
- - null
- null
- null
- null
doc: ROI masks for each ROI. Each image mask is the size of the original imaging
plane (or volume) and members of the ROI are finite non-zero.
quantity: '?'
- name: pixel_mask_index
neurodata_type_inc: VectorIndex
doc: Index into pixel_mask.
quantity: '?'
- name: pixel_mask
neurodata_type_inc: VectorData
dtype:
- name: x
dtype: uint32
doc: Pixel x-coordinate.
- name: y
dtype: uint32
doc: Pixel y-coordinate.
- name: weight
dtype: float32
doc: Weight of the pixel.
doc: 'Pixel masks for each ROI: a list of indices and weights for the ROI. Pixel
masks are concatenated and parsing of this dataset is maintained by the PlaneSegmentation'
quantity: '?'
- name: voxel_mask_index
neurodata_type_inc: VectorIndex
doc: Index into voxel_mask.
quantity: '?'
- name: voxel_mask
neurodata_type_inc: VectorData
dtype:
- name: x
dtype: uint32
doc: Voxel x-coordinate.
- name: y
dtype: uint32
doc: Voxel y-coordinate.
- name: z
dtype: uint32
doc: Voxel z-coordinate.
- name: weight
dtype: float32
doc: Weight of the voxel.
doc: 'Voxel masks for each ROI: a list of indices and weights for the ROI. Voxel
masks are concatenated and parsing of this dataset is maintained by the PlaneSegmentation'
quantity: '?'
groups:
- name: reference_images
doc: Image stacks that the segmentation masks apply to.
groups:
- neurodata_type_inc: ImageSeries
doc: One or more image stacks that the masks apply to (can be one-element
stack).
quantity: '*'
links:
- name: imaging_space
target_type: ImagingSpace
doc: Link to ImagingSpace object from which this data was generated.


- neurodata_type_def: MicroscopySeries
neurodata_type_inc: TimeSeries
doc: Imaging data acquired over time from an optical channel in a microscope while a light source illuminates the
Expand Down Expand Up @@ -227,7 +311,7 @@ groups:
datasets:
- name: depth_per_frame_in_mm
doc: Depth of each frame in the data array.
These values offset the 'z' value of the origin_coordinates of the linked imaging_space object.
These values offset the 'z' value of the `origin_coordinates` of the linked `imaging_space` object.
dtype: numeric
dims:
- frames
Expand All @@ -247,13 +331,160 @@ groups:
- frames
- height
- width
- depth
- depths
shape:
- null
- null
- null
- null
links:
- name: imaging_space
doc: Link to VolumetricImagingSpace object containing metadata about the region of physical space this imaging data
was recorded from.
target_type: VolumetricImagingSpace


- neurodata_type_def: MultiChannelMicroscopyVolume
neurodata_type_inc: NWBDataInterface
doc: Static (not time-varying) volumetric imaging data acquired from multiple optical channels.
attributes:
- name: description
dtype: text
doc: Description of the MultiChannelVolume.
required: false
- name: unit
dtype: text
doc: Base unit of measurement for working with the data. Actual stored values are
not necessarily stored in these units. To access the data in these units,
multiply 'data' by 'conversion' and add 'offset'.
- name: conversion
dtype: float32
default_value: 1.0
doc: Scalar to multiply each element in data to convert it to the specified 'unit'.
If the data are stored in acquisition system units or other units
that require a conversion to be interpretable, multiply the data by 'conversion'
to convert the data to the specified 'unit'. e.g. if the data acquisition system
stores values in this object as signed 16-bit integers (int16 range
-32,768 to 32,767) that correspond to a 5V range (-2.5V to 2.5V), and the data
acquisition system gain is 8000X, then the 'conversion' multiplier to get from
raw data acquisition values to recorded volts is 2.5/32768/8000 = 9.5367e-9.
required: false
- name: offset
dtype: float32
default_value: 0.0
doc: Scalar to add to the data after scaling by 'conversion' to finalize its coercion
to the specified 'unit'. Two common examples of this include (a) data stored in an
unsigned type that requires a shift after scaling to re-center the data,
and (b) specialized recording devices that naturally cause a scalar offset with
respect to the true units.
required: false
datasets:
- name: data
doc: Recorded imaging data, shaped by (frame height, frame width, number of depth planes, number of optical
channels).
dtype: numeric
dims:
- height
- width
- depths
- channels
shape:
- null
- null
- null
- null
- name: light_sources
doc: Reference to a list of MicroscopyLightSource objects containing metadata about the excitation methods.
neurodata_type_inc: VectorData
dtype:
reftype: object
target_type: MicroscopyLightSource
- name: optical_channels
doc: Reference to an ordered list of MicroscopyOpticalChannel objects containing metadata about the indicator and filters used to collect this data.
neurodata_type_inc: VectorData
dtype:
reftype: object
target_type: MicroscopyOpticalChannel
links:
- name: microscope
doc: Link to a Microscope object containing metadata about the device used to acquire this imaging data.
target_type: Microscope
- name: imaging_space
doc: Link to VolumetricImagingSpace object containing metadata about the region of physical space this imaging data
was recorded from.
target_type: VolumetricImagingSpace


- neurodata_type_def: VariableDepthMultiChannelMicroscopyVolume
CodyCBakerPhD marked this conversation as resolved.
Show resolved Hide resolved
neurodata_type_inc: NWBDataInterface
doc: Static (not time-varying) irregularly spaced volumetric imaging data acquired from multiple optical channels.
attributes:
- name: description
dtype: text
doc: Description of the VariableDepthMultiChannelMicroscopyVolume.
required: false
- name: unit
dtype: text
doc: Base unit of measurement for working with the data. Actual stored values are
not necessarily stored in these units. To access the data in these units,
multiply 'data' by 'conversion' and add 'offset'.
- name: conversion
dtype: float32
default_value: 1.0
doc: Scalar to multiply each element in data to convert it to the specified 'unit'.
If the data are stored in acquisition system units or other units
that require a conversion to be interpretable, multiply the data by 'conversion'
to convert the data to the specified 'unit'. e.g. if the data acquisition system
stores values in this object as signed 16-bit integers (int16 range
-32,768 to 32,767) that correspond to a 5V range (-2.5V to 2.5V), and the data
acquisition system gain is 8000X, then the 'conversion' multiplier to get from
raw data acquisition values to recorded volts is 2.5/32768/8000 = 9.5367e-9.
required: false
- name: offset
dtype: float32
default_value: 0.0
doc: Scalar to add to the data after scaling by 'conversion' to finalize its coercion
to the specified 'unit'. Two common examples of this include (a) data stored in an
unsigned type that requires a shift after scaling to re-center the data,
and (b) specialized recording devices that naturally cause a scalar offset with
respect to the true units.
required: false
datasets:
- name: data
doc: Recorded imaging data, shaped by (frame height, frame width, number of depth planes, number of optical
channels).
dtype: numeric
dims:
- height
- width
- depths
- channels
shape:
- null
- null
- null
- null
- name: depth_per_frame_in_um
CodyCBakerPhD marked this conversation as resolved.
Show resolved Hide resolved
doc: Depth of each frame in the data array.
These values offset the 'z' value of the `origin_coordinates` of the linked `imaging_space` object.
dtype: numeric
dims:
- depths
shape:
- null
links:
- name: microscope
doc: Link to a Microscope object containing metadata about the device used to acquire this imaging data.
target_type: Microscope
# TODO: figure out best way to link to list of light sources
- name: light_sources
doc: Link to an ordered list of MicroscopyLightSource objects containing metadata about the excitation method.
target_type: MicroscopyLightSource
# TODO: figure out best way to link to list of optical channels
- name: optical_channels
doc: Link to an ordered list of MicroscopyOpticalChannel objects containing metadata about the indicator and
filters used to collect this data.
target_type: MicroscopyOpticalChannel
- name: imaging_space
doc: Link to VolumetricImagingSpace object containing metadata about the region of physical space this imaging data
was recorded from.
Expand Down
8 changes: 8 additions & 0 deletions src/pynwb/ndx_microscopy/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,14 @@
ImagingSpace = get_class("ImagingSpace", extension_name)
PlanarImagingSpace = get_class("PlanarImagingSpace", extension_name)
VolumetricImagingSpace = get_class("VolumetricImagingSpace", extension_name)
MicroscopyImageSegmentation = get_class("MicroscopyImageSegmentation", extension_name)
MicroscopyPlaneSegmentation = get_class("MicroscopyPlaneSegmentation", extension_name)
MicroscopySeries = get_class("MicroscopySeries", extension_name)
PlanarMicroscopySeries = get_class("PlanarMicroscopySeries", extension_name)
VariableDepthMicroscopySeries = get_class("VariableDepthMicroscopySeries", extension_name)
VolumetricMicroscopySeries = get_class("VolumetricMicroscopySeries", extension_name)
MultiChannelMicroscopyVolume = get_class("MultiChannelMicroscopyVolume", extension_name)
VariableDepthMultiChannelMicroscopyVolume = get_class("VariableDepthMultiChannelMicroscopyVolume", extension_name)

__all__ = [
"Microscope",
Expand All @@ -38,8 +42,12 @@
"ImagingSpace",
"PlanarImagingSpace",
"VolumetricImagingSpace",
"MicroscopyImageSegmentation",
"MicroscopyPlaneSegmentation",
"MicroscopySeries",
"PlanarMicroscopySeries",
"VariableDepthMicroscopySeries",
"VolumetricMicroscopySeries",
"MultiChannelMicroscopyVolume",
"VariableDepthMultiChannelMicroscopyVolume",
]
8 changes: 8 additions & 0 deletions src/pynwb/ndx_microscopy/testing/__init__.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,14 @@
from ._mock import (
mock_Microscope,
mock_MicroscopyImageSegmentation,
mock_MicroscopyLightSource,
mock_MicroscopyOpticalChannel,
mock_MicroscopyPlaneSegmentation,
mock_MultiChannelMicroscopyVolume,
mock_PlanarImagingSpace,
mock_PlanarMicroscopySeries,
mock_VariableDepthMicroscopySeries,
mock_VariableDepthMultiChannelMicroscopyVolume,
mock_VolumetricImagingSpace,
mock_VolumetricMicroscopySeries,
)
Expand All @@ -15,7 +19,11 @@
"mock_MicroscopyOpticalChannel",
"mock_PlanarImagingSpace",
"mock_VolumetricImagingSpace",
"mock_MicroscopyImageSegmentation",
"mock_MicroscopyPlaneSegmentation",
"mock_PlanarMicroscopySeries",
"mock_VariableDepthMicroscopySeries",
"mock_VolumetricMicroscopySeries",
"mock_MultiChannelMicroscopyVolume",
"mock_VariableDepthMultiChannelMicroscopyVolume",
]
Loading
Loading