Mass proficient extraction for NAS storage #342

alex-l-kong · 2023-04-06T23:15:07Z

What is the purpose of this PR?

Closes #327. Closes #322.

How did you implement your changes

See issue link.

Remaining issues

Need to test this on Christine's run next week.

…o prof_extract

review-notebook-app · 2023-04-12T18:25:54Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

…in case missing FOVs different for both)

…tebook

…ction in notebook 3b

…metric data

srivarra

LGTM!

camisowers · 2023-04-18T16:40:30Z

Agreed, looks good!

ngreenwald

Looks good. You should also confirm that the offsets are being handled correctly for mass extraction, since I remember some trickiness about the sign of the proficient and deficient extraction.

Should be straightforward to extract proficient, deficient, and the whole range, and make sure that proficient + deficient = combined

src/toffy/watcher_callbacks.py

templates/4b_normalize_image_data.ipynb

alex-l-kong · 2023-04-19T23:48:27Z

Looks good. You should also confirm that the offsets are being handled correctly for mass extraction, since I remember some trickiness about the sign of the proficient and deficient extraction.

Should be straightforward to extract proficient, deficient, and the whole range, and make sure that proficient + deficient = combined

Proficient + deficient unfortunately does not equal combined via the current offset. For most channels, the vast majority of values (about 95-99%) match up, but some channels (ex. Noodle, Fe, Au) have little in common between proficient + deficient and combined. Other channels like H3K27me3 and H3K9ac are also a bit more problematic (70-79%).

In terms of offset, I've been working with:

Combined from -0.3 to 0.3
Proficient from 0 to 0.3
Deficient from -0.3 to 0

…ls and alpineer

…range values

alex-l-kong · 2023-07-14T20:05:55Z

Need to test this PR on the next CIMAC/SPAIN runs before requesting a review.

alex-l-kong · 2023-08-15T22:45:32Z

Per our meeting a few weeks ago, we'll be reverting the definition of deficient and proficient back to:

Deficient: (-0.3, 0)
Proficient: (0, 0.3)

The long-term project would be for someone in the lab to granularly test each channel and see which ones need customized extraction ranges.

…more

ngreenwald

Looks good, see below

pyproject.toml

src/toffy/watcher_callbacks.py

srivarra

Looks good. Just a comment about the setting the ranges.

src/toffy/watcher_callbacks.py

srivarra

Wild stuff!

Initial commit of mass proficient extraction

4690d25

alex-l-kong self-assigned this Apr 6, 2023

alex-l-kong marked this pull request as draft April 6, 2023 23:15

alex-l-kong and others added 3 commits April 10, 2023 13:10

Merge branch 'main' into prof_extract

b6f7807

Add mass proficient MPH extraction

f9e7bb2

Merge branch 'prof_extract' of https://github.com/angelolab/toffy int…

7ed4c08

…o prof_extract

alex-l-kong and others added 12 commits April 12, 2023 16:35

Ensure bin_extraction also handles proficient metrics extraction

203b04e

Handle mass proficient and deficient bin file extraction separately (…

8f45246

…in case missing FOVs different for both)

Add mass proficient MPH normalization generation to 4b notebook

72159df

OCD comment change

cdde418

Ensure mass proficient panel passed to proficient extraction in 3b no…

099a317

…tebook

Typo fix

5318005

Ensure mass proficient directory actually gets created prior to extra…

106cfe6

…ction in notebook 3b

Make comment about mass proficient extraction more clear

00b9439

Ensure pulse height generation doesn't attempt to read in proficient …

7acdee2

…metric data

Merge branch 'main' into prof_extract

91cb986

Remove duplicate intermediate callback

865fda7

Typo in FOV callbacks

c2dd4df

alex-l-kong requested review from srivarra and camisowers April 18, 2023 01:19

alex-l-kong marked this pull request as ready for review April 18, 2023 01:19

srivarra approved these changes Apr 18, 2023

View reviewed changes

alex-l-kong requested a review from ngreenwald April 18, 2023 17:28

ngreenwald requested changes Apr 18, 2023

View reviewed changes

src/toffy/watcher_callbacks.py Outdated Show resolved Hide resolved

ngreenwald reviewed Apr 18, 2023

View reviewed changes

templates/4b_normalize_image_data.ipynb Show resolved Hide resolved

alex-l-kong added 2 commits April 19, 2023 14:15

Add flag to control proficient extraction or not

8f02717

Nuke proficient MPH extraction

58b5b9c

alex-l-kong added 3 commits July 14, 2023 11:38

Update pyproject.toml dependencies to be compatible with mibi-bin-too…

d8615a4

…ls and alpineer

Standardize the deficient and proficient ranges based on agreed upon …

0666c2d

…range values

Delete timeit print statements for MPH

a53f981

alex-l-kong and others added 12 commits July 14, 2023 13:39

Merge branch 'main' into prof_extract

0d475c0

Begin reducing test cases

4c54cf0

Merge remote-tracking branch 'origin/prof_extract' into prof_extract

0ecec00

Document the start and stop offset params

b24f2b7

Merge branch 'main' into prof_extract

5128dcf

Update poetry lock file to ensure this actually installs

62327f7

Clarify comment about extraction ranges

4a95055

Fix another extraction window comment

26a2f8a

Merge branch 'main' into prof_extract

c54f40b

Make sure extract_prof is included as a param to watcher tests

156f79f

Merge branch 'main' into prof_extract

4fbc76b

Add extract_prof param to inter_callback test

e6f70cb

Revert back to old deficient and proficient extraction ranges

91c7d9e

alex-l-kong requested a review from ngreenwald August 21, 2023 18:41

Pulse heights are not extracted for deficient and proficient data any…

496566e

…more

ngreenwald requested changes Aug 21, 2023

View reviewed changes

pyproject.toml Show resolved Hide resolved

src/toffy/watcher_callbacks.py Show resolved Hide resolved

ngreenwald requested a review from srivarra August 21, 2023 19:09

srivarra requested changes Aug 21, 2023

View reviewed changes

src/toffy/watcher_callbacks.py Show resolved Hide resolved

Clarify comment

56d18cb

alex-l-kong requested a review from srivarra August 22, 2023 00:24

srivarra approved these changes Aug 22, 2023

View reviewed changes

camisowers approved these changes Aug 22, 2023

View reviewed changes

alex-l-kong requested a review from ngreenwald August 22, 2023 18:18

ngreenwald merged commit 49d26ab into main Aug 22, 2023
4 checks passed

ngreenwald deleted the prof_extract branch August 22, 2023 19:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mass proficient extraction for NAS storage #342

Mass proficient extraction for NAS storage #342

alex-l-kong commented Apr 6, 2023 •

edited

Loading

review-notebook-app bot commented Apr 12, 2023

srivarra left a comment

camisowers commented Apr 18, 2023

ngreenwald left a comment

alex-l-kong commented Apr 19, 2023

alex-l-kong commented Jul 14, 2023 •

edited

Loading

alex-l-kong commented Aug 15, 2023 •

edited

Loading

ngreenwald left a comment

srivarra left a comment

srivarra left a comment

Mass proficient extraction for NAS storage #342

Mass proficient extraction for NAS storage #342

Conversation

alex-l-kong commented Apr 6, 2023 • edited Loading

review-notebook-app bot commented Apr 12, 2023

srivarra left a comment

Choose a reason for hiding this comment

camisowers commented Apr 18, 2023

ngreenwald left a comment

Choose a reason for hiding this comment

alex-l-kong commented Apr 19, 2023

alex-l-kong commented Jul 14, 2023 • edited Loading

alex-l-kong commented Aug 15, 2023 • edited Loading

ngreenwald left a comment

Choose a reason for hiding this comment

srivarra left a comment

Choose a reason for hiding this comment

srivarra left a comment

Choose a reason for hiding this comment

alex-l-kong commented Apr 6, 2023 •

edited

Loading

alex-l-kong commented Jul 14, 2023 •

edited

Loading

alex-l-kong commented Aug 15, 2023 •

edited

Loading