Releases · angelolab/ark-analysis

Major Updates

Ark-Analysis now uses Python 3.7.
Added FlowSOM implementation and pipeline.

🚀 Features

remove external manual clustering @camisowers (#615)

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?

Closes #574.
Removes the example_manually_adjust_metaclusters notebook and the metaclustering folder from ark-analysis/data/example_dataset.

Run expensive per-FOV pixel clustering processes in parallel @alex-l-kong (#596)

What is the purpose of this PR?

Closes #532. There are many processes for pixel clustering that run on a one-per-FOV basis that can be optimized with parallelization.

How did you implement your changes

We target four functions:

create_pixel_matrix
run_pixel_som
pixel_consensus_cluster
apply_pixel_meta_cluster_remapping

1 and 4 are done in Python and rely on the multiprocessing module to generate separate processes in parallel. 2 and 3 are done in R, which provides a much cleaner solution using parallel, doParallel, and foreach.

Added Release Drafter @srivarra (#598)

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?

Closes #597.

Adds Release Drafter using Github Actions.

How did you implement your changes

Added:

workflows/release-drafter.yaml: Uses Github Actions Workflows to build the release notes.
release-drafter.yaml: Organizes the version resolver, and organizes labeled PRs into the following categories:
1. Features
2. Bug Fixes
3. Maintenance
4. Documentation

Remaining issues

We can add more categories as well using labels, and projects.

Remove intermediate `_clustered`/`_consensus` directories/files from pixel and cell clustering pipeline @alex-l-kong (#586)

What is the purpose of this PR?

Closes #571. There is no need to create a separate pixel_mat_clustered directory/cell_mat_clustered file as the data is just the channel expression/pixel cluster count expression values with an additional pixel_som_cluster/cell_som_cluster attached. Ditto for the pixel_mat_consensus directory/cell_mat_consensus file. These should be removed from the pixel and cell clustering processes.

How did you implement your changes

We should overwrite the original data with the newly created data in the cluster_pixels/pixel_consensus_cluster (and similarly for the cell clustering functions).

Remaining issues

There are a few extra checks we need to include for validation if the user decides to rerun the clustering pipeline in the same session.

Pix preprocessing @ngreenwald (#579)

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?
Adds in functionality to normalize each channel of image data data separately prior to pixel clustering. This helps to make sure that markers which have different intensity values are treated equally right from the beginning of the clustering process.

In addition, it changes from removing pixels that have 0 total counts to removing pixels in the bottom 5% of total counts from the image. This better matches the format of the data following rosetta, where there are very few true zeros.

Remaining issues
The testing I put together is very basic. @alex-l-kong, if you could go in and double check that everything is working as intended, and if needed adding more testing, that would be great. Also feel free to change the organization/saving structure if you think it would be better some other way. I didn't add any new tests for create_pixel_matrix, that will likely need to be checked as well.

Added `sub_dir` option for `save_fov_images` @srivarra (#580)

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?

Closes #533.

Allows fov_image masks to be saved in a separate subfolder instead of at the top level.

How did you implement your changes

Added a new subdirectory parameter in save_fov_images. Added another test as well.

Remaining issues

None atm.

Custom channel smoothing @ngreenwald (#581)

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?
Adds in functionality for different smoothing of individual channels in pixel clustering

How did you implement your changes
Provides the user a separate step to specify individual channels for increased smoothing, along with custom smoothing coefficients.

Also adds in an error check to create_pixel_matrix which identifies cases where the user provided the original channel name, but a modified channel name exists. This will be useful for other pre-processing steps we introduce which produce modified channels

Spatial lda @bcollica (#437)

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?

Creates the directories and some files for the spatial-LDA pipeline and gives an initial example of code for preprocessing steps.

How did you implement your changes

Added a function for formatting a cell table to be compatible with the existing spatial_lda library. Also added a function for checking valid arguments along with a test.

Remaining issues

Fixing code format to be pep8 compliant so builds can pass.

Update FlowSOM pipeline with updated `MapDataToCodes` optional flag in `SOM` function @alex-l-kong (#570)

What is the purpose of this PR?

Closes #568. This addresses the issue caused by the current functionality of SOM in the FlowSOM library which causes MapDataToCodes to run by default.

How did you implement your changes

We've created a fork of the main FlowSOM repo and added an optional flag map to the SOM function which allows us to prevent the call to MapDataToCodes. This parameter is also added to our SOM calls in create_pixel_som.R and create_cell_som.R.
For the time being, we add an explicit command to overwrite the default FlowSOM installation with our forked FlowSOM package. This will be removed whenever the devs of the main FlowSOM repo review the PR I've opened addressing the map parameter.

Remaining issues

It is possible there may be other memory-related issues we'll run into that will need further addressing.
It is possible the main FlowSOM devs do not approve our PR. In that case, we'll have to continue piggybacking on the forked FlowSOM repo for the foreseeable future. That might be a good idea considering we may want to make more optimizations to the C code and what not.

Append cell meta cluster labels to cell table @alex-l-kong (#567)

What is the purpose of this PR?

Closes #551 and closes #523. The cell table is a requirement for several downstream analysis scripts, not just pixel clustering. To ensure analysis can be performed based on the cell meta cluster labels, these will need to be appended to the cell table as well.

How did you implement your changes

A function add_consensus_labels_cell_table will run this appending process, with the call in example_cell_clustering.ipynb placed at the end of the notebook. The original cell_table_name will be overwritten with the same data including the 'cell_meta_cluster_rename' column in cell_consensus_name (simply 'cell_meta_cluster' in the cell table).

Users will be able to save multiple times and can also re-run the cell SOM training with the newly-saved cell table if desired.

Additionally, we now allow more flexible locations for the cell table. Initially, it was forced to be in base_dir. Now, we default the location to segmentation_dir, with options to change the root. Note that the updated cell table will be saved in the same root directory as the original cell table.

Delete references to Docker locally rebuilding in `README.md` @alex-l-kong (#560)

What is the purpose of this PR?

There shouldn't be any more references to Docker needing to rebuild locally now that we have Dockerhub integration.

How did you implement your changes

This necessitates the removal of the line indicating that Docker would rebuild with the -u or --update flag pass to start_docker.sh.

Speed up generation of pixel cluster masks overlay @alex-l-kong (#546)

What is the purpose of this PR?

Closes #535. generate_pixel_cluster_masks in data_utils.py runs very slowly and needs to be optimized.

How did you implement your changes

The main issue is in the way the mask for each FOV is currently indexed. Currently, each FOV image is accessed in its original 2D format. Due to ho...