Release Notes: v0.7.2 - 10/10/24

You may download the latest release with:

git clone -b v0.7.2 https://github.com/angelolab/ark-analysis.git

What's new?

Cell table and cell distances updates

Updates to marker_quantification.generate_cell_table() which allows for segmentation masks to have various filenames. Rather than requiring fov1_whole_cell.tiff, files can be either fov1.tiff or fov1-any_suffix.tiff. For spatial analysis, distance matrices are now generated using simply the cell centroids from the cell table, rather than requiring the segmentation masks and running regionprops to locate centroids.

Packaging updates

Spatial Enrichment has been removed along with it's Cython components. This allows Ark to be a pure python package reducing build times, and not requiring operating system specific compiled versions.

In addition as part of removing the custom Docker builds, we have removed Docker related information from the README.

🚀 Features

• Adjust mask suffix input to generate_cell_table @camisowers (#1164)
• Distance table generation without segmentation masks @camisowers (#1159)

🧰 Maintenance

• Removed Spatial Enrichment + Cython @srivarra (#1163)
• Remove anything Docker related from the README @alex-l-kong (#1158)

📚️ Documentation

• Update release version in readme @camisowers (#1166)

@alex-l-kong, @camisowers and @srivarra

Release Notes: v0.7.1 - 06/27/24

You may download the latest release with:

git clone -b v0.7.1 https://github.com/angelolab/ark-analysis.git

What's new?

Pixie updates

In certain datasets, it is impossible to assign all unique SOM clusters. This was previously prohibitive because error checks required all unique SOM clusters to have at least 1 pixel/cell assigned to it. With this update, this is no longer the case, and users may continue in spite of this.

To prevent users from generating duplicate cluster_id columns by accidentally running the mask generation function multiple times, a safeguard has been added.

An ongoing issue with visualizing unordered cluster IDs using the existing plotting functions has been resolved. Regardless of the spacing and ordering of the IDs internally, they are mapped to the correct color in the overlays.

Update the Mantis directory conversion process to turn off erosion and use an updated cluster_id column to generate correct and clearer overlays.

Lab members often have multiple resolutions of clustering analysis that need to be resolved. The post clustering notebook has been updated to allow users to define new cluster groups based on user-set mappings.

Spatial Analysis Updates

Our repo is gradually transitioning our internal single-cell data representation format to AnnData. We need a way to convert our existing format to be scverse compatible. The AnnData conversion notebook is the first step of this large-scale conversion process.

Prevent incorrect ordering of kmeans clusters by adding control params.

Correct a bug in fiber segmentation that causes metric averages for a fiber to be the same at the first metric value.

ezSegmenter Updates

merge_masks.py took a while to run on very long datasets with thousands of FOVs. This has been optimized by only merging cells that are in the areas surrounding objects.

Due to requests from lab members, we've added a generalized masking script to create masks from single or composite channel tiffs. This is implemented using the channel masking function from the TNBC repo.

Update image loading process to use np.int32 for compatibility of composite array addition and subtraction.

With the addition of various types of masks, it's important for the cell table generation notebook to differentiate only _whole_cell.tiff and _nuclear.tiff files if desired. Support for this has been added. Also prevents FOVs with duplicate substrings from breaking the cell table generation process by requiring specific names.

Optimize the generation of nuclear masks to prevent repetitive computation.

Deployment + Environment Updates

Our deployment stack requires enhanced security protocols, including OIDC and 2FA. These have been updated to allow for future releases.

Downgraded Python to 3.10 while waiting for scikit-image support for 3.11.

---

🚀 Features

• Generalized masking @camisowers (#1092)
• Don't require user to have all pixel SOM clusters represented during assignment @alex-l-kong (#1124)
• 1119 reducing time to merge in merge maskspy wi ez segmenter @bryjcannon (#1123)
• Add multiple clustering resolution functionality @camisowers (#1095)
• AnnData Conversion Notebook @srivarra (#1079)

🐛 Bug Fixes

• Fix kmeans notebook mask generation @camisowers (#1130)
• Manual mask names @camisowers (#1126)
• Fiber stats bugs @camisowers (#1128)
• Fix color masking in Pixie @camisowers (#1127)
• Changes process lists search for matching fov and mask names to use a… @bryjcannon (#1104)
• 1100 empty composites masks in ez segmenter notebook @bryjcannon (#1101)
• Don't generate duplicate cluster_id columns if generate_and_save_cluster_masks run twice @alex-l-kong (#1110)
• Only load whole cell masks if nuclear_counts=False in generate_cell_table @alex-l-kong (#1107)
• General CI Updates @srivarra (#1114)
• Fix cell cluster visualization in Mantis @camisowers (#1083)

🧰 Maintenance

• Change python version in conda env file @srivarra (#1134)
• Next Release 0.7.1 @srivarra (#1144)
• Updated GitHub Workflows @srivarra (#1139)

@alex-l-kong, @bryjcannon, @camisowers and @srivarra

Release Notes: v0.7.0 - 12/06/23

You may download the latest release with:

git clone -b v0.7.0 https://github.com/angelolab/ark-analysis.git

What's new?

ezSegmenter support

Starting with v0.7.0, support for ezSegmenter masks has been added. The Jupyter notebook for this can be found in templates/ez_segmenter.ipynb.

Please note that this feature may experience frequent updates. Updates may be required for Pixie integration and further downstream analysis.

Updated mask visualization

v0.7.0 introduces significant updates to the visualization process of masks. Shared functionality (ex. colormap generation, plotting clusters) has been abstracted away into internal functions and dataclasses. Erosion support has been added to decrease the size of the borders between cells, and interpolation has been turned off to prevent interpolation of cell borders.

In the past, users reported problems of incorrectly sorted colors in the cmap file. This has been addressed. Additionally, users may now specify a dpi and fig_size, as well as multi-compatibility support for saving matplotlib figures.

Mantis saving updates

Starting with v0.7.0, only a single top-level Mantis directory will be created, and support for subfolders (via the img_sub_folder param) has been added.

Pixie updates

During Pixie SOM assignment for pixel clustering, some users reported it taking unusually long. This was caused by the batch size for cluster assignment being too small. With this update, the batch size can now be set by the user to speed the process up, and defaults to a more reasonable value.

Additionally, in some Pixie applications, certain cells won't have any pixel clusters associated with them. These cells now get dropped prior to running the cell clustering portion.

Previously, the overwrite flag did not work for certain functions because it wasn't propagated correctly. This has also been addressed by v0.7.0.

Versioning Updates

ark-analysis now uses scikit-image==0.19.3 and alpineer==v0.1.10.

Licensing

ark-analysis has migrated from the Modified Apache 2.0 License to the MIT license. This was made due to the cell segmentation algorithm, Mesmer, having a more restrictive license.

---

🚀 Features

• ezSegmentation @srivarra (#958)
• Mask Generation Refactoring @srivarra (#1030)
• Include img_sub_folder as parameter in Mantis calls @alex-l-kong (#1050)
• Remove cells that don't have any pixel clusters expressed prior to Pixie @alex-l-kong (#1051)
• Single mantis directory @camisowers (#1061)
• Allow user to dynamically set batch size of pixel SOM assignments @alex-l-kong (#1069)

🐛 Bug Fixes

• Make sure overwrite functionality for pixel clustering propagates into helper functions @alex-l-kong (#1058)
• Adjustments to Mass Plotting Workflow @srivarra (#1067)

🧰 Maintenance

• Update README from v0.6.5 to v0.6.6 @alex-l-kong (#1048)
• Update HuggingFace to main @alex-l-kong (#1053)
• Pin to scikit-image to v0.19.3 @camisowers (#1060)
• Alpineer v0.1.10 @camisowers (#1057)
  @alex-l-kong, @camisowers and @srivarra
• MIT license @srivarra (#1080)
• Bump to v0.7.0 @alex-l-kong (#1093)

v0.6.6

Fixes a bug where the tables dependency was causing HDF5 incompatibility issues. This module is never used in our repo, so it has been removed as a dependency.

🐛 Bug Fixes

• Fix color_map setting in visualization notebook @camisowers (#1043)
  @alex-l-kong, @camisowers and @cliu72
• Release 0.6.6 without tables dependency @alex-l-kong (#1047)

🧰 Maintenance

• Update README with Pixie reference @cliu72 (#1042)

v0.6.5

🚀 Features

• Update progress bars @camisowers (#1031)
• Optional normalization for generic_cell_clustering.ipynb @camisowers (#1027)
• Deepcell upload loop cleaning @camisowers (#1023)
• Add channel normalization and pixel thresholding back @alex-l-kong (#985)

🐛 Bug Fixes

• Update progress bars @camisowers (#1031)
  @alex-l-kong and @camisowers

Release Notes: v0.6.4 - 07/24/23

You may download the latest release with:

git clone -b v0.6.4 https://github.com/angelolab/ark-analysis.git

What's new in Ark Analysis?

Python Compatibility

The minimum supported Python is now version 3.9.

We have an Anaconda environment.yml file so users do not need to use the Docker Image if they wish. This will create a Python 3.11 environment. Here is how to use it.

cd ark-analysis
conda env create -f environment.yml
conda activate ark_env

Pixie Improvements

Pixie now features improved fallback protection during the clustering step, preventing duplicate cluster names and the ability to save cluster masks as colored images.

K-Means Improvements

K-means now has improved plots, default parameters, and has received minor improvements.

Cell Neighbors Analysis

We have a new analysis pipeline containing two independent features: analyzing the homogeneity/diversity of the neighbors surrounding each of the cells and analyzing the proximity/distance between cell phenotypes in samples.

Fiber Segmentation

Fiber segmentation acquires more properties and has received optimizations regarding extracting distinct fiber objects.

---

🚀 Features

• Add progress bars in Cell Neighbors Analysis notebook @camisowers (#1022)
• Drop 3rd dimension in segmentation masks @camisowers (#1018)
• Mixing score default thresholds @camisowers (#1015)
• Include all fiber properties in stats generation @camisowers (#1014)
• Save Colored Pixie Masks @srivarra (#1007)
• Initial commit of phenotype by neighbour analysis @khoulahan (#988)
• Cell neighborhood diversity @camisowers (#989)
• Downstream fiber analysis updates @camisowers (#974)

🐛 Bug Fixes

• Maximize Available GitHub Storage Space @srivarra (#1025)
• Allow FOVs to contain various maximum clusters @srivarra (#1021)
• Pixie percentile saving @camisowers (#1020)
• Fix error from zero fibers in an image @camisowers (#1011)
• Fix segmentation rescale_factor @camisowers (#1008)
• Add fallback protection to Pixie cell clustering @alex-l-kong (#999)
• Prevent duplicate renamed metaclusters in Pixie remapping @alex-l-kong (#993)
• Pixel Clustering - Run without segmentaiton files @srivarra (#998)
• Mantis Directory with no segmentation files @srivarra (#994)
• Extended Attributes on macOS @srivarra (#973)
• Mutiple tiled stitching tweaks @camisowers (#981)
• Fix pixel and cell cluster mask generation index for rectangular images @alex-l-kong (#983)
• Kmeans plotting cell border fix @camisowers (#978)
• Set Kmeans parameter to prior default @camisowers (#977)
• Kmeans notebook minor fixes @camisowers (#966)
• Stitching Large Images @srivarra (#939)

🧰 Maintenance

• GitHub Actions, Python 3.9+ Support, Jupyterlab Extensions @srivarra (#991)
• Added Conda environment file @srivarra (#1006)
• Skip spatial LDA visualization tests due to Numpy error @alex-l-kong (#984)
  @alex-l-kong, @camisowers, @khoulahan and @srivarra

v0.6.3

• Update citation information @ngreenwald (#961)
  @ngreenwald

v0.6.2

🚀 Features

• Kmeans notebook cleanup @camisowers (#946)

🐛 Bug Fixes

• For docker.yml checkout the tag used in pypi_publish @srivarra (#948)
• Fix zero pixels @cliu72 (#956)

🧰 Maintenance

• Pinned ipywidgets @srivarra (#953)

📚️ Documentation

• Next Release - v0.6.2 @srivarra (#959)
  @camisowers, @cliu72 and @srivarra

v0.6.1

• Change os.rename to shutil.move @cliu72 (#951)
• Deprecate use of cluster, metacluster, and mc_name outside of remapping @alex-l-kong (#950)
  @alex-l-kong and @cliu72

v0.6.0

This latest release contains the revised Pixie pixel and cell clustering algorithm.

Pixie Paper Implementation

• Next Release - v0.6.0 @alex-l-kong (#945)
• Temporarily remove pixel preprocessing normalization for Candace's paper @alex-l-kong (#913)
• Pixie notebook text changes @cliu72 (#942)

🚀 Features

• Refactor Pixie modules @alex-l-kong (#940)
• Split spatial analysis module @camisowers (#938)
• Ark Refactor @srivarra (#935)
• Github Actions - Merge Queue @srivarra (#932)
• Sunset Pairwise Distance Notebook @srivarra (#929)
• Mixing score @camisowers (#836)
• Integrate existing Pixie cell clustering process with generic cell clustering process @alex-l-kong (#885)

🐛 Bug Fixes

• Add a hidden parameter to overwrite certain portions of the pipeline @alex-l-kong (#914)

🧰 Maintenance

• ⬆️ Bump release-drafter/release-drafter from 5.20.1 to 5.22.0 @dependabot (#916)

📚️ Documentation

• Make sure data_types.md displays newlines @alex-l-kong (#930)
  @alex-l-kong, @camisowers, @cliu72, @dependabot, @dependabot[bot] and @srivarra