Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add image_diff comparison method for test output verification using images #17556

Merged
merged 40 commits into from
Mar 4, 2024

Conversation

kostrykin
Copy link
Contributor

@kostrykin kostrykin commented Feb 27, 2024

As part of the Imaging Hackathon in Freiburg.

This PR adds the image_diff comparison method for test output verification using images.

Previous comparison methods comprise diff, re_match, lines_diff, re_match_multiline, contains, and sim_size, all of which are either unsuitable for image data, or dangerous to be used (because the size of a file does not really represent its content). This PR adds the image_diff comparison method, which is specifically tailored for single-channel and multi-channel image data (e.g., RGB), and can be used like this:

<test>
    <param name="in" value="actual.png" />
    <output name="out" value="expected.tif" compare="image_diff" metric="mae" />
</test>

The difference of a pair of images is quantified as the pixel-wise distance between the images, for which different metrics can be used. For different applications and different types of images (e.g., intensity vs. label images), different types of metrics are common. The following metrics are added for intensity images:

metric Corresponding metric (for intensity images)
mae Mean absolute error
mse Mean squared error
rms Root mean squared error
fro Frobenius norm

For label images (label maps), it is usually irrelevant which particular label is used for labeling an image region, but only the sets of pixels corresponding to individual image regions need to be compared. This can be accomplished using one minus the intersection over the union of the regions:

metric Corresponding metric (for label images)
iou One minus the intersection over the union

To decide whether a pair of images is equal, a tolerance is required, because different compression algorithms (or different versions thereof) can lead to different compression artifacts. A pair of images is considered to be equal in terms of the specified metric, if the distance of the images computed with respect to the metric is not above a threshold eps (0.01 by default).

How to test the changes?

(Select all options that apply)

  • I've included appropriate automated tests.
  • This is a refactoring of components with existing test coverage.
  • Instructions for manual testing are as follows:
    1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

  • I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

@nsoranzo
Copy link
Member

@kostrykin I've added the changes to the package requirements, and also some type annotations if you'd like to double check them.

@jmchilton Are you OK with numpy and pillow being added as requirements to the galaxy-tool-util package?
If yes, I can also move the functions that currently use pillow conditionally (i.e. those for which I've added type annotations here) from lib/galaxy/util/ to tool_util/ .

@jmchilton
Copy link
Member

This is really, really cool stuff - sorry about being a bit picky about the packaging stuff.

@kostrykin
Copy link
Contributor Author

@kostrykin I've added the changes to the package requirements, and also some type annotations if you'd like to double check them.

Thanks, the type annotations look right to me!

@kostrykin kostrykin marked this pull request as ready for review March 1, 2024 08:19
@github-actions github-actions bot added this to the 24.1 milestone Mar 1, 2024
together pysam, which is also conditionally imported in
``lib/galaxy/tool_util/verify/__init__.py`` .
@nsoranzo
Copy link
Member

nsoranzo commented Mar 2, 2024

@jmchilton I've added the extra as you suggested, thanks!

@nsoranzo nsoranzo merged commit 3f0f789 into galaxyproject:dev Mar 4, 2024
54 checks passed
@nsoranzo
Copy link
Member

nsoranzo commented Mar 4, 2024

Thanks @kostrykin !

@nsoranzo nsoranzo added kind/enhancement highlight/dev Included in admin/dev release notes labels Mar 4, 2024
@jdavcs jdavcs modified the milestones: 24.1, 24.0 Mar 4, 2024
@kostrykin kostrykin deleted the files_diff_images branch March 5, 2024 15:49
@galaxyproject galaxyproject deleted a comment from github-actions bot Mar 13, 2024
kostrykin pushed a commit to kostrykin/galaxy that referenced this pull request Mar 18, 2024
Add `image_diff` comparison method for test output verification using images
kostrykin added a commit to kostrykin/galaxy-image-analysis that referenced this pull request Mar 18, 2024
kostrykin added a commit to BMCV/galaxy-image-analysis that referenced this pull request Mar 18, 2024
* Update pr.yaml

* Update

* Update pr.yaml

* Add `pip install pillow`

* Temporarilly fix CI

* Update pr.yaml

* Update ci.yaml

* Add test tool from `/test/functional/tools/image_diff.xml`

* Update pr.yaml

* Add .shed.yml

* Add missing test files

* Switch CI to galaxy fork `kostrykin/galaxy` branch `galaxy-image-analysis`

This is a temporary change to enable the features from:

 - galaxyproject/galaxy#17556
 - galaxyproject/galaxy#17581

* Remove `test` tool
kostrykin added a commit to kostrykin/galaxy-image-analysis that referenced this pull request Mar 21, 2024
* Rename tools

Change the `Performs` in the name of the two tools

- `Performs projective transformation with/without labels`
- `Performs projective transformation`

to `Perform`, which is more consistent with the name of the other tools (imperative form).

* Update projective_transformation.xml

* Update projective_transformation_points.xml

* Add CONVENTIONS.md

* wip

* wip

* Update CONVENTIONS.md

* Update CONVENTIONS.md

* Update CONVENTIONS.md

* Update CONVENTIONS.md

* Update CONVENTIONS.md

* Update CONVENTIONS.md

* Remove help section

* Update CONVENTIONS.md

* Update CONVENTIONS.md

* Update CONVENTIONS.md

* Update

* Update test data

* Add content assertions

* Update pr.yaml

* Update

* Update pr.yaml

* Add `pip install pillow`

* Merge CONVENTIONS.md into CONTRIBUTING.md

* Temporarilly fix CI

* Update `2d_feature_extraction` tool

* Update `2d_auto_threshold` tool

* Fix `2d_feature_extraction` tests

* Fix test data

* Fix tool

* Start refactoring `2d_simple_filter` tool

* Fix `2d_simple_filter` tool

* Add more tests

* Fix tests

* Update

* Fix

* Update CONTRIBUTING.md

* Fix bugs

* Update CONTRIBUTING.md

* Update CONTRIBUTING.md

* Fix linting issues

* Update CONTRIBUTING.md

* Refactor tests using macros

* Update tests

* Update CONTRIBUTING.md

* Update CONTRIBUTING.md

* Update pr.yaml

* Update `anisotropic_diffusion` tool

* Fix spelling (was BE, using AE is more consistent)

* Update `2d_histogram_equalization` tool

* Update `colorize_labels` tool

* Update `bf2raw` tool

* Add `macros/creators.xml`

* Update `rfove` tool

* Fix `bioformats2raw` and `voronoi_tessellation` tools

* Fix labels

* Fix `rfove` tool

* Update `overlay_images` tool

* Add RGB test for `overlay_images`

* Update `overlay_images` help

* Add image verification support to CI (BMCV#117)

* Update pr.yaml

* Update

* Update pr.yaml

* Add `pip install pillow`

* Temporarilly fix CI

* Update pr.yaml

* Update ci.yaml

* Add test tool from `/test/functional/tools/image_diff.xml`

* Update pr.yaml

* Add .shed.yml

* Add missing test files

* Switch CI to galaxy fork `kostrykin/galaxy` branch `galaxy-image-analysis`

This is a temporary change to enable the features from:

 - galaxyproject/galaxy#17556
 - galaxyproject/galaxy#17581

* Remove `test` tool

* Update CONTRIBUTING.md

* Update CONTRIBUTING.md

* Update CONTRIBUTING.md

* Update test data

* Add RGB test for `overlay_images`

* Update `anisotropic_diffusion` to medpy 0.4.0

* Update PULL_REQUEST_TEMPLATE.md

* Update `projective_transformation_points` tool

* Update PULL_REQUEST_TEMPLATE.md

* Update `bioformats2raw`

* Update `projective_transformation` tool

* Update `bfconvert` tool (tests failing)

* Update CONTRIBUTING.md

* Fix `bfconvert` tests

* Fix test data size

* Update `projective_transformation_points` tool

* Remove spurious files

* Update requirements using `@TOOL_VERSION@` macro

* Update `segmetrics` tool

* Remove spurious sym links

* Update `superdsm` tool

* Update `scale_image` tool (not finished yet)

* Fix tests

* Fix `scale_image` tool

* Refactor

* Fix linting issues

* Update tests and help of `2d_simple_filter` tool

* Update `scale_image` help

* Update CI to use latest version of `galaxy` fork

* Fix tests for float TIFF in `scale_image` tool

* Add tifffile to CI workflows

* Fix tests of `bfconvert` tool

* Update `2d_filter_segmentation_by_features` tool

* Update `image_math` tool

* Update `orientationpy` tool

* Update `morphological_operations` tool

* Update `split_labelmap` tool

* Add float TIFF test for `2d_simple_filter`

* Update `slice_image` tool

* Add missing macros

* Update help of `2d_filter_segmentation_by_features` tool

* Update `concat_images` tool

* Fix linting issues

* Fix file sizes

* Fix missing `profile`
kostrykin added a commit to BMCV/galaxy-image-analysis that referenced this pull request Apr 4, 2024
Update 23 tools and macros

* Update 23 tools

* Rename tools

Change the `Performs` in the name of the two tools

- `Performs projective transformation with/without labels`
- `Performs projective transformation`

to `Perform`, which is more consistent with the name of the other tools (imperative form).

* Update projective_transformation.xml

* Update projective_transformation_points.xml

* Add CONVENTIONS.md

* wip

* wip

* Update CONVENTIONS.md

* Update CONVENTIONS.md

* Update CONVENTIONS.md

* Update CONVENTIONS.md

* Update CONVENTIONS.md

* Update CONVENTIONS.md

* Remove help section

* Update CONVENTIONS.md

* Update CONVENTIONS.md

* Update CONVENTIONS.md

* Update

* Update test data

* Add content assertions

* Update pr.yaml

* Update

* Update pr.yaml

* Add `pip install pillow`

* Merge CONVENTIONS.md into CONTRIBUTING.md

* Temporarilly fix CI

* Update `2d_feature_extraction` tool

* Update `2d_auto_threshold` tool

* Fix `2d_feature_extraction` tests

* Fix test data

* Fix tool

* Start refactoring `2d_simple_filter` tool

* Fix `2d_simple_filter` tool

* Add more tests

* Fix tests

* Update

* Fix

* Update CONTRIBUTING.md

* Fix bugs

* Update CONTRIBUTING.md

* Update CONTRIBUTING.md

* Fix linting issues

* Update CONTRIBUTING.md

* Refactor tests using macros

* Update tests

* Update CONTRIBUTING.md

* Update CONTRIBUTING.md

* Update pr.yaml

* Update `anisotropic_diffusion` tool

* Fix spelling (was BE, using AE is more consistent)

* Update `2d_histogram_equalization` tool

* Update `colorize_labels` tool

* Update `bf2raw` tool

* Add `macros/creators.xml`

* Update `rfove` tool

* Fix `bioformats2raw` and `voronoi_tessellation` tools

* Fix labels

* Fix `rfove` tool

* Update `overlay_images` tool

* Add RGB test for `overlay_images`

* Update `overlay_images` help

* Add image verification support to CI (#117)

* Update pr.yaml

* Update

* Update pr.yaml

* Add `pip install pillow`

* Temporarilly fix CI

* Update pr.yaml

* Update ci.yaml

* Add test tool from `/test/functional/tools/image_diff.xml`

* Update pr.yaml

* Add .shed.yml

* Add missing test files

* Switch CI to galaxy fork `kostrykin/galaxy` branch `galaxy-image-analysis`

This is a temporary change to enable the features from:

 - galaxyproject/galaxy#17556
 - galaxyproject/galaxy#17581

* Remove `test` tool

* Update CONTRIBUTING.md

* Update CONTRIBUTING.md

* Update CONTRIBUTING.md

* Update test data

* Add RGB test for `overlay_images`

* Update `anisotropic_diffusion` to medpy 0.4.0

* Update PULL_REQUEST_TEMPLATE.md

* Update `projective_transformation_points` tool

* Update PULL_REQUEST_TEMPLATE.md

* Update `bioformats2raw`

* Update `projective_transformation` tool

* Update `bfconvert` tool (tests failing)

* Update CONTRIBUTING.md

* Fix `bfconvert` tests

* Fix test data size

* Update `projective_transformation_points` tool

* Remove spurious files

* Update requirements using `@TOOL_VERSION@` macro

* Update `segmetrics` tool

* Remove spurious sym links

* Update `superdsm` tool

* Update `scale_image` tool (not finished yet)

* Fix tests

* Fix `scale_image` tool

* Refactor

* Fix linting issues

* Update tests and help of `2d_simple_filter` tool

* Update `scale_image` help

* Update CI to use latest version of `galaxy` fork

* Fix tests for float TIFF in `scale_image` tool

* Add tifffile to CI workflows

* Fix tests of `bfconvert` tool

* Update `2d_filter_segmentation_by_features` tool

* Update `image_math` tool

* Update `orientationpy` tool

* Update `morphological_operations` tool

* Update `split_labelmap` tool

* Add float TIFF test for `2d_simple_filter`

* Update `slice_image` tool

* Add missing macros

* Update help of `2d_filter_segmentation_by_features` tool

* Update `concat_images` tool

* Fix linting issues

* Fix file sizes

* Fix missing `profile`

* Update test macros
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants