-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add image_diff
comparison method for test output verification using images
#17556
Conversation
@kostrykin I've added the changes to the package requirements, and also some type annotations if you'd like to double check them. @jmchilton Are you OK with numpy and pillow being added as requirements to the galaxy-tool-util package? |
This is really, really cool stuff - sorry about being a bit picky about the packaging stuff. |
Thanks, the type annotations look right to me! |
together pysam, which is also conditionally imported in ``lib/galaxy/tool_util/verify/__init__.py`` .
@jmchilton I've added the extra as you suggested, thanks! |
Thanks @kostrykin ! |
Add `image_diff` comparison method for test output verification using images
…ysis` This is a temporary change to enable the features from: - galaxyproject/galaxy#17556 - galaxyproject/galaxy#17581
* Update pr.yaml * Update * Update pr.yaml * Add `pip install pillow` * Temporarilly fix CI * Update pr.yaml * Update ci.yaml * Add test tool from `/test/functional/tools/image_diff.xml` * Update pr.yaml * Add .shed.yml * Add missing test files * Switch CI to galaxy fork `kostrykin/galaxy` branch `galaxy-image-analysis` This is a temporary change to enable the features from: - galaxyproject/galaxy#17556 - galaxyproject/galaxy#17581 * Remove `test` tool
* Rename tools Change the `Performs` in the name of the two tools - `Performs projective transformation with/without labels` - `Performs projective transformation` to `Perform`, which is more consistent with the name of the other tools (imperative form). * Update projective_transformation.xml * Update projective_transformation_points.xml * Add CONVENTIONS.md * wip * wip * Update CONVENTIONS.md * Update CONVENTIONS.md * Update CONVENTIONS.md * Update CONVENTIONS.md * Update CONVENTIONS.md * Update CONVENTIONS.md * Remove help section * Update CONVENTIONS.md * Update CONVENTIONS.md * Update CONVENTIONS.md * Update * Update test data * Add content assertions * Update pr.yaml * Update * Update pr.yaml * Add `pip install pillow` * Merge CONVENTIONS.md into CONTRIBUTING.md * Temporarilly fix CI * Update `2d_feature_extraction` tool * Update `2d_auto_threshold` tool * Fix `2d_feature_extraction` tests * Fix test data * Fix tool * Start refactoring `2d_simple_filter` tool * Fix `2d_simple_filter` tool * Add more tests * Fix tests * Update * Fix * Update CONTRIBUTING.md * Fix bugs * Update CONTRIBUTING.md * Update CONTRIBUTING.md * Fix linting issues * Update CONTRIBUTING.md * Refactor tests using macros * Update tests * Update CONTRIBUTING.md * Update CONTRIBUTING.md * Update pr.yaml * Update `anisotropic_diffusion` tool * Fix spelling (was BE, using AE is more consistent) * Update `2d_histogram_equalization` tool * Update `colorize_labels` tool * Update `bf2raw` tool * Add `macros/creators.xml` * Update `rfove` tool * Fix `bioformats2raw` and `voronoi_tessellation` tools * Fix labels * Fix `rfove` tool * Update `overlay_images` tool * Add RGB test for `overlay_images` * Update `overlay_images` help * Add image verification support to CI (BMCV#117) * Update pr.yaml * Update * Update pr.yaml * Add `pip install pillow` * Temporarilly fix CI * Update pr.yaml * Update ci.yaml * Add test tool from `/test/functional/tools/image_diff.xml` * Update pr.yaml * Add .shed.yml * Add missing test files * Switch CI to galaxy fork `kostrykin/galaxy` branch `galaxy-image-analysis` This is a temporary change to enable the features from: - galaxyproject/galaxy#17556 - galaxyproject/galaxy#17581 * Remove `test` tool * Update CONTRIBUTING.md * Update CONTRIBUTING.md * Update CONTRIBUTING.md * Update test data * Add RGB test for `overlay_images` * Update `anisotropic_diffusion` to medpy 0.4.0 * Update PULL_REQUEST_TEMPLATE.md * Update `projective_transformation_points` tool * Update PULL_REQUEST_TEMPLATE.md * Update `bioformats2raw` * Update `projective_transformation` tool * Update `bfconvert` tool (tests failing) * Update CONTRIBUTING.md * Fix `bfconvert` tests * Fix test data size * Update `projective_transformation_points` tool * Remove spurious files * Update requirements using `@TOOL_VERSION@` macro * Update `segmetrics` tool * Remove spurious sym links * Update `superdsm` tool * Update `scale_image` tool (not finished yet) * Fix tests * Fix `scale_image` tool * Refactor * Fix linting issues * Update tests and help of `2d_simple_filter` tool * Update `scale_image` help * Update CI to use latest version of `galaxy` fork * Fix tests for float TIFF in `scale_image` tool * Add tifffile to CI workflows * Fix tests of `bfconvert` tool * Update `2d_filter_segmentation_by_features` tool * Update `image_math` tool * Update `orientationpy` tool * Update `morphological_operations` tool * Update `split_labelmap` tool * Add float TIFF test for `2d_simple_filter` * Update `slice_image` tool * Add missing macros * Update help of `2d_filter_segmentation_by_features` tool * Update `concat_images` tool * Fix linting issues * Fix file sizes * Fix missing `profile`
Update 23 tools and macros * Update 23 tools * Rename tools Change the `Performs` in the name of the two tools - `Performs projective transformation with/without labels` - `Performs projective transformation` to `Perform`, which is more consistent with the name of the other tools (imperative form). * Update projective_transformation.xml * Update projective_transformation_points.xml * Add CONVENTIONS.md * wip * wip * Update CONVENTIONS.md * Update CONVENTIONS.md * Update CONVENTIONS.md * Update CONVENTIONS.md * Update CONVENTIONS.md * Update CONVENTIONS.md * Remove help section * Update CONVENTIONS.md * Update CONVENTIONS.md * Update CONVENTIONS.md * Update * Update test data * Add content assertions * Update pr.yaml * Update * Update pr.yaml * Add `pip install pillow` * Merge CONVENTIONS.md into CONTRIBUTING.md * Temporarilly fix CI * Update `2d_feature_extraction` tool * Update `2d_auto_threshold` tool * Fix `2d_feature_extraction` tests * Fix test data * Fix tool * Start refactoring `2d_simple_filter` tool * Fix `2d_simple_filter` tool * Add more tests * Fix tests * Update * Fix * Update CONTRIBUTING.md * Fix bugs * Update CONTRIBUTING.md * Update CONTRIBUTING.md * Fix linting issues * Update CONTRIBUTING.md * Refactor tests using macros * Update tests * Update CONTRIBUTING.md * Update CONTRIBUTING.md * Update pr.yaml * Update `anisotropic_diffusion` tool * Fix spelling (was BE, using AE is more consistent) * Update `2d_histogram_equalization` tool * Update `colorize_labels` tool * Update `bf2raw` tool * Add `macros/creators.xml` * Update `rfove` tool * Fix `bioformats2raw` and `voronoi_tessellation` tools * Fix labels * Fix `rfove` tool * Update `overlay_images` tool * Add RGB test for `overlay_images` * Update `overlay_images` help * Add image verification support to CI (#117) * Update pr.yaml * Update * Update pr.yaml * Add `pip install pillow` * Temporarilly fix CI * Update pr.yaml * Update ci.yaml * Add test tool from `/test/functional/tools/image_diff.xml` * Update pr.yaml * Add .shed.yml * Add missing test files * Switch CI to galaxy fork `kostrykin/galaxy` branch `galaxy-image-analysis` This is a temporary change to enable the features from: - galaxyproject/galaxy#17556 - galaxyproject/galaxy#17581 * Remove `test` tool * Update CONTRIBUTING.md * Update CONTRIBUTING.md * Update CONTRIBUTING.md * Update test data * Add RGB test for `overlay_images` * Update `anisotropic_diffusion` to medpy 0.4.0 * Update PULL_REQUEST_TEMPLATE.md * Update `projective_transformation_points` tool * Update PULL_REQUEST_TEMPLATE.md * Update `bioformats2raw` * Update `projective_transformation` tool * Update `bfconvert` tool (tests failing) * Update CONTRIBUTING.md * Fix `bfconvert` tests * Fix test data size * Update `projective_transformation_points` tool * Remove spurious files * Update requirements using `@TOOL_VERSION@` macro * Update `segmetrics` tool * Remove spurious sym links * Update `superdsm` tool * Update `scale_image` tool (not finished yet) * Fix tests * Fix `scale_image` tool * Refactor * Fix linting issues * Update tests and help of `2d_simple_filter` tool * Update `scale_image` help * Update CI to use latest version of `galaxy` fork * Fix tests for float TIFF in `scale_image` tool * Add tifffile to CI workflows * Fix tests of `bfconvert` tool * Update `2d_filter_segmentation_by_features` tool * Update `image_math` tool * Update `orientationpy` tool * Update `morphological_operations` tool * Update `split_labelmap` tool * Add float TIFF test for `2d_simple_filter` * Update `slice_image` tool * Add missing macros * Update help of `2d_filter_segmentation_by_features` tool * Update `concat_images` tool * Fix linting issues * Fix file sizes * Fix missing `profile` * Update test macros
As part of the Imaging Hackathon in Freiburg.
This PR adds the
image_diff
comparison method for test output verification using images.Previous comparison methods comprise
diff
,re_match
,lines_diff
,re_match_multiline
,contains
, andsim_size
, all of which are either unsuitable for image data, or dangerous to be used (because the size of a file does not really represent its content). This PR adds theimage_diff
comparison method, which is specifically tailored for single-channel and multi-channel image data (e.g., RGB), and can be used like this:The difference of a pair of images is quantified as the pixel-wise distance between the images, for which different metrics can be used. For different applications and different types of images (e.g., intensity vs. label images), different types of metrics are common. The following metrics are added for intensity images:
metric
mae
mse
rms
fro
For label images (label maps), it is usually irrelevant which particular label is used for labeling an image region, but only the sets of pixels corresponding to individual image regions need to be compared. This can be accomplished using one minus the intersection over the union of the regions:
metric
iou
To decide whether a pair of images is equal, a tolerance is required, because different compression algorithms (or different versions thereof) can lead to different compression artifacts. A pair of images is considered to be equal in terms of the specified metric, if the distance of the images computed with respect to the metric is not above a threshold
eps
(0.01 by default).How to test the changes?
(Select all options that apply)
License