Greyscale images with ICC color profiles that are output as AVIF renditions display incorrectly in Firefox

[coredumperror]

This one's a really obscure issue, and I'm not actually sure why the combination of greyscale + color profile causes this, but I have example files to show it in action.

The issue appears to be that Firefox still doesn't support grid-based AVIF files in its AVIF renderer, despite the bug being active for almost 4 years. The ultimate source of that problem is that Firefox uses mp4parse-rust as its AVIF renderer, and that doesn't support grid-based AVIFs. This issue on pdn-avif may also shed some light on the underlying problem.

To see Firefox's grid issue in a non-Wagtail environment, go to https://0xc0000054.github.io/pdn-avif/using-image-grids.html and click the "Image grid example.avif" link in the second section. In Chrome, it'll display the same 1/2/3/4 image seen below the link. In Firefox, you'll get an error message.

I've also attached three files to this issue that can be uploaded to a Wagtail site to show off how greyscale + color profile makes Willow render the AVIF in a way that breaks in Firefox.

This image, which is a greyscale jpeg with an ICC Color Profile of "Dot Gain 20%" triggers the render bug:

This image, which is a greyscale jpeg with no ICC profile, does not trigger the bug:

(Note that, if you flip between the above images, you'll see that the one with no profile is dramatically darker)

This image, which is an RGB jpeg with an sRGB color profile, does not trigger the bug:

(This image looks identical to the original one with the color profile intact)

Upload these three to a Wagtail site that's configured to display renditions as AVIF, and you'll see that the first one displays wrong in Firefox:

I'm not 100% sure that this problem is actually with grid-based AVIFs, because I don't know how to determine if the broken one is actually a grid-based AVIF, but it seems likely.

My one clue about a potential fix for this is that pdn-avif issue that I linked. Apparently, using a "very slow" preset for the AVIF rendering step would prevent the image from being generated as a grid, and thus prevent the image corruption. Maybe that's something that Willow could do when it runs into a greyscale+color profile image that it's being asked to render as AVIF?

[coredumperror]

Further testing shows that greyscale PNGs with color profiles also cause the problem in Firefox. Converting the PNG to RGB fixes it, like it did with the jpegs.

[coredumperror]

I was writing a monkey patch to PillowImage.save_as_avif() when I discovered that simply converting greyscale images with color profiles to sRGB colorspace, like what that function already does for CMYK images, actually fixes the problem perfectly.

I changed line 440 (in v1.8.0) from:

            if image.mode == "CMYK":

to

            if image.mode in ("CMYK", "L"):

And now the image renditions aren't corrupted when viewed in Firefox, and they still look the same as the original.

Would this be a viable patch to Willow?

[zerolab]

cc @Stormheg as I think this relates to #142 and it may affect your projects too

[coredumperror]

Looks like there needs to be a small tweak to my "patch", because there are actually two different image modes that cover greyscale:

            if image.mode in ("CMYK", "L", "LA"):

EDIT: Oh, maybe this one isn't a good idea... Still testing, and getting some strange results.

[coredumperror]

OK, so it turns out that doing an sRGB colorspace conversion on a greyscale image with an alpha channel gives you really borked results. I ended up having to convert the image to non-transparent via the method described here: https://stackoverflow.com/a/33507138/464318, before doing the sRGB colorspace conversion. Now my patch is:

        if image.mode in ("CMYK", "L", "LA"):
            if image.mode == "LA":
                background = PILImage.new("RGBA", image.size, (255, 255, 255))
                rgb_image = image.convert("RGBA")
                self.image = PILImage.alpha_composite(background, rgb_image)

[Stormheg]

Interesting case! Looks like you have gone down a similar rabbit hole as I did in #142, have fun 😄

I wouldn't be opposed to converting obscure formats like lightness to the more common sRGB, like I did with CMYK. It would probably avoid a bunch of equally obscure render bugs in browsers.

Happy to review patches 👍

[Stormheg]

@coredumperror did you end up getting anywhere?

[coredumperror]

Here's the monkey patch to PillowImage.save_as_avif(). that we've been using in prod for a few months:

@Image.operation
def patched_PillowImage_save_as_avif(self, f, quality=80, lossless=False, apply_optimizers=True):
    kwargs = {"quality": quality}
    if lossless:
        kwargs = {"quality": -1, "chroma": 444}

    image = self.image
    icc_profile = self.get_icc_profile()
    if icc_profile is not None:
        # If the image is in CMYK mode *and* has an ICC profile, we need to be more diligent
        # about how we handle the color space. AVIF will encode as RGB so we need to do extra
        # work to ensure the colors are as accurate as possible. We don't want to retain
        # the color profile as-is because it is not meant for RGB images and
        # will result in inaccurate colors. The transformation to sRGB should result
        # in a more accurate representation of the original image, though
        # it will likely not be perfect.
        # BEGIN PATCH - Convert both CMYK _and_ Greyscale images to sRGB color space.
        if image.mode in ("CMYK", "L", "LA"):
            # TODO: An alternate approach to all this faffing about with color spaces and transparency
            #  would be to just give up and throw a ValueError from here if the image is in a mode known to be
            #  problematic for AVIF conversion. That will get caught by the {% responsive_image %} tag, and an AVIF
            #  rendition will simply not be rendered. That would cause this function to be run every time the image
            #  gets displayed, but that shouldn't cause too much of a performance issue, since we'd
            #  be skipping the entire image conversion process.
            if image.mode == "LA":
                # This is a greyscale image with an alpha channel. For some reason, doing the usual sRGB conversion
                # will result in a fully transparent AVIF. The only solution I've found is to convert the image
                # from a transparent gif to a non-transparent one with a white background.
                # This algorithm is from https://stackoverflow.com/a/33507138/464318
                background = PILImage.new("RGBA", image.size, (255, 255, 255))
                # Can only run alpha_composite() on two RGBA images, so we need to convert this one to that mode.
                rgb_image = image.convert("RGBA")
                self.image = PILImage.alpha_composite(background, rgb_image)
        # END PATCH
            pillow_image = self.transform_colorspace_to_srgb()
            image = pillow_image.image
            kwargs["icc_profile"] = pillow_image.get_icc_profile()
        else:
            kwargs["icc_profile"] = icc_profile

    image.save(f, "AVIF", **kwargs)

    if not lossless and apply_optimizers:
        self.optimize(f, "heic")

    return AvifImageFile(f)
PillowImage.save_as_avif = patched_PillowImage_save_as_avif

It adds 'L' and 'LA' image modes to those that the function converts to sRGB, and also does an extra step on 'LA' mode images (greyscale w/ transparency) to get around another issue we found.

[Stormheg]

@coredumperror do you think Willow should do anything here to improve? Ideally, we'd like to fix the issue - so you do not need to monkey-patch 😉

[coredumperror]

Well, if that code change (minus the TODO, I guess) is acceptable, applying it directly to Willow would certainly work for us. We'd love to have one less monkey patch in our code.

[Stormheg]

@zerolab how do you feel about the monkey patch Robert is using? Is this an acceptable approach to implement in Willow?

1. I'm okay with converting 'L' mode images with color profiles to sRGB and removing the profile. Similar to what I did with CMYK. One less obscure image mode for cross platform browsers to struggle with. Visually there shouldn't be a noticeable change in the image. It is already expected behavior that Willow rewrites and 'optimizes' the image. As long as the result is visually similar enough, I don't see an issue with this.

2. I feel less confident about the second patch marked as TODO. @coredumperror if I understand correctly, it forces all transparency in the image to white for the conversion to succeed. This white background is retained in the resulting image - am I correct? Do you happen to have an image file for me that I could play with?
   2.1. Assuming the answer is 'yes' on the question above, perhaps an alternative could be to store all alpha pixel values in a buffer somewhere in overlay that back on the image after sRGB conversion? Would that fix the issue while retaining transparency?

[coredumperror]

I believe this is the transparent PNG I was using to test that feature:

[zerolab]

Will have a think sometime next week.