Label-changing augmentations

[Callidior]

Hi and thanks for providing FFCV! Looks like a great package which I would be keen to try out.

I was just wondering: Is it possible to use image augmentations that also change the label? For example, using random cropping in a segmentation scenario would not only change the image but also the segmentation mask, which would need to be cropped at the same coordinates as the image. Some transformations would affect both image and label (in this case, all geometric ones), others would only affect the image (e.g., color and contrast augmentations).

I looked through the examples and API documentation but could figure out a straightforward way to achieve this with FFCV.

[GuillaumeLeclerc]

Yes, It is possible, we actually have some operations that work this way

• https://github.com/libffcv/ffcv/blob/main/ffcv/transforms/mixup.py
  and to a lesser extent
• https://github.com/libffcv/ffcv/blob/main/ffcv/transforms/poisoning.py

[Callidior]

Thanks for the quick response!

If I see it correctly, the proposed solution is to use separate instances of the transform in the different pipelines and, for each batch, seed the random number generator with the index of the last sample in the batch.

That means that even for standard transforms such as RandomHorizontalFlip, one would need to create a new transform that differs from the original one only by one additional line at the beginning of the transformation code:

class ConsistentRandomHorizontalFlip(Operation):
    ...
    def generate_code(self) -> Callable:
        ...
        def flip(images, dst, indices):
            np.random.seed(indices[-1])
            ...

Is that correct? If so, it would be nice to have such a feature built-in to the standard augmentations. Otherwise, I would have to copy and modify many of them for scenarios where the same transformation should be applied to multiple inputs (e.g., segmentation or image-to-image translation).

[GuillaumeLeclerc]

@Callidior we are currently trying to find a nice API in #86 Feel free to jump in there to give your opinion.

[netw0rkf10w]

@GuillaumeLeclerc @Callidior Sorry I couldn't quite get the idea. Could you please give an example of how to use Mixup? Thanks a lot!