feat: Add rotation to iOS (the sequel)

[juanmartin8a]

Continuation of #540. Added a new variable to the InputImageMetadata class, cameraLensDirection, it is used only on iOS to determine if the image is mirrored (for example when using the front camera) or not.

[juanmartin8a]

The example must change for this to work, sadly I am unable to run the example app on my computer. On android things can stay the same however on iOS the rotation compensation must be calculated based on the orientation of the device. Like this:

int getRotationCompensation(int orientation, int sensorOrientation, CameraLensDirection cameraLensDirection) {
  if (Platform.isAndroid) {
    return cameraLensDirection == CameraLensDirection.front 
      ? (sensorOrientation + orientation) % 360
      : (sensorOrientation - orientation + 360) % 360;
  }

  return (360 - orientation) % 360;
}

Orientation can be calculated using the native_device_orientation package.

Unlike Android, iOS does not require different coordinate translation when the device orientation changes, it can be done with just x * canvasSize.width / imageSize.width for x, and y * canvasSize.height / imageSize.height for y. This applies for all orientations.

This works when the camera is locked to portrait up in iOS. If the camera is not being locked, then a rotation of 0 in the InputImage will do.

Finally, the InputImage must be called like this now,

InputImage inputImage = InputImage.fromBytes(
            bytes: image.planes[0].bytes, 
            metadata: InputImageMetadata(
              size: Size(image.width.toDouble(), image.height.toDouble()), 
              rotation: InputImageRotationValue.fromRawValue(rotation)!, 
              format: InputImageFormatValue.fromRawValue(image.format.raw)!, 
              bytesPerRow: image.planes[0].bytesPerRow,
              cameraLensDirection: InputImageCameraLensDirectionValue.fromRawValue(InputImageCameraLensDirection.back.rawValue)!
              )
            );

In Android the cameraLensRotation parameter is not used so any value will output the same result. :)

[fbernaly]

I need you to test with the example app and edit it if needed before merging this.

[fbernaly]

This is not needed. When the image is a file, camera lens is not used.
Also, check what it is happening inside filePathToVisionImage : https://github.com/flutter-ml/google_ml_kit_flutter/blob/develop/packages/google_mlkit_commons/ios/Classes/MLKVisionImage%2BFlutterPlugin.m#L20-L25

[fbernaly]

Orientation is already set in that method for files.

[juanmartin8a]

Sorry, I didn't noticed that. I already made a new commit with those changes.

[juanmartin8a]

I need you to test with the example app and edit it if needed before merging this.

It doesn't run in my computer (m1 macbook pro), I tried many times. The alternative is my parents computer which will probably melt if I tried to run it haha.

It does work as explained above on a personal project that I am building using face detection.

[fbernaly]

I will give it a try

[juanmartin8a]

I will give it a try

Thank you!

[fbernaly]

@juanmartin8a : Thanks for using this plugin and your contribution.

I merged your PR into this branch: feature/rotation
and I was able to compile it and play with it, but unfortunately the imageOrientationFromRotation:cameraLensDirection: method in this file is breaking things for our example app.

Since we are not setting MLKVisionImage.orientation, by default the display orientation of the image is .up:

and with your method you are introducing different rotations that are not handled in the example app and as a result it is not detecting any objects/faces/etc because the rotation is messed up.

I believe most of the devs consuming this plugin are doing the image rotation as we have it in the example app and explained in the README:
https://github.com/flutter-ml/google_ml_kit_flutter/tree/feature/rotation/packages/google_mlkit_commons#creating-an-inputimage

If we release your changes we will break a lot of projects once they pull the latest version with your changes.

If you were to fix the example app to work with your changes we will merge them and release them. In the meantime we will keep them for a while in the feature/rotation branch.

Let me know if you are not interested in contributing more so we could archive this branch.

Let me know your comments.

[juanmartin8a]

@juanmartin8a : Thanks for using this plugin and your contribution.

I merged your PR into this branch: feature/rotation and I was able to compile it and play with it, but unfortunately the imageOrientationFromRotation:cameraLensDirection: method in this file is breaking things for our example app.

Since we are not setting MLKVisionImage.orientation, by default the display orientation of the image is .up:

and with your method you are introducing different rotations that are not handled in the example app and as a result it is not detecting any objects/faces/etc because the rotation is messed up.

I believe most of the devs consuming this plugin are doing the image rotation as we have it in the example app and explained in the README: https://github.com/flutter-ml/google_ml_kit_flutter/tree/feature/rotation/packages/google_mlkit_commons#creating-an-inputimage

If we release your changes we will break a lot of projects once they pull the latest version with your changes.

If you were to fix the example app to work with your changes we will merge them and release them. In the meantime we will keep them for a while in the feature/rotation branch.

Let me know if you are not interested in contributing more so we could archive this branch.

Let me know your comments.

Hi! Sadly I am unable to test the example app as mentioned above because it does not run on my laptop (m1 MacBook). I will work on an update that is backwards compatible and doesn't break other projects. Thanks for letting me know.

[fbernaly]

I have an M1 as well, why you can not run it?

[juanmartin8a]

It was something about the compile size being too big and it being a common problem in the ARM architecture. I'm going to try again today and will let you know :)

[fbernaly]

maybe it was this issue:

https://forums.developer.apple.com/forums/thread/738275

got that one when I ran the app in Xcode 15, but I fixed it: 2d3b4f8

Pull the latest code from feature/rotation

[juanmartin8a]

Hi! Sorry for the late reply. The example works! I pulled from the develop branch and will be working on making the rotation backwards compatible.