Handle model state in benchmark CI jobs

[metascroy]

Previously when running a pte file that has model state, our CI would fail with an error like:

2024-12-19 14:15:43.359148-0800 Benchmark[532:16686] [coreml] The input feature for layers_10_attention_SDPA_kv_cache_v_cache must be an MLState, but it was not.

This is because if you build the xcode frameworks with xcode 15, the ExecuTorch CoreML backend is built to not handle MLState variables:

executorch/backends/apple/coreml/runtime/delegate/MLModel_Prewarm.h

Line 11 in fc42a4e

#if !defined(MODEL_STATE_IS_SUPPORTED) && __has_include(<CoreML/MLModel+MLState.h>)

Note the prebuilt frameworks cannot handle MLState (https://github.com/pytorch/executorch/tree/main/extension/benchmark/apple/Benchmark#set-up-the-frameworks).

This PR:

• Edits the benchmark README.md to mention that Xcode 16.0 must be used to build frameworks that handle pte files that use CoreML MLState
• Reenables using MLState for iOS18 models

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/7432

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 7 New Failures, 1 Unrelated Failure

As of commit 93be0ab with merge base ac7a950 ():

NEW FAILURES - The following jobs have failed:

• apple-perf / benchmark-on-device (llama, coreml_fp16, apple_iphone_15, arn:aws:devicefarm:us-west-2:3085353851... / mobile-job (ios) (gh)
  Process completed with exit code 22.
• apple-perf / benchmark-on-device (llama, mps, apple_iphone_15, arn:aws:devicefarm:us-west-2:308535385114:devic... / mobile-job (ios) (gh)
  Process completed with exit code 22.
• apple-perf / benchmark-on-device (llama, xnnpack_q8, apple_iphone_15, arn:aws:devicefarm:us-west-2:30853538511... / mobile-job (ios) (gh)
  Process completed with exit code 22.
• apple-perf / build-benchmark-app / macos-job (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 70
• apple-perf / upload-benchmark-results (gh)
  Process completed with exit code 1.
• Check Labels / Check labels (gh)
  RuntimeError: Error checking labels: PR does not have required labels
• Lint / lintrunner / linux-job (gh)
  >>> Lint for exir/tests/test_quant_fusion_pass.py:

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Apple / test-demo-ios / mobile-job (ios) (gh) (trunk failure)
  Final attempt failed. Child_process exited with error code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your changes are user facing and intended to be a part of release notes, please use a label starting with release notes:.

If not, please add the topic: not user facing label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "topic: not user facing"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[metascroy]

@guangy10 the apple-perf/build-benchmark-app job fails with "xcodebuild: error: Unable to find a device matching the provided destination specifier:2025-01-02T18:01:02.3778750Z { platform:iOS }"

Do you know what could cause this?