AWS Device Farm

[mattbajorek]

Does anybody have a working setup with AWS Device Farm? I am unable to get the proper setup with the test_spec and using the Android Test Orchestrator.

[mattbajorek]

Here is the test spec AWS recommends for the Android Test Orchestrator. When running this locally with the proper variables it succeeds to run. However on the AWS Device Farm emulators the testing app does not show to be opened.

version: 0.1

# This flag enables your test to run using Device Farm's Amazon Linux 2 test host. For more information,
# please see https://docs.aws.amazon.com/devicefarm/latest/developerguide/amazon-linux-2.html
android_test_host: amazon_linux_2

# Phases represent collections of commands that are executed during your test run on the test host.
phases:

  # The install phase contains commands for installing dependencies to run your tests.
  # For your convenience, certain dependencies are preinstalled on the test host. To lean about which
  # software is included with the host, and how to install additional software, please see:
  # https://docs.aws.amazon.com/devicefarm/latest/developerguide/amazon-linux-2-supported-software.html
  install:
    commands:
 
  # The pre-test phase contains commands for setting up your test environment.
  pre_test:
    commands:
      
  # The test phase contains commands for running your tests.
  test:
    commands:
      # To run your tests using "Android Test Orchestrator", you can use the following commands instead.
      # Note: you will need to include the orchestrator APK and test services APK as auxiliary apps with your
      # ScheduleRun request as auxiliary apps. For more information, please see:
      # https://developer.android.com/training/testing/instrumented-tests/androidx-test-libraries/runner#use-android
      - |
        adb -s $DEVICEFARM_DEVICE_UDID shell "CLASSPATH=\$(pm path androidx.test.services) app_process / \
          androidx.test.services.shellexecutor.ShellMain am instrument -w --no-window-animation -e \
          clearPackageData true -e targetInstrumentation $DEVICEFARM_TEST_PACKAGE_NAME/$DEVICEFARM_TEST_PACKAGE_RUNNER \
          androidx.test.orchestrator/.AndroidTestOrchestrator || echo \"FAILURES...\"" 2>&1 |
          tee $DEVICEFARM_LOG_DIR/instrument.log

      # If Android Test Orchestrator produces any reports on the device, they will be pulled into your customer
      # artifacts directory using the following commands:
      - |
        adb -s $DEVICEFARM_DEVICE_UDID pull /sdcard/odo >/dev/null &&
        mv odo/* $DEVICEFARM_LOG_DIR || true

      # The below command is used to detect if any of the Orchestrator tests have failed
      - |
        if [ $(cat $DEVICEFARM_LOG_DIR/instrument.log |
          egrep "^(FAILURES...|INSTRUMENTATION_RESULT: shortMsg=Process crashed.)$" | wc -l) -ge 1 ];
        then
          echo "Marking the test suite as failed because Android Orchestrator found that some of the tests failed";
          false;
        fi;
      
  # The post-test phase contains commands that are run after your tests have completed.
  # If you need to run any commands to generating logs and reports on how your test performed,
  # we recommend adding them to this section.
  post_test:
    commands:

# Artifacts are a list of paths on the filesystem where you can store test output and reports.
# All files in these paths will be collected by Device Farm.
# These files will be available through the ListArtifacts API as your "Customer Artifacts".
artifacts:
  # By default, Device Farm will collect your artifacts from the $DEVICEFARM_LOG_DIR directory.
  - $DEVICEFARM_LOG_DIR

[Nithsua]

Hey, where you able to figure this out?

[mattbajorek]

I was finally able to get this to work. That .yml file above must be used, but also there has to be 4 apks:

1. app (build/app/outputs/apk/debug/app-debug.apk) -> Initial app application uploaded
2. test (build/app/outputs/apk/androidTest/debug/app-debug-androidTest.apk) -> Change Test Framework from Built-in: Fuzz to Instrumentation
3. orchestrator 1.4.2 (Download apk here: https://mvnrepository.com/artifact/androidx.test/orchestrator/1.4.2) -> Install other apps
4. test services 1.4.2 (Download apk here: https://mvnrepository.com/artifact/androidx.test.services/test-services/1.4.2) -> Install other apps

However, the tests all run in one large tests and individual failures are not show in the same way that Firebase Test Labs works.