diff --git a/automated-testing/README.md b/automated-testing/README.md index ff73a59..db85304 100644 --- a/automated-testing/README.md +++ b/automated-testing/README.md @@ -16,7 +16,7 @@ The subsequent demonstration showcases *automated testing* and specifically addr > Make sure that all [system requirements](../utils/requirements.md) are fulfilled. > Additionally, this demo requires a [self-hosted GitHub Runner](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners) to execute scenarios within a CI workflow. The specific requirements for such a runner are listed [below](#self-hosted-github-runner). -This demo aims to automatically evaluate predefined test scenarios. For this purpose, a test catalog can be defined using OpenSCENARIO files as contained in the [scenarios](../utils/scenarios) folder. These scenarios can be simulated and evaluated using the [carla-scenario-runner](https://github.com/ika-rwth-aachen/carla-scenario-runner). Thus, a basic [docker-compose setup](./docker-compose.yml) only includes the `carla-simulator` and a `carla-scenario-runner` Docker service. So, in general, the demo enables the efficient execution of multiple scenario-based tests with CARLA, both in local environments and within an automated GitHub CI process. +This demo aims to automatically evaluate predefined test scenarios. For this purpose, a test catalog can be defined using OpenSCENARIO files as contained in the [scenarios](../utils/scenarios) folder. These scenarios can be simulated and evaluated using the [carla-scenario-runner](https://github.com/ika-rwth-aachen/carla-scenario-runner). Thus, a basic [docker-compose template](./template.yml) only includes the `carla-simulator` and a `carla-scenario-runner` Docker service. So, in general, the demo enables the efficient execution of multiple scenario-based tests with CARLA, both in local environments and within an automated GitHub CI process. ### Manual Testing Pipeline @@ -34,6 +34,10 @@ or ### Automatic CI Pipeline +All scenarios within the test catalog are also simulated and evaluated in an automatic [CI pipeline on GitHub](https://github.com/ika-rwth-aachen/carlos/actions/workflows/automated-testing.yml). A detailed look in the [scenarios folder](../utils/scenarios/) shows that a few of them have the postfix `.opt` marking them as optional. This means a failure in test evaluation is allowed for those specific scenarios. The CI pipeline processes required scenarios first, and than considered all optional scenarios. In both cases a job matrix is generated before consecutive jobs are created to simulate the specific scenario. As an example, a workflow is shown below. + +

+ #### Actions - we provide open [GitHub actions](../.github/actions/) for CARLOS diff --git a/automated-testing/docker-compose.yml b/automated-testing/docker-compose.yml deleted file mode 100644 index cccc28a..0000000 --- a/automated-testing/docker-compose.yml +++ /dev/null @@ -1,26 +0,0 @@ -services: - - carla-simulator: - deploy: - resources: - reservations: - devices: - - driver: nvidia - count: 1 - capabilities: [gpu] - environment: - DISPLAY: $DISPLAY - volumes: - - /tmp/.X11-unix:/tmp/.X11-unix - privileged: True - image: rwthika/carla-simulator:server - command: bash -ic './CarlaUE4.sh -nosound $SIMULATOR_FLAGS 2>/dev/null' - - carla-scenario-runner: - depends_on: - carla-simulator: - condition: service_healthy - volumes: - - ../utils/scenarios:/scenarios - image: rwthika/carla-scenario-runner:latest - command: bash -ic "python ./scenario_runner.py --host carla-simulator --openscenario /scenarios/town10.xosc.opt --output"