Restructure and Improvement of E2E Tests Workflow (#5089)

* Test: see boolean input type for e2e.yml

* Add all e2e tests as booleans

* Test: see what description looks like for boolean

* Format that runs all tests, designated working tests, or one test

* Put all options in dropdown

* Put a line between categories and single tests

* setup file structure for e2e workflows

* temporarily put back specFiles input to make the e2e workflow runnable

* note about ApexLSP

* Update and rename miscellaneousE2E.yml to coreE2E.yml

* add boolean checking for lspE2E.yml

* fix syntax errors

* try to fix errors + add logging

* add finish to if statement block

* fix syntax error

* add logging

* move apexLSP from lspE2E to apexE2E + rename apexDebuggerE2E to apexE2E

* use 'with' instead of 'run' for if statements

* try putting the check for each boolean in a separate step

* add logging to see what tests are going to be run

* add logging to e2e.yml to see value of .specFiles

* replace testsToRun with {{ env.testsToRun }} and add logging

* try removing setting the value of env.testsToRun and check if if statements work

* remove single quote from true

* add logging for if statements

* print log statement if true

* put if statement inside run statement

* fix syntax of if statements in run statement

* try moving true out of if statement variable

* hard code env.testsToRun

* add parentheses around if conditions to fix Windows problem

* try adding logic to make semicolon separated string of e2e tests

* variable assignment instead of echo

* add curly braces around TESTS_TO_RUN

* comment out everything but checking which e2e tests are selected

* add dollar sign and curly braces to TESTS_TO_RUN when setting variable in if statements

* move TESTS_TO_RUN from an environment variable to a bash variable

* add dollar sign when declaring TESTS_TO_RUN variable

* add logging

* dummy test - comment out all references to TESTS_TO_RUN

* add back declaration of TESTS_TO_RUN

* try removing spaces when declaring TESTS_TO_RUN

* add dollar sign to echo statement for TESTS_TO_RUN

* try resetting value of TESTS_TO_RUN in if statements

* try removing spaces for setting auraLSP

* try removing spaces for visualforceLSP

* add plus sign in visualforceLSP

* replace plus sign with semicolon

* switch back from semicolons to plus signs

* remove semicolon and quotes from end of test file names

* make aura case the same as the other two cases

* remove semicolon from auraLSP case

* try to fix Windows issue by removing semicolon from if statements

* force windows to use bash shell

* add env.ENV_TESTS_TO_RUN variable

* check if the error is caused in line 106

* try setting env variable without dollar sign and curlies

* try using Write-Host to set env variable

* try removing env. prefix

* use GITHUB_ENV to set environment variable

* change logging

* put back environment variable in build

* try to set env.ENV_TESTS_TO_RUN again

* try another method to set environment variable

* try exporting TESTS_TO_RUN to environment variable

* try overriding env.ENV_TESTS_TO_RUN to abcdefg

* another method

* try adding double quotes around GITHUB_ENV

* uncomment line 106 to set environment variable

* move printing value of env.ENV_TESTS_TO_RUN to next step

* set environment variable value back to TESTS_TO_RUN

* remove declaration of environment variable in build

* uncomment remaining steps to run full e2e tests

* make changes to apexE2E.ymlfor Apex-related tests to run on Remote

* make changes to coreE2E.ymlfor Core-related tests to run on Remote

* fix: if no e2e tests are checked then run all of them

* chore: remove quotes

* chore: add logging

* chore: remove spaces

* chore: formatting

* chore: more formatting

* chore: uncomment remaining steps

* fix: change apexE2E.yml and coreE2E.yml to run all tests if no boxes are selected

* feat: run LSP E2E tests if the box is selected

* chore: uncomment remaining steps in e2e.yml

* chore: change workflow from lspE2E.yml to e2e.yml

* fix: move checkbox checking to outside steps

* fix: add on.workflow_call to lspE2E.yml

* fix: change e2e.yml to lspE2E.yml

* chore: comment out run statement

* fix: change lspE2E.yml back to e2e.yml

* fix: in lspE2E.yml select all tests on workflow call + in e2e.yml uncomment build

* chore: comment out build

* fix: add a check to only run lspE2E if the box is selected

* feat: run apex e2e tests if selected and run core e2e tests if selected

* fix: add on.workflow_call triggers for apexE2E.yml and coreE2E.yml

* fix: inherit secrets

* chore: make checked tests run in parallel

* chore: clean up code - jobs should run in parallel in e2e.yml

* fix: pass automationBranch from e2e.yml to the three workflows that it calls

* fix: (1) have all boxes be automatically checked (2) do not run all tests if nothing is checked

* refactor: make lspE2E.yml run each E2E test as a separate job by calling runE2ETest.yml

* fix: testToRun parameter in each job of lspE2E.yml

* chore: formatting

* refactor: change apexE2E.yml and coreE2E.yml to run E2E tests in parallel by calling runE2ETest.yml

* chore: rename jobs

* fix: setup java

* chore: rename e2e.yml to pull from forcedotcom/salesforcedx-vscode

* chore: rename back to e2e.yml after successful merge

* chore: remove lwcLSP and orgBrowser options from inputs

* chore: alphabetize inputs in e2e.yml

* chore: remove default of 'placeholder text' for testToRun from runE2ETest.yml

.github/workflows/apexE2E.yml

@@ -0,0 +1,111 @@
+name: Apex End to End Tests
+
+on:
+
+  workflow_dispatch:
+    inputs:
+      automationBranch:
+        description: 'Set the branch to use for automation tests'
+        required: false
+        default: 'develop'
+        type: string
+      apexLsp:
+        description: 'apexLsp.e2e.ts'
+        required: false
+        default: true
+        type: boolean
+      apexReplayDebugger:
+        description: 'apexReplayDebugger.e2e.ts'
+        required: false
+        default: true
+        type: boolean
+      debugApexTests:
+        description: 'debugApexTests.e2e.ts'
+        required: false
+        default: true
+        type: boolean
+      runApexTests:
+        description: 'runApexTests.e2e.ts'
+        required: false
+        default: true
+        type: boolean
+      trailApexReplayDebugger:
+        description: 'trailApexReplayDebugger.e2e.ts'
+        required: false
+        default: true
+        type: boolean
+
+  workflow_call:
+    inputs:
+      automationBranch:
+        description: 'Set the branch to use for automation tests'
+        required: false
+        default: 'develop'
+        type: string
+      apexLsp:
+        description: 'apexLsp.e2e.ts'
+        required: false
+        default: true
+        type: boolean
+      apexReplayDebugger:
+        description: 'apexReplayDebugger.e2e.ts'
+        required: false
+        default: true
+        type: boolean
+      debugApexTests:
+        description: 'debugApexTests.e2e.ts'
+        required: false
+        default: true
+        type: boolean
+      runApexTests:
+        description: 'runApexTests.e2e.ts'
+        required: false
+        default: true
+        type: boolean
+      trailApexReplayDebugger:
+        description: 'trailApexReplayDebugger.e2e.ts'
+        required: false
+        default: true
+        type: boolean
+
+jobs:
+
+  apexLSP:
+    if: ${{ inputs.apexLsp }}
+    uses: ./.github/workflows/runE2ETest.yml
+    secrets: inherit
+    with:
+      automationBranch: ${{ inputs.automationBranch }}
+      testToRun: 'apexLsp.e2e.ts'
+
+  apexReplayDebugger:
+    if: ${{ inputs.apexReplayDebugger }}
+    uses: ./.github/workflows/runE2ETest.yml
+    secrets: inherit
+    with:
+      automationBranch: ${{ inputs.automationBranch }}
+      testToRun: 'apexReplayDebugger.e2e.ts'
+
+  debugApexTests:
+    if: ${{ inputs.debugApexTests }}
+    uses: ./.github/workflows/runE2ETest.yml
+    secrets: inherit
+    with:
+      automationBranch: ${{ inputs.automationBranch }}
+      testToRun: 'debugApexTests.e2e.ts'
+
+  runApexTests:
+    if: ${{ inputs.runApexTests }}
+    uses: ./.github/workflows/runE2ETest.yml
+    secrets: inherit
+    with:
+      automationBranch: ${{ inputs.automationBranch }}
+      testToRun: 'runApexTests.e2e.ts'
+
+  trailApexReplayDebugger:
+    if: ${{ inputs.trailApexReplayDebugger }}
+    uses: ./.github/workflows/runE2ETest.yml
+    secrets: inherit
+    with:
+      automationBranch: ${{ inputs.automationBranch }}
+      testToRun: 'trailApexReplayDebugger.e2e.ts'

.github/workflows/coreE2E.yml

@@ -0,0 +1,147 @@
+name: Core End to End Tests
+
+on:
+
+  workflow_dispatch:
+    inputs:
+      automationBranch:
+        description: 'Set the branch to use for automation tests'
+        required: false
+        default: 'develop'
+        type: string
+      anInitialSuite:
+        description: 'anInitialSuite.e2e.ts'
+        required: false
+        default: true
+        type: boolean
+      authentication:
+        description: 'authentication.e2e.ts'
+        required: false
+        default: true
+        type: boolean
+      deployAndRetrieve:
+        description: 'deployAndRetrieve.e2e.ts'
+        required: false
+        default: true
+        type: boolean
+      manifestBuilder:
+        description: 'manifestBuilder.e2e.ts'
+        required: false
+        default: true
+        type: boolean
+      pushAndPull:
+        description: 'pushAndPull.e2e.ts'
+        required: false
+        default: true
+        type: boolean
+      sObjectsDefinitions:
+        description: 'sObjectsDefinitions.e2e.ts'
+        required: false
+        default: true
+        type: boolean
+      templates:
+        description: 'templates.e2e.ts'
+        required: false
+        default: true
+        type: boolean
+
+  workflow_call:
+    inputs:
+      automationBranch:
+        description: 'Set the branch to use for automation tests'
+        required: false
+        default: 'develop'
+        type: string
+      anInitialSuite:
+        description: 'anInitialSuite.e2e.ts'
+        required: false
+        default: true
+        type: boolean
+      authentication:
+        description: 'authentication.e2e.ts'
+        required: false
+        default: true
+        type: boolean
+      deployAndRetrieve:
+        description: 'deployAndRetrieve.e2e.ts'
+        required: false
+        default: true
+        type: boolean
+      manifestBuilder:
+        description: 'manifestBuilder.e2e.ts'
+        required: false
+        default: true
+        type: boolean
+      pushAndPull:
+        description: 'pushAndPull.e2e.ts'
+        required: false
+        default: true
+        type: boolean
+      sObjectsDefinitions:
+        description: 'sObjectsDefinitions.e2e.ts'
+        required: false
+        default: true
+        type: boolean
+      templates:
+        description: 'templates.e2e.ts'
+        required: false
+        default: true
+        type: boolean
+
+jobs:
+
+  anInitialSuite:
+    if: ${{ inputs.anInitialSuite }}
+    uses: ./.github/workflows/runE2ETest.yml
+    secrets: inherit
+    with:
+      automationBranch: ${{ inputs.automationBranch }}
+      testToRun: 'anInitialSuite.e2e.ts'
+
+  authentication:
+    if: ${{ inputs.authentication }}
+    uses: ./.github/workflows/runE2ETest.yml
+    secrets: inherit
+    with:
+      automationBranch: ${{ inputs.automationBranch }}
+      testToRun: 'authentication.e2e.ts'
+
+  deployAndRetrieve:
+    if: ${{ inputs.deployAndRetrieve }}
+    uses: ./.github/workflows/runE2ETest.yml
+    secrets: inherit
+    with:
+      automationBranch: ${{ inputs.automationBranch }}
+      testToRun: 'deployAndRetrieve.e2e.ts'
+
+  manifestBuilder:
+    if: ${{ inputs.manifestBuilder }}
+    uses: ./.github/workflows/runE2ETest.yml
+    secrets: inherit
+    with:
+      automationBranch: ${{ inputs.automationBranch }}
+      testToRun: 'manifestBuilder.e2e.ts'
+
+  pushAndPull:
+    if: ${{ inputs.pushAndPull }}
+    uses: ./.github/workflows/runE2ETest.yml
+    secrets: inherit
+    with:
+      automationBranch: ${{ inputs.automationBranch }}
+      testToRun: 'pushAndPull.e2e.ts'
+
+  sObjectsDefinitions:
+    if: ${{ inputs.sObjectsDefinitions }}
+    uses: ./.github/workflows/runE2ETest.yml
+    secrets: inherit
+    with:
+      automationBranch: ${{ inputs.automationBranch }}
+      testToRun: 'sObjectsDefinitions.e2e.ts'
+
+  templates:
+    if: ${{ inputs.templates }}
+    uses: ./.github/workflows/runE2ETest.yml
+    secrets: inherit
+    with:
+      automationBranch: ${{ inputs.automationBranch }}
+      testToRun: 'templates.e2e.ts'

.github/workflows/e2e.yml

@@ -7,103 +7,41 @@ on:
         required: false
         default: 'develop'
         type: string
-      specFiles:
-        description: 'Run a specific spec file. Runs all by default.'
+      apexE2ETests:
+        description: 'Apex E2E Tests'
         required: false
-        type: choice
-        options:
-          - '*.e2e.ts'
-          - 'anInitialSuite.e2e.ts'
-          - 'apexLsp.e2e.ts'
-          - 'apexReplayDebugger.e2e.ts'
-          - 'auraLsp.e2e.ts'
-          - 'authentication.e2e.ts'
-          - 'debugApexTests.e2e.ts'
-          - 'deployAndRetrieve.e2e.ts'
-          - 'lwcLsp.e2e.ts'
-          - 'manifestBuilder.e2e.ts'
-          - 'orgBrowser.e2e.ts'
-          - 'pushAndPull.e2e.ts'
-          - 'runApexTests.e2e.ts'
-          - 'sObjectsDefinitions.e2e.ts'
-          - 'templates.e2e.ts'
-          - 'trailApexReplayDebugger.e2e.ts'
-          - 'visualforceLsp.e2e.ts'
+        default: true
+        type: boolean
+      coreE2ETests:
+        description: 'Core E2E Tests'
+        required: false
+        default: true
+        type: boolean
+      lspE2ETests:
+        description: 'LSP E2E Tests'
+        required: false
+        default: true
+        type: boolean
 
 jobs:
-  build:
-    runs-on: ${{ matrix.os }}
-    strategy:
-      fail-fast: false
-      matrix:
-        os: [macos-latest, windows-latest, ubuntu-latest]
-        nodeVersion:
-          - 16.15.1
-        vscodeVersion:
-          - stable
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v3
-        with:
-          path: ./salesforcedx-vscode
-          ref: ${{ github.event.ref }}
-      - name: Setup node
-        uses: actions/setup-node@v3
-        with:
-          node-version: ${{ matrix.nodeVersion }}
-          cache: npm
-          cache-dependency-path: |
-            salesforcedx-vscode/package-lock.json
-            salesforcedx-vscode-automation-tests/package-lock.json
-      - name: Setup java
-        uses: actions/setup-java@v3
-        with:
-          distribution: 'zulu'
-          java-version: '11'
-      - name: Clone automation tests
-        uses: actions/checkout@v3
-        with:
-          repository: forcedotcom/salesforcedx-vscode-automation-tests
-          path: salesforcedx-vscode-automation-tests
-          ref: ${{ inputs.automationBranch }}
-      - name: Install Extension Dependencies
-        run: |
-          npm install
-          npm run compile
-        working-directory: salesforcedx-vscode
-      - name: Install Test Dependencies
-        run: |
-          npm install
-        working-directory: salesforcedx-vscode-automation-tests
-      - name: Install the SFDX CLI
-        run: npm install -g sfdx-cli
-      - name: Verify CLI
-        shell: bash
-        run: |
-          set -e
-          sfdx version
-          SFDX_CLI_VERSION=$(sfdx version)
-          if [[ ((`echo $SFDX_CLI_VERSION | grep -c "sfdx-cli/"` > 0))]]
-          then
-            echo "sfdx-cli installed -" $SFDX_CLI_VERSION
-          else
-            echo "The sfdx-cli installation could not be verified"
-            exit 1
-          fi
-      - name: Run headless test
-        uses: coactions/setup-xvfb@b6b4fcfb9f5a895edadc3bc76318fae0ac17c8b3
-        with:
-          run: |
-            npm run setup
-            npm run automation-tests
-          working-directory: salesforcedx-vscode-automation-tests
-        env:
-          VSCODE_VERSION: ${{ matrix.vscodeVersion }}
-          SPEC_FILES: ${{ inputs.specFiles || '*.e2e.ts' }}
-          SFDX_AUTH_URL: ${{ secrets.SFDX_AUTH_URL_E2E }}
-          ORG_ID: ${{ secrets.ORG_ID_E2E }}
-      - uses: actions/upload-artifact@v3
-        if: failure()
-        with:
-          name: screenshots
-          path: ./salesforcedx-vscode-automation-tests/screenshots
+
+  run_LSP_E2E_tests_if_selected:
+    if: ${{ inputs.lspE2ETests }}
+    uses: ./.github/workflows/lspE2E.yml
+    secrets: inherit
+    with:
+      automationBranch: ${{ inputs.automationBranch }}
+
+  run_Apex_E2E_tests_if_selected:
+    if: ${{ inputs.apexE2ETests }}
+    uses: ./.github/workflows/apexE2E.yml
+    secrets: inherit
+    with:
+      automationBranch: ${{ inputs.automationBranch }}
+
+  run_Core_E2E_tests_if_selected:
+    if: ${{ inputs.coreE2ETests }}
+    uses: ./.github/workflows/coreE2E.yml
+    secrets: inherit
+    with:
+      automationBranch: ${{ inputs.automationBranch }}