-
Notifications
You must be signed in to change notification settings - Fork 92
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
e2e-test: convert to playwright runner (#5234)
### Summary This PR introduces a significant update to our e2e testing framework by migrating from Mocha to Playwright as our test runner. The new setup not only achieves feature parity with the existing Mocha runner but also adds enhanced capabilities, offering an improved developer experience and streamlined debugging. For an in-depth explanation of the changes and rationale, please refer to the [POC Document](https://positpbc.atlassian.net/wiki/spaces/POSITRON/pages/1224999131/Proof+of+Concept+Playwright). **Key notes:** * Replaced Mocha with Playwright for e2e test execution. * All existing Positron tests have been converted to Playwright syntax and are fully operational. * Updated all CI workflows and added publishing of HTML reports to S3 bucket ### QA Notes Example Workflow Runs: * [Full Test Suite](https://github.com/posit-dev/positron/actions/runs/11933152185) * [Windows Test Suite](https://github.com/posit-dev/positron/actions/runs/11933795345) * [Tests against Latest Build](https://github.com/posit-dev/positron/actions/runs/11933100000) ### Screenshots <img width="362" alt="Screenshot 2024-11-18 at 11 37 22 AM" src="https://github.com/user-attachments/assets/1483e0f1-664b-4de9-98b9-0638a169d881">
- Loading branch information
Showing
113 changed files
with
4,510 additions
and
4,957 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,117 @@ | ||
| name: "E2E Report" | ||
| description: "Generate and upload E2E reports, and notify stakeholders" | ||
| inputs: | ||
| aws-s3-bucket: | ||
| description: "AWS S3 bucket name" | ||
| required: true | ||
| aws-role: | ||
| description: "AWS role to assume" | ||
| required: true | ||
| junit-pattern: | ||
| description: "Pattern to match JUnit reports" | ||
| default: "junit-report-*" | ||
| blob-pattern: | ||
| description: "Pattern to match blob reports" | ||
| default: "blob-report-*" | ||
| slack-token: | ||
| description: "Slack token for notifications" | ||
| required: true | ||
| slack-channel: | ||
| description: "Slack channel for notifications" | ||
| required: true | ||
| slack-title: | ||
| description: "Slack title for the test suite" | ||
| default: "Test Suite Results" | ||
| testrail-api-key: | ||
| description: "TestRail API Key" | ||
| required: true | ||
| github-token: | ||
| description: "GitHub token for API calls" | ||
| required: true | ||
| testrail-project: | ||
| description: "TestRail project name" | ||
| default: "Positron" | ||
| testrail-title: | ||
| description: "Title for TestRail runs" | ||
| default: "E2E Test Run" | ||
|
|
||
| runs: | ||
| using: "composite" | ||
| steps: | ||
| - name: Checkout Repository | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Setup Node.js | ||
| uses: actions/setup-node@v4 | ||
| with: | ||
| node-version-file: .nvmrc | ||
|
|
||
| - name: Install AWS CLI | ||
| shell: bash | ||
| run: | | ||
| apt-get update && apt-get install -y unzip python3-pip | ||
| curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o /tmp/awscliv2.zip | ||
| unzip -q /tmp/awscliv2.zip -d /tmp | ||
| /tmp/aws/install -i $HOME/aws-cli -b $HOME/bin | ||
| rm -rf /tmp/aws /tmp/awscliv2.zip | ||
| - name: Add AWS CLI to PATH | ||
| shell: bash | ||
| run: echo "$HOME/bin" >> $GITHUB_PATH | ||
|
|
||
| - name: Verify AWS CLI installation | ||
| shell: bash | ||
| run: aws --version | ||
|
|
||
| - name: Download Blob Reports | ||
| uses: actions/download-artifact@v4 | ||
| with: | ||
| path: all-blob-reports | ||
| pattern: ${{ inputs.blob-pattern }} | ||
| merge-multiple: true | ||
|
|
||
| - name: Merge Blobs into HTML Report | ||
| shell: bash | ||
| run: npx playwright merge-reports --reporter html ./all-blob-reports | ||
|
|
||
| - name: Upload Playwright Report to S3 | ||
| uses: ./.github/actions/upload-report-to-s3 | ||
| with: | ||
| role-to-assume: ${{ inputs.aws-role }} | ||
|
|
||
| - name: Send HTML Report URL to GitHub Summary | ||
| shell: bash | ||
| run: | | ||
| REPORT_URL="https://d38p2avprg8il3.cloudfront.net/${{ env.REPORT_DIR }}/index.html" | ||
| echo "Report URL: $REPORT_URL" | ||
| echo "📄 [Playwright Report]($REPORT_URL) <br>" > $GITHUB_STEP_SUMMARY | ||
| - name: Download JUnit Reports | ||
| uses: actions/download-artifact@v4 | ||
| with: | ||
| path: all-junit-reports | ||
| pattern: ${{ inputs.junit-pattern }} | ||
| merge-multiple: true | ||
|
|
||
| - name: Merge JUnit Reports | ||
| shell: bash | ||
| run: npm install -g junit-report-merger && npx jrm junit.xml "all-junit-reports/*.xml" | ||
|
|
||
| - name: Install trcli | ||
| shell: bash | ||
| run: apt-get update && apt-get install -y python3-pip && pip3 install trcli | ||
|
|
||
| - name: Upload Test Results to TestRail | ||
| shell: bash | ||
| run: | | ||
| TESTRAIL_TITLE="$(date +'%Y-%m-%d') ${{ inputs.testrail-title }} - $GITHUB_REF_NAME" | ||
| echo "TESTRAIL_TITLE=$TESTRAIL_TITLE" >> $GITHUB_ENV | ||
| trcli --host "https://posit.testrail.io/" --project "${{ inputs.testrail-project }}" --username testrailautomation@posit.co --key "${{ inputs.testrail-api-key }}" parse_junit --file "./junit.xml" --case-matcher name --title "$TESTRAIL_TITLE" --close-run | ||
| - name: Send Slack Notification | ||
| uses: testlabauto/action-test-results-to-slack@v0.0.6 | ||
| with: | ||
| github_token: ${{ inputs.github-token }} | ||
| slack_token: ${{ inputs.slack-token }} | ||
| slack_channel: ${{ inputs.slack-channel }} | ||
| suite_name: ${{ inputs.slack-title }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| name: "Upload Playwright Report to S3" | ||
| description: "Configures AWS credentials and uploads the Playwright report to S3" | ||
| inputs: | ||
| role-to-assume: | ||
| description: "The AWS role to assume" | ||
| required: true | ||
|
|
||
| runs: | ||
| using: "composite" | ||
| steps: | ||
| - name: Configure AWS Credentials | ||
| uses: aws-actions/configure-aws-credentials@v4 | ||
| if: ${{ !cancelled() }} | ||
| with: | ||
| role-to-assume: ${{ inputs.role-to-assume }} | ||
| aws-region: "us-east-1" | ||
|
|
||
| - name: Upload Playwright Report to S3 Bucket | ||
| if: ${{ !cancelled() }} | ||
| shell: bash | ||
| run: | | ||
| aws s3 cp playwright-report/. s3://positron-test-reports/playwright-report-${{ github.run_id }} --recursive |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.