Add accessibility test job running against home page (#14773)

.github/workflows/a11y_tests.yml

@@ -0,0 +1,74 @@
+# Workflow that runs accessibility tests once per day against dev infra.
+
+name: Accessibility Tests
+run-name: Accessibility Tests for ${{ github.sha }}
+env:
+  SLACK_CHANNEL_ID: CBX0KH5GA # #www-notify in MoCo Slack
+  SLACK_BOT_TOKEN: ${{secrets.SLACK_BOT_TOKEN_FOR_MEAO_NOTIFICATIONS_APP}}
+on:
+  schedule:
+    - cron: "0 15 * * *" # 3pm daily
+  workflow_dispatch:
+    inputs:
+      mozorg_service_hostname:
+        description: The root URL of the Mozorg service to run tests against. eg "https://www.mozilla.org/"
+        required: true
+
+jobs:
+  notify-of-test-run-start:
+    if: github.repository == 'mozilla/bedrock'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Notify via Slack that tests are starting
+        uses: ./.github/actions/slack
+        with:
+          env_name: test
+          label: "Accessibility Tests [${{ github.sha }}]"
+          status: info
+          channel_id: ${{ env.SLACK_CHANNEL_ID }}
+          slack_bot_token: ${{ env.SLACK_BOT_TOKEN }}
+          ref: ${{ github.sha }}
+          message: "Accessibility Tests started"
+
+  accessibility-tests:
+    runs-on: ubuntu-latest
+    needs: notify-of-test-run-start
+    env:
+      PLAYWRIGHT_BASE_URL: ${{ github.event.inputs.mozorg_service_hostname || 'https://dev.bedrock.nonprod.webservices.mozgcp.net/' }}
+      CI: true
+      CI_JOB_ID: ${{ github.run_id }}
+      DRIVER: ""
+      LABEL: accessibility-tests
+    steps:
+    - uses: actions/checkout@v4
+    - uses: actions/setup-node@v4
+      with:
+        node-version: 20
+    - name: Install dependencies
+      run: cd tests/playwright && npm ci && npm run install-deps
+    - name: Run Playwright tests
+      run: npm run a11y-tests
+    - uses: actions/upload-artifact@v4
+      if: ${{ !cancelled() }}
+      with:
+        name: a11y-report-${{ github.sha }}
+        path: a11y-report/
+        retention-days: 30
+
+  notify-of-test-run-completion:
+    if: github.repository == 'mozilla/bedrock' && always()
+    runs-on: ubuntu-latest
+    needs: [notify-of-test-run-start, accessibility-tests]
+    steps:
+      - uses: actions/checkout@v4
+      - name: Notify via Slack of test-run outcome
+        uses: ./.github/actions/slack
+        with:
+          env_name: test
+          label: "Playwright tests [${{ github.sha }}]"
+          status: ${{ needs.accessibility-tests.result }}
+          channel_id: ${{ env.SLACK_CHANNEL_ID }}
+          slack_bot_token: ${{ env.SLACK_BOT_TOKEN }}
+          ref: ${{ github.sha }}
+          message: "Playwright tests completed. Status: ${{ needs.accessibility-tests.result }}"

.github/workflows/integration_tests.yml

@@ -124,7 +124,7 @@ jobs:
     - name: Install dependencies
       run: cd tests/playwright && npm ci && npm run install-deps
     - name: Run Playwright tests
-      run: cd tests/playwright && npx playwright test
+      run: cd tests/playwright && npm run functional-tests
 
   notify-of-test-run-completion:
     if: always()

.gitignore

@@ -59,6 +59,7 @@ tests/functional/results.html
 tests/unit/coverage
 tests/unit/dist/
 tests/playwright/test-results/
+tests/playwright/a11y-report/
 Thumbs.db
 tmp/*
 venv

docs/testing.rst

@@ -8,103 +8,127 @@
 Front-end testing
 =================
 
-Bedrock runs a suite of front-end `Jasmine`_ behavioral/unit tests, which use
-`Jasmine Browser Runner`_ as a test runner. We also have a suite of functional tests using
-`Selenium`_ and `pytest`_. This allows us to emulate users interacting with a
-real browser. All these test suites live in the ``tests`` directory. To run the tests locally,
-you must also first download `geckodriver`_  and `chromedriver`_ and make it available
-in your system path. You can alternatively specify the path to geckodriver and chromedriver
-using the command line (see the `pytest-selenium documentation`_ for more information).
+Bedrock runs several different types of front-end tests to ensure that the site is working
+correctly and that new changes don't break existing functionality.
 
-The ``tests`` directory comprises of:
+- `Jasmine`_ unit/behavioral tests are used to test JavaScript code that runs in the browser.
+  These tests are run against both Firefox and Chrome browsers via a GitHub action, which is
+  triggered against all pull requests and commits to the main branch.
+- `Playwright`_ integration tests are used to run end-to-end tests in a real browser
+  environment. These tests are run automatically post-merge as part of our CI deployment
+  process against dev, stage, and prod. Playwright tests are run against Firefox, Chromium,
+  and Webkit headless browsers for cross-engine coverage.
+- `Axe Core`_ accessibility (a11y) tests are run as sub-suite of the Playwright tests, and
+  are used to check for a11y issues on key pages of the site, as well as common global
+  elements such as the site header and footer. These tests are not part of our CI deployment
+  pipeline, but are instead run via a GitHub action as a daily job against www-dev.
+- `Selenium`_ integration tests are bedrock legacy test suite, which will eventually be
+  replaced by Playwright. These tests are run against Firefox, Chrome, and Internet Explorer
+  (via a mix of both a local Selenium Grid and Sauce Labs), as part of our CI pipeline, and
+  run alongside the Playwright tests. New tests should be written using Playwright, but we
+  will continue to run the Selenium tests until they are all ported over.
 
-* ``/functional`` contains pytest tests.
-* ``/pages`` contains Python page objects.
-* ``/unit`` contains the Jasmine tests and Jasmine Browser Runner config file.
+The test specs for all of the above suites can be found in the root ``./tests`` directory:
 
-.. note::
+* ``./tests/functional/`` for Selenium tests.
+* ``./tests/pages/`` for Selenium page objects.
+* ``./tests/playwright/`` Playwright and Axe tests.
+* ``./tests/unit/`` for Jasmine tests.
 
-    As of July 2024, bedrock has started to use `Playwright`_ for some integration tests, with
-    the aim of replacing Selenium in the near future. Playwright is a Node.js library for
-    automating modern browsers, and aims to make it easier to write, run, and debug tests.
-    This documentation will be updated as we transition to Playwright, but for now you can
-    find more information in the `README`_ file in the ``tests/playwright`` directory.
+Automating the browser
+======================
 
-Installation
-------------
+Jasmine, Playwright and Selenium all require a browser to run. In order to automate browsers
+such as Firefox and Chrome, you also need to have the appropriate drivers installed. To
+download ``geckodriver`` and ``chromedriver`` and have them ready to run in your system,
+there are a couple of ways:
 
-First follow the :ref:`installation instructions for bedrock<install>`, which
-will install the dependencies required to run the various front-end test suites.
+Download `geckodriver`_ and add it to your system path:
 
-To download geckodriver and chromedriver and have it ready to run in your system, there are a couple of ways:
+    .. code-block:: bash
 
-- `Download geckdriver <https://github.com/mozilla/geckodriver/releases/latest>`_ and add it to your system path:
+        cd /path/to/your/downloaded/files/
+        mv geckodriver /usr/local/bin/
 
-        .. code-block:: bash
+If you're on MacOS, download ``geckodriver`` directly using Homebrew, which automatically
+places it in your system path:
 
-            cd /path/to/your/downloaded/files/
-            mv geckodriver /usr/local/bin/
+    .. code-block:: bash
 
-- If you're on MacOS, download geckodriver directly using Homebrew, which automatically places it in your system path:
+        brew install geckodriver
 
-        .. code-block:: bash
 
-            brew install geckodriver
+Download `chromedriver`_ and add it to your system path:
 
+    .. code-block:: bash
 
-- `Download chromedriver <https://chromedriver.chromium.org/downloads>`_ and add it to your system path:
+        cd /path/to/your/downloaded/files/
+        mv chromedriver /usr/local/bin/
 
-        .. code-block:: bash
+If you're on MacOS, download ``chromedriver`` directly using Homebrew/Cask, which
+automatically places it in your system path:
 
-            cd /path/to/your/downloaded/files/
-            mv chromedriver /usr/local/bin/
+    .. code-block:: bash
 
-- If you're on MacOS, download chromedriver directly using Homebrew/Cask, which automatically places it in your system path:
+        brew tap homebrew/cask
+        brew cask install chromedriver
 
-        .. code-block:: bash
+Running Jasmine tests
+=====================
 
-            brew tap homebrew/cask
-
-            brew cask install chromedriver
+Jasmine and its dependencies are installed via npm and are included when you run
+``make preflight`` to install bedrock's main dependencies.
 
+.. code-block:: bash
 
-Running Jasmine tests using Jasmine Browser Runner
---------------------------------------------------
+    $ make preflight
 
-To perform a single run of the Jasmine test suite using Firefox and Chrome,
-first make sure you have both browsers installed locally, and then activate
-your bedrock virtual env.
+Next, make sure you activate your bedrock virtual env.
 
 .. code-block:: bash
 
     $ pyenv activate bedrock
 
-You can then run the tests with the following command:
+You can then run the full suite of Jasmine tests with the following command:
 
 .. code-block:: bash
 
     $ npm run test
 
-This will run all our front-end linters and formatting checks before running
+This will also run all our front-end linters and formatting checks before running
 the Jasmine test suite. If you only want to run the tests themselves, you can
 run:
 
 .. code-block:: bash
 
-    $ npm run test
+    $ npm run jasmine
+
+Writing Jasmine tests
+---------------------
 
 See the `Jasmine`_ documentation for tips on how to write JS behavioral or unit
 tests. We also use `Sinon`_ for creating test spies, stubs and mocks.
 
-Running functional tests
-------------------------
+Debugging Jasmine tests
+-----------------------
 
-.. Note::
+If you need to debug Jasmine tests, you can also run them interactively in the
+browser using the following command:
+
+.. code-block:: bash
+
+    $ npm run jasmine-serve
+
+Running Playwright tests
+========================
+
+Running Selenium tests
+======================
 
-  Before running the functional tests, please make sure to follow the bedrock
-  :ref:`installation docs<install>`, including the database sync that is needed
-  to pull in external data such as event/blog feeds etc. These are required for
-  some of the tests to pass.
+Before running the Selenium tests, please make sure to follow the bedrock
+:ref:`installation docs<install>`, including the database sync that is needed
+to pull in external data such as event/blog feeds etc. These are required for
+some of the tests to pass.
 
 To run the full functional test suite against your local bedrock instance in
 Mozorg mode:
@@ -322,7 +346,7 @@ Guidelines for writing functional tests
 See also the `Web QA style guide`_ for Python based testing.
 
 Testing Basket email forms
---------------------------
+==========================
 
 When writing functional tests for front-end email newsletter forms that submit to
 `Basket`_, we have some special case email addresses that can be used just for testing:
@@ -346,6 +370,7 @@ via product details are well formed and return valid 200 responses.
 .. _Jasmine: https://jasmine.github.io/index.html
 .. _Jasmine Browser Runner: https://jasmine.github.io/setup/browser.html
 .. _Sinon: http://sinonjs.org/
+.. _Axe Core: https://github.com/dequelabs/axe-core
 .. _Selenium: http://docs.seleniumhq.org/
 .. _pytest: http://pytest.org/latest/
 .. _pytest documentation: http://pytest.org/latest/