Add a11y test job running against key pages (#14773) #14862
Conversation
alexgibson
force-pushed
the
playwright-ally
branch
3 times, most recently
from
4038107
to
63085cc
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #14862 +/- ##
=======================================
Coverage 77.33% 77.33%
=======================================
Files 161 161
Lines 8344 8344
=======================================
Hits 6453 6453
Misses 1891 1891 ☔ View full report in Codecov by Sentry. |
alexgibson
force-pushed
the
playwright-ally
branch
3 times, most recently
from
32d831e
to
169e16f
Compare
alexgibson
force-pushed
the
playwright-ally
branch
from
169e16f
to
cdd77a9
Compare
alexgibson
force-pushed
the
playwright-ally
branch
2 times, most recently
from
910c5ee
to
1b3c9ce
Compare
alexgibson
changed the title
alexgibson
force-pushed
the
playwright-ally
branch
7 times, most recently
from
de41ea2
to
8927c7b
Compare
There was a problem hiding this comment.
Do the tests pass currently? I'm getting (besides color contrast, heading order etc.):
CRITICAL: Elements Must Only Use Allowed ARIA Attributes More Info
Fix the following: ARIA attribute is not allowed:aria-expanded="false"for<div class="c-navigation-items" id="c-navigation-items" data-testid="navigation-menu-items" aria-expanded="false">
(So this would be better resolved first? Adding a button role, or marking up menu, menubar, menuitem roles and using aria-haspopup too?)
Also, nit:
Oh this will absolutely fail at first and that's very intentional. We'll likely have a lot of small a11y issues that the tests identify, but we can get to those after we have this job running and reporting correctly. |
alexgibson
force-pushed
the
playwright-ally
branch
5 times, most recently
from
394b106
to
354fb8e
Compare
alexgibson
changed the title
alexgibson
force-pushed
the
playwright-ally
branch
from
354fb8e
to
a486eb6
Compare
alexgibson
force-pushed
the
playwright-ally
branch
from
a486eb6
to
4c4d612
Compare
alexgibson
force-pushed
the
playwright-ally
branch
from
4c4d612
to
95f6e59
Compare
alexgibson
added
Needs Review
| tag: '@a11y' | ||
| }, | ||
| () => { | ||
| test.use({ viewport: { width: 360, height: 780 } }); |
There was a problem hiding this comment.
Excited to have mobile specific tests 🔥
|
|
||
| // Open Firefox menu | ||
| await expect(firefoxMenu).not.toBeVisible(); | ||
| await firefoxLink.click(); |
There was a problem hiding this comment.
Enhancement/Thought: Since these are a11y tests it would be good to test that keyboard navigation works but it's not a blocker for this PR.
There was a problem hiding this comment.
I'd probably consider those to be functional tests as opposed to something specific to the a11y test job, but it would be a good bit of extra coverage agreed 👍
One-line summary
Adds a new Playwright a11y test suite, powered via Axe Core.
Significant changes and points to review
These initial tests are against a small-ish set of our key, most high-traffic pages. The test runs will fail for now and that's fine. They will not block deployments, and will instead run once per-day against www-dev and report via Slack. We can expand the number of pages this test suite covers as we work through issues it identifies.
Issue / Bugzilla link
#14773
Testing
cd tests/playwright && npm ci && npm run install-depsnpm run a11y-testsOnce the test run completes, it should fail (which is expected).
The resulting failures will be captured as HTML reports and saved in
./tests/playwright/test-reports-a11y/. You can open these files in your browser and read more information about each issue.