Skip to content

test: add advisory axe-core accessibility sweep for every component - #736

Open
johnleider wants to merge 3 commits into
masterfrom
worktree-axe-audit
Open

test: add advisory axe-core accessibility sweep for every component#736
johnleider wants to merge 3 commits into
masterfrom
worktree-axe-audit

Conversation

@johnleider

@johnleider johnleider commented Jul 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Closes #732.

What lands

  • axe-core in the workspace catalog + packages/0 devDependencies.
  • packages/0/src/components/a11y.browser.test.ts — a sweep over all 40 shipped components, running in the v0:browser project's real Chromium.
  • packages/0/src/components/fixtures/ — one .vue per component holding the usage its own @example JSDoc documents, plus the audit/report helper.
  • An a11y CI job, continue-on-error: true, that tees the breakdown into the GitHub step summary.
  • A correction to guide/features/accessibility.md — see "Not vitest-axe" below.

No violations are fixed here. The point of the first pass is the number.

The number

axe: 10 violations (13 elements) across 7 components, 8 distinct rules
rule impact elements components
aria-required-parent critical 3 Select
aria-allowed-attr critical 2 Selection
button-name critical 2 Select
label critical 1 NumberField
aria-input-field-name serious 2 Rating, Slider
aria-progressbar-name serious 1 Progress
nested-interactive serious 1 Rating
region moderate 1 Portal

Three of these are the open batch, found independently:

Five are new:

  • aria-required-parent on Select — three role="option" elements whose nearest ancestor is not a listbox/group. aria-controls points at the listbox id, but the options are not inside it.
  • nested-interactive + aria-input-field-name on Rating — RatingRoot renders role="slider" and Rating.Item as="button" puts focusable buttons inside it. The slider also has no accessible name.
  • aria-input-field-name on Slider — role="slider" with aria-valuetext but no aria-label/aria-labelledby.
  • label on NumberField — the spinbutton <input> has no associated label. Adjacent to fix(NumberField): add labelledBy prop for aria-labelledby on spinbutton #640, which adds labelledBy for the aria-labelledby case; this is the plain <label> case.
  • region on Portal — content teleported to <body> sits outside any landmark. Arguably inherent to teleporting and arguably a documentation note rather than a fix; flagging, not asserting.

Follow-ups are filed:

finding issue
aria-required-parent — Select #738
nested-interactive + aria-input-field-name — Rating #739
aria-input-field-name — Slider #740
label — NumberField #741
region — Portal #742 (filed as a decision, not a defect)

#738 turned out not to be the finding it looked like. Select.Content does render role="listbox" and the items are nested inside it — but SelectItem binds its attrs onto its own non-renderless Atom while Select.Item's @example spreads that same attrs onto a child <div>, so every option renders twice and the inner one's nearest role ancestor is another option. That is also the attrs double-fire hazard .claude/rules/components.md already documents, so one correction to the example fixes both.

This was a floor, not a ceiling — and the second commit raises it. See "Degenerate shapes" below.

Degenerate shapes

The fixtures above are each component's documented usage, and a documented example is written by someone thinking about the example. The failure mode that ships is the omission: the optional sub-component nobody mounted, the optional prop nobody passed. #608 is exactly that shape, and the canonical sweep caught only its Progress third — by the accident that Progress's own example happens not to render a Progress.Label.

packages/0/src/components/fixtures/degenerate/ is the other half of #732's open question: 11 fixtures, each the canonical shape with one documented-optional part removed and nothing else changed.

axe: 10 violations (11 elements) across 10 components, 4 distinct rules
rule impact elements components
button-name critical 6 Button, Checkbox, Radio, Switch, Toggle
aria-dialog-name serious 2 AlertDialog, Dialog
image-alt critical 2 Avatar, Image
label critical 1 Combobox

Two of these are ours. Eight are not, and that is the useful result.

aria-dialog-name on Dialog and AlertDialog is the finding this pass exists for:

<dialog role="dialog" aria-modal="true" aria-labelledby="v-0-title" aria-describedby="v-0-description" open>

No Dialog.Title is mounted, so v-0-title resolves to nothing. The dialog has a dangling IDREF and no accessible name. That is #608's actual failure mode and the two-thirds of it #638 does not fix — #638 is Progress-scoped. It is invisible from the documented shape, which mounts a Title, which is the whole argument for this pass.

The remaining eight are consumer omissions. A headless library cannot stop a consumer from failing to name a widget, and it should not try. The question the sweep is actually asking is narrower and is v0's: does the component degrade to a merely-unnamed widget, or to a broken one — a dangling IDREF, a role stripped of a property it requires? Everywhere except the two Dialogs, v0 degrades correctly. button-name, image-alt and label firing on a nameless control is axe working as intended, not v0 misbehaving.

Design notes:

  • A stated membership rule, not a grab bag. A component earns a degenerate fixture when its canonical fixture supplies an accessible name, or mounts a labelling sub-component, that the API leaves optional. NumberField, Progress, Rating, Select and Slider are absent because their canonical fixtures already omit theirs — which is why they are in the first-pass count above.
  • Counted separately, never summed. The canonical number stays the one a consumer following the docs would hit; it is unchanged at 10 / 13 / 7 / 8. Both sections sit inside the existing report fence, so the a11y job's awk lifts them together with no workflow change.
  • A known gap, stated rather than papered over. The sweep opens a collapsed widget by clicking [aria-expanded="false"]. Combobox's listbox does not open from that click, so its options are never in the DOM to audit — and Combobox.Item is structurally identical to Select.Item, down to the same <div v-bind="attrs"> in its own example. Read Combobox's clean line in the report as unverified, not clean. Noted in [Bug] Select.Item renders role="option" twice — nested options have no required listbox parent #738.
  • The manifest check is deliberately not a completeness check. Degenerate shapes are opt-in — most components have no optional name to remove. What is asserted is that every degenerate key has a canonical fixture beside it: without the baseline, a violation cannot be attributed to the omission rather than to the component.

Not vitest-axe

#732 proposed vitest-axe, on the reasoning that the docs already recommend it. It does not work here:

FAIL  src/components/a11y.browser.test.ts
Caused by: TypeError: (0 , __commonJSMin(...)(...).createRequire) is not a function
 ❯ node_modules/vitest-axe/dist/index.js:9:29

vitest-axe@0.1.0 calls module.createRequire at import time. 1.0.0-pre.5 drops that but pulls chalk, which fails to resolve tinyrainbow in the browser environment. Both are Node packages; vitest-axe is a jsdom/happy-dom tool, which is why reka-ui's use of it comes bundled with a getComputedStyle monkeypatch.

Running the auditor where layout is simulated is the one thing #732 argues against, so the wrapper goes and the engine stays — vitest-axe wraps axe-core, and axe-core is a browser library that needs no wrapper to run in a browser. guide/features/accessibility.md now documents both forms and says which to reach for.

Design decisions

Sweep, not per-component assertions. #732 left this open, leaning sweep. A dedicated file guarantees no component is silently unaudited and keeps axe out of the hot path of the 25 existing *.browser.test.ts files. The manifest check derives its expected set from maturity.json — the registry that already drives the docs badges — so promoting a component out of draft without writing a fixture fails the sweep. Enforcement lives outside this file on purpose.

Explicit fixture list, not import.meta.glob. Matches the convention src/surface.test.ts sets and states outright: a committed array so a surface change is a reviewable diff rather than a silent side effect.

A <main> harness instead of disabling region. Mounted into a bare <div>, 25 of 40 components reported region — content not contained by a landmark, true of the harness and not of the component. reka-ui's answer is a global configureAxe disabling the rule. Wrapping the mount point in <main> fixes the actual defect instead, and region still fires for Portal, correctly. No axe rule is disabled and no runOnly is passed — an auditor configured down to the rules you already pass is not an auditor.

Separate job, not a step on browser. Mirrors the existing vapor job, which is this repo's own pattern for non-blocking signal. Keeps the breakdown in its own step summary rather than buried under the coverage upload, leaves the browser job's Codecov behavior untouched, and makes the flip to blocking a one-line deletion. Same Playwright cache key, so Chromium restores rather than re-downloads.

Verification

All run at this branch:

pnpm lint                                  clean
pnpm typecheck                             packages/0, paper, genesis — Done
pnpm repo:check                            knip + sherif — No issues found
pnpm build                                 Build complete
pnpm test:run --project '!v0:browser'      147 files, 4769 passed | 1 skipped
pnpm exec vitest run --project v0:browser   34 files, 1756 passed | 2 skipped

Sweep itself: 53 tests passed, ~4s, zero [Vue warn] output (the repo's zero-stderr policy). The CI extraction step was verified locally by running the same awk over the run log. Re-run after the degenerate commit: pnpm lint clean, pnpm typecheck Done, pnpm exec vitest run --project v0:browser 34 files, 1768 passed | 2 skipped.

Related

Stacked conceptually on #735 (coverage globs) but textually independent — different hunks of pr-checks.yml, no conflict either merge order. The fixtures land under **/fixtures/**, already excluded by the root coverage config, so they do not enter coverage under either PR's include set.

@johnleider
test: add advisory axe-core accessibility sweep for every component (#…
e8aa4bd 
…732)

35 of 171 test files assert `aria-*` by hand. That is real coverage and it is
not the gap — every one of those assertions is an attribute someone thought to
check. Nothing audits the ones nobody thought of, which is how the open a11y
batch (#608, #611-#616) survived a suite those 35 files already passed.

Adds a sweep that mounts each of the 40 shipped components in the shape its own
`@example` JSDoc documents, in the `v0:browser` project's real Chromium, and
hands it to axe-core. Advisory: violations are collected and reported, never
asserted. Gating on day one buys either a red master or suppressions written to
deadline, and a suppression written to get CI green is a permanently invisible
violation.

First pass: 10 violations, 13 elements, 7 components, 8 rules. Three reproduce
open issues (#608 via Progress, #613 via Selection and Select). The other five
are new. Full breakdown in the PR body; not fixed here.

Uses axe-core directly rather than `vitest-axe`, which issue #732 proposed and
the accessibility guide recommends to users. `vitest-axe` reaches for
`node:module` at import time and fails to load in Vitest browser mode at both
0.1.0 and 1.0.0-pre.5 — it is a jsdom/happy-dom tool. axe-core is the engine
underneath it either way. The guide now says so.

The manifest check derives its expected set from maturity.json rather than from
this file, so promoting a component out of `draft` without writing it a fixture
fails the sweep.

CI runs it as a separate `a11y` job with `continue-on-error: true`, mirroring
the existing `vapor` job's non-blocking-signal pattern, and tees the breakdown
into the GitHub step summary. Flipping to a gate is deleting that one line.
@johnleider johnleider added this to the v1.1.0 milestone Jul 28, 2026
@johnleider johnleider added the enhancement New feature or request label Jul 28, 2026
@johnleider johnleider self-assigned this Jul 28, 2026
@github-actions

github-actions Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

⚠️ No changeset found

This PR changes packages/* source but has no changeset, so it won't be in the next release's version bump or notes. Add one:

pnpm changeset

Pick the affected package(s) — @vuetify/v0 carries @vuetify/paper automatically; @paper/genesis (and other @paper/* design systems) version separately — a bump type, and a short summary, then commit the generated .changeset/*.md. Docs-only, chore, or CI PRs can ignore this.

@johnleider johnleider modified the milestones: v1.1.0, v1.0.x Jul 28, 2026
@johnleider
test: audit degenerate component shapes in the accessibility sweep
757234d 
The canonical sweep audits each component in the shape its own `@example`
documents, and a documented example is written by someone thinking about the
example. The failure mode that ships is the omission — the optional
sub-component nobody mounted, the optional prop nobody passed.

Adds `fixtures/degenerate/`: 11 fixtures, each the canonical shape with one
documented-optional part removed and nothing else changed. Membership rule is
stated at the map so it does not drift into a grab bag; components whose
canonical fixture already omits its optional name (NumberField, Progress,
Rating, Select, Slider) are absent because they are already in the first-pass
count.

Counted and reported separately from the canonical number, which is unchanged
at 10 violations / 13 elements / 7 components / 8 rules. No CI change — both
sections sit inside the existing report fence.

The result the pass exists for: `aria-dialog-name` on Dialog and AlertDialog,
2 elements. Both emit `aria-labelledby={titleId}` whether or not the Title is
mounted, so omitting it leaves a `<dialog>` with a dangling IDREF and no
accessible name. That is #608's actual failure mode and the two-thirds of it
#638 does not fix — invisible from the documented shape, which mounts a Title.

The other 8 are consumer omissions that degrade correctly to merely-unnamed
widgets (button-name x6, image-alt x2, label x1), which is what a headless
library should do and is the useful negative result.

`sweep()` extracted so both passes share the mount/audit body; the render
assertion moves to the callers rather than being hidden behind a helper.
This was referenced Jul 28, 2026
Open
Open
Open
Open
Open
Open
@johnleider
test(a11y): close false-confidence holes in the axe sweep
a74c5b2 
Make the advisory baseline trustworthy enough to burn down against:
mount real Snackbar surface, drop attrs double-fire in Select/Combobox
fixtures, open disclosures with focus-for-input / click-for-button,
dedupe open-state HTML churn, and tag unverified opens in the report.
Also force Treeview open-all, Carousel LiveRegion, Breadcrumbs overflow,
and correct the region / gate-comment wording from review.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@johnleider johnleider

Labels

enhancement New feature or request

Projects

None yet

Milestone

v1.0.x

Development

Successfully merging this pull request may close these issues.

[Chore] Automated accessibility auditing — vitest-axe in the v0:browser project + CI gate

1 participant

@johnleider