Skip to content

feat(webkit): add the consumer adoption pipeline (report, canary, reusable gate) - #964

Closed
isaque-bock-azion wants to merge 4 commits into
mainfrom
feat/webkit-consumer-pipeline
Closed

isaque-bock-azion wants to merge 4 commits into
mainfrom
feat/webkit-consumer-pipeline

Conversation

@isaque-bock-azion

@isaque-bock-azion isaque-bock-azion commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Why

The design system publishes twelve ESLint rules, a stylelint config, a versioned catalog, an MCP and an adoption CLI — and had no workflow_call. A consumer could not call the gate. So docs wrote its own script to aggregate ESLint JSON, and site and console-kit would have written the same one again.

This is the esteira: two commands the design system owns, and one reusable workflow that wires them into any repo with a single uses:.

jobs:
  webkit:
    uses: aziontech/webkit/.github/workflows/webkit-consumer-gate.yml@main

The four stages, deliberately not equally strict

Stage Blocks? Why
wiringwebkit doctor yes If the wiring is broken the stages below measure the wrong thing. Covers version drift too: it warns on any dependency still pinned to latest.
canarywebkit canary yes The only stage that fails when the measurement itself stops working.
adoptionwebkit report no Score and tables to the run Summary, JSON as an artifact. fail-on: new turns it into a ratchet.
style — stylelint no Tokens in authored CSS.

webkit-gate is the single check to mark required: it passes when every stage succeeded or was cleanly skipped, so a consumer can disable a stage without changing which check is required. Same declare -A aggregator and the same SHA-pinned actions as governance.yml.

Three findings that shaped the code

These were measured on the real docs project (pnpm + Astro, eslint 9.39.5), not reasoned about.

1. The preset never applied to .astro

FILES did not list .astro, so on an Astro consumer none of the twelve rules applied to any Astro file. It failed open and silently — the files were linted, reported nothing, and read as clean.

Canary plus a .vue control: identical content reported 3 violations as .vue and zero as .astro. With the extension in the list, four rules fire and it surfaced 3 real hardcoded colours that had been invisible.

⚠️ Consumer impact: an Astro project that upgrades will see new findings in files that were previously unmeasured. That is the fix working, but it is a behaviour change on a previously-green lint. Say so if the review would rather ship it as breaking.

2. The ESLint Node API and the ESLint CLI disagree

Entry point .astro findings
new ESLint().lintFiles(['.']) 0 — fatal Unexpected token interface per file
node node_modules/…/eslint/bin/eslint.js . 0 — same failure
node_modules/.bin/eslint . 3 — correct

The API returned a fatal parse error for every .astro file even though calculateConfigForFile() reported the right parser and processor for those same files. Spawning the resolved bin file with node failed identically. The shim is what differs: pnpm's .bin/eslint exports NODE_PATH into its .pnpm directories before exec'ing node, and without it ESLint cannot resolve the Astro parser and silently falls back to the default one.

All three paths succeed. Two just return a smaller number. So webkit report goes through the same entry point the project's own lint script uses — the number is correct by construction, not by luck.

3. Every way of losing the rules is silent — hence the canary

What breaks What you see
The catalog does not resolve one stderr line; 8 of 12 rules quietly disabled
An extension missing from files nothing at all for those files
A config edit drops the preset a lint that still passes

In all three cases the adoption number reads better, not worse. I hit two of them while building this. The canary writes fixtures that violate a rule on purpose and asserts each is still flagged by that exact rule — it is the one check that goes red when the measurement stops working.

Design decisions worth reviewing

  • The score counts clean UI files, not violations. A file with twenty findings weighs the same as one with a single finding, so the number moves when a file is finished rather than when the cheapest findings across the repo are cleared. Same shape as the console-kit architecture report, so the two are comparable.
  • The baseline is a multiset (<file>::<rule>, one key per occurrence), so a second violation of an already-baselined rule in the same file counts as introduced. A plain set would let it evade. Copied deliberately from scripts/check-authoring.mjs.
  • The report always states what it did not look atno-style-override cannot run on .astro, raw markup is caught by no rule yet, and an unresolved catalog is announced before the numbers. A report that hides its own blind spots is worse than no report.
  • --fail-on new refuses to run without a baseline rather than inventing an empty one and reporting everything as new.

Verification

  • test:toolkit134 pass, 0 fail (19 new: 15 for the report, 4 pinning the presets)
  • authoring ratchet — 229 known, 0 new
  • catalog:check — no drift · eslint src --max-warnings 0 — clean · prettier — clean
  • End to end against docs: webkit report72 violations in 30 files, 65% — identical to that repo's own eslint and to its CI run
  • Ratchet exercised: 72 recorded → 0 new on a clean run (exit 0) → one violation introduced → exit 1 with the file and rule named, score 65% → 64%
  • Canary: 5/5 fire in docs; correctly reports all-fail in a project the rules do not reach; removes its fixtures in a finally

Not in this PR

  • Rolling it out. docs currently runs a local script that this replaces; swapping it for the reusable workflow is a follow-up in that repo, then site and console-kit.
  • A rule for raw markup (<button> where a component exists). Named as a gap in every report until it exists.
  • webkit init does not yet add .webkit-canary/ to .gitignore. The canary removes the folder in a finally, so only a hard crash could leave it behind.

Note

Pushed over SSH: the HTTPS credential in this environment is an OAuth App without the workflow scope, so it refuses to create .github/workflows/*. The repo's origin was left on HTTPS, unchanged.


CI

Governance Gate green — all 18 jobs, including the four Vitest browser shards, the four visual-regression shards, and Consumer smoke test (init + build).

One fixup was needed: lint-staged matches packages/webkit/**/*.{js,ts,vue}, which does not include .mjs, so the new test file skipped the pre-commit formatter and only the CI format:check (whose glob does include .mjs) caught it. Worth widening that glob separately — every test/**/*.test.mjs in this package is in the same blind spot.

The preset's FILES list did not include `.astro`, so on an Astro consumer none of
the twelve rules applied to any Astro file. It failed open and silently: the files
were linted, reported nothing, and read as clean.

Verified with a canary plus a `.vue` control on the docs project: identical content
(a deep-internal import, an unpublished export, a hardcoded hex) reported 3
violations as `.vue` and **zero** as `.astro`. With `.astro` in the list, four rules
fire correctly, and it surfaced 3 real hardcoded colours that had been invisible.

Two rules degrade rather than throw on Astro: `no-style-override` needs
vue-eslint-parser's template visitor, which astro-eslint-parser does not provide, so
it falls back to its script visitor and reports nothing there. The report command
states that as a coverage gap.

`presets.test.mjs` pins the extension list, that every rule is in a preset, that
nothing is a warning, and that `performance` stays a strict subset of `recommended`.

Impact for consumers: an Astro project that upgrades will see new findings in files
that were previously unmeasured. That is the fix working, but it is a behaviour
change on a previously-green lint — flag it in the release notes, and say so if the
review would rather ship it as breaking.
Consumers had no way to measure their own adoption. The docs project had to write a
local script to aggregate ESLint JSON, and site and console-kit would have had to
write the same one again. These two commands make it the design system's job.

`webkit report` runs the project's ESLint, keeps only `webkit/*` results, and prints
totals, a table per rule, the worst files, and an adoption score — Markdown on
stdout, progress on stderr, so a CI step can redirect it into a run summary.
`--format json` for machines. With `--baseline` it becomes a ratchet: existing debt
frozen in a committed file, `--fail-on new` failing only on what a PR introduces.
Keys are `<file>::<rule>`, one per occurrence, so a second violation of an
already-baselined rule counts as new — the same multiset semantics as
`scripts/check-authoring.mjs`.

The score is the share of **clean UI files**, not of violations: a file with twenty
findings weighs the same as one with a single finding, so the number moves when a
file is finished rather than when the cheapest findings are cleared. Same shape as
the console-kit architecture report, so the two are comparable.

`webkit canary` writes fixtures that violate a rule on purpose and asserts each is
still flagged by that exact rule. It exists because every way of losing the rules is
silent — an unresolved catalog disables eight of twelve with one stderr line, a
missing extension reports nothing, a dropped preset leaves a lint that still passes —
and in all three cases the adoption number reads *better*.

The report spawns `node_modules/.bin/eslint`, not the Node API and not the resolved
bin file. Measured on pnpm + Astro with eslint 9.39.5: the API returned a fatal
"Unexpected token interface" for every `.astro` file while the CLI linted them fine,
and `node <bin>` reproduced the failure. pnpm's shim exports NODE_PATH before
exec'ing node; without it ESLint cannot resolve the Astro parser and silently uses
the default one. All three paths succeed — two just return a smaller number. The
full measurement is in docs/toolkit/report.md.

Verified against the real docs project: 72 violations in 30 files, 65% — identical to
what that repo's own `eslint` and its CI report. The ratchet was exercised end to
end: 72 recorded, 0 new on a clean run, exit 1 with the file and rule named when a
violation is introduced.
The design system published rules, a stylelint config, a catalog and a CLI, and had
no `workflow_call` — so a consumer could not call the gate. Each one wired its own,
or did not wire one at all.

Four stages, deliberately not equally strict:

- **wiring** (`webkit doctor`) blocks. If the wiring is broken the stages below
  measure the wrong thing. It also covers version drift — it warns on any dependency
  still pinned to `latest`.
- **canary** blocks. It is the only stage that fails when the measurement itself
  stops working.
- **adoption** (`webkit report`) reports to the run Summary and uploads the JSON.
  `fail-on: new` turns it into a ratchet.
- **style** (stylelint) reports.

`webkit-gate` is the single check to mark required: it passes when every stage
succeeded *or was cleanly skipped*, so a consumer can disable a stage without
changing which check is required. Same `declare -A` aggregator shape as
governance.yml, and the same SHA-pinned actions.

The report is published in a step of its own, before the ratchet enforces anything,
so the number always reaches the Summary whatever the gate decides.

It lives in a public repo, so the private consumers (site, console-kit) can call it
too — unlike the org's private compliance action, which cannot run on public repos.
`lint-staged` matches `packages/webkit/**/*.{js,ts,vue}`, which does not include
`.mjs` — so the test file skipped the pre-commit formatter and only the CI
`format:check` (whose glob does include `.mjs`) caught it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant