feat(webkit): add the consumer adoption pipeline (report, canary, reusable gate) - #964
Closed
isaque-bock-azion wants to merge 4 commits into
Closed
isaque-bock-azion wants to merge 4 commits into
isaque-bock-azion wants to merge 4 commits into
Conversation
The preset's FILES list did not include `.astro`, so on an Astro consumer none of the twelve rules applied to any Astro file. It failed open and silently: the files were linted, reported nothing, and read as clean. Verified with a canary plus a `.vue` control on the docs project: identical content (a deep-internal import, an unpublished export, a hardcoded hex) reported 3 violations as `.vue` and **zero** as `.astro`. With `.astro` in the list, four rules fire correctly, and it surfaced 3 real hardcoded colours that had been invisible. Two rules degrade rather than throw on Astro: `no-style-override` needs vue-eslint-parser's template visitor, which astro-eslint-parser does not provide, so it falls back to its script visitor and reports nothing there. The report command states that as a coverage gap. `presets.test.mjs` pins the extension list, that every rule is in a preset, that nothing is a warning, and that `performance` stays a strict subset of `recommended`. Impact for consumers: an Astro project that upgrades will see new findings in files that were previously unmeasured. That is the fix working, but it is a behaviour change on a previously-green lint — flag it in the release notes, and say so if the review would rather ship it as breaking.
Consumers had no way to measure their own adoption. The docs project had to write a local script to aggregate ESLint JSON, and site and console-kit would have had to write the same one again. These two commands make it the design system's job. `webkit report` runs the project's ESLint, keeps only `webkit/*` results, and prints totals, a table per rule, the worst files, and an adoption score — Markdown on stdout, progress on stderr, so a CI step can redirect it into a run summary. `--format json` for machines. With `--baseline` it becomes a ratchet: existing debt frozen in a committed file, `--fail-on new` failing only on what a PR introduces. Keys are `<file>::<rule>`, one per occurrence, so a second violation of an already-baselined rule counts as new — the same multiset semantics as `scripts/check-authoring.mjs`. The score is the share of **clean UI files**, not of violations: a file with twenty findings weighs the same as one with a single finding, so the number moves when a file is finished rather than when the cheapest findings are cleared. Same shape as the console-kit architecture report, so the two are comparable. `webkit canary` writes fixtures that violate a rule on purpose and asserts each is still flagged by that exact rule. It exists because every way of losing the rules is silent — an unresolved catalog disables eight of twelve with one stderr line, a missing extension reports nothing, a dropped preset leaves a lint that still passes — and in all three cases the adoption number reads *better*. The report spawns `node_modules/.bin/eslint`, not the Node API and not the resolved bin file. Measured on pnpm + Astro with eslint 9.39.5: the API returned a fatal "Unexpected token interface" for every `.astro` file while the CLI linted them fine, and `node <bin>` reproduced the failure. pnpm's shim exports NODE_PATH before exec'ing node; without it ESLint cannot resolve the Astro parser and silently uses the default one. All three paths succeed — two just return a smaller number. The full measurement is in docs/toolkit/report.md. Verified against the real docs project: 72 violations in 30 files, 65% — identical to what that repo's own `eslint` and its CI report. The ratchet was exercised end to end: 72 recorded, 0 new on a clean run, exit 1 with the file and rule named when a violation is introduced.
The design system published rules, a stylelint config, a catalog and a CLI, and had no `workflow_call` — so a consumer could not call the gate. Each one wired its own, or did not wire one at all. Four stages, deliberately not equally strict: - **wiring** (`webkit doctor`) blocks. If the wiring is broken the stages below measure the wrong thing. It also covers version drift — it warns on any dependency still pinned to `latest`. - **canary** blocks. It is the only stage that fails when the measurement itself stops working. - **adoption** (`webkit report`) reports to the run Summary and uploads the JSON. `fail-on: new` turns it into a ratchet. - **style** (stylelint) reports. `webkit-gate` is the single check to mark required: it passes when every stage succeeded *or was cleanly skipped*, so a consumer can disable a stage without changing which check is required. Same `declare -A` aggregator shape as governance.yml, and the same SHA-pinned actions. The report is published in a step of its own, before the ratchet enforces anything, so the number always reaches the Summary whatever the gate decides. It lives in a public repo, so the private consumers (site, console-kit) can call it too — unlike the org's private compliance action, which cannot run on public repos.
`lint-staged` matches `packages/webkit/**/*.{js,ts,vue}`, which does not include
`.mjs` — so the test file skipped the pre-commit formatter and only the CI
`format:check` (whose glob does include `.mjs`) caught it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The design system publishes twelve ESLint rules, a stylelint config, a versioned catalog, an MCP and an adoption CLI — and had no
workflow_call. A consumer could not call the gate. Sodocswrote its own script to aggregate ESLint JSON, andsiteandconsole-kitwould have written the same one again.This is the esteira: two commands the design system owns, and one reusable workflow that wires them into any repo with a single
uses:.The four stages, deliberately not equally strict
webkit doctorlatest.webkit canarywebkit reportfail-on: newturns it into a ratchet.webkit-gateis the single check to mark required: it passes when every stage succeeded or was cleanly skipped, so a consumer can disable a stage without changing which check is required. Samedeclare -Aaggregator and the same SHA-pinned actions asgovernance.yml.Three findings that shaped the code
These were measured on the real
docsproject (pnpm + Astro, eslint 9.39.5), not reasoned about.1. The preset never applied to
.astroFILESdid not list.astro, so on an Astro consumer none of the twelve rules applied to any Astro file. It failed open and silently — the files were linted, reported nothing, and read as clean.Canary plus a
.vuecontrol: identical content reported 3 violations as.vueand zero as.astro. With the extension in the list, four rules fire and it surfaced 3 real hardcoded colours that had been invisible.2. The ESLint Node API and the ESLint CLI disagree
.astrofindingsnew ESLint().lintFiles(['.'])Unexpected token interfaceper filenode node_modules/…/eslint/bin/eslint.js .node_modules/.bin/eslint .The API returned a fatal parse error for every
.astrofile even thoughcalculateConfigForFile()reported the right parser and processor for those same files. Spawning the resolved bin file withnodefailed identically. The shim is what differs: pnpm's.bin/eslintexportsNODE_PATHinto its.pnpmdirectories before exec'ing node, and without it ESLint cannot resolve the Astro parser and silently falls back to the default one.All three paths succeed. Two just return a smaller number. So
webkit reportgoes through the same entry point the project's ownlintscript uses — the number is correct by construction, not by luck.3. Every way of losing the rules is silent — hence the canary
filesIn all three cases the adoption number reads better, not worse. I hit two of them while building this. The canary writes fixtures that violate a rule on purpose and asserts each is still flagged by that exact rule — it is the one check that goes red when the measurement stops working.
Design decisions worth reviewing
<file>::<rule>, one key per occurrence), so a second violation of an already-baselined rule in the same file counts as introduced. A plain set would let it evade. Copied deliberately fromscripts/check-authoring.mjs.no-style-overridecannot run on.astro, raw markup is caught by no rule yet, and an unresolved catalog is announced before the numbers. A report that hides its own blind spots is worse than no report.--fail-on newrefuses to run without a baseline rather than inventing an empty one and reporting everything as new.Verification
test:toolkit— 134 pass, 0 fail (19 new: 15 for the report, 4 pinning the presets)authoringratchet — 229 known, 0 newcatalog:check— no drift ·eslint src --max-warnings 0— clean · prettier — cleandocs:webkit report→ 72 violations in 30 files, 65% — identical to that repo's owneslintand to its CI rundocs; correctly reports all-fail in a project the rules do not reach; removes its fixtures in afinallyNot in this PR
docscurrently runs a local script that this replaces; swapping it for the reusable workflow is a follow-up in that repo, thensiteandconsole-kit.<button>where a component exists). Named as a gap in every report until it exists.webkit initdoes not yet add.webkit-canary/to.gitignore. The canary removes the folder in afinally, so only a hard crash could leave it behind.Note
Pushed over SSH: the HTTPS credential in this environment is an OAuth App without the
workflowscope, so it refuses to create.github/workflows/*. The repo'soriginwas left on HTTPS, unchanged.CI
Governance Gate green — all 18 jobs, including the four Vitest browser shards, the four visual-regression shards, and
Consumer smoke test (init + build).One fixup was needed:
lint-stagedmatchespackages/webkit/**/*.{js,ts,vue}, which does not include.mjs, so the new test file skipped the pre-commit formatter and only the CIformat:check(whose glob does include.mjs) caught it. Worth widening that glob separately — everytest/**/*.test.mjsin this package is in the same blind spot.