chore(webkit): run the design-system gate in four stages - #2336
Closed
isaque-bock-azion wants to merge 2 commits into
Closed
isaque-bock-azion wants to merge 2 commits into
isaque-bock-azion wants to merge 2 commits into
Conversation
The adoption report has been a single non-blocking step since #2335. That measures, but nothing verifies the measurement, and nothing checks the wiring it depends on. This makes it a gate with stages that are deliberately not equally strict. **canary — blocks.** `scripts/webkit-canary.mjs` writes five fixtures that violate a rule on purpose and asserts each is still flagged by that exact rule. It exists because every way of losing the rules is silent: an unresolved catalog disables 8 of the 12 with one stderr line, an extension missing from the preset reports nothing at all, and a config edit that drops the preset leaves a lint that still passes. In all three cases the adoption number reads *better*. Two of the three happened while this was being built, which is why this is the one stage that blocks. **wiring — reports, for now.** `webkit doctor` decides whether webkit is registered with Tailwind by looking only at `src/webkit.css`; this project does it correctly in `src/styles/main.css`, so the check returns a false FAIL. It becomes blocking as soon as the doctor reads the project's real CSS entry — the job carries that note. **adoption — reports.** The script is now self-contained: it runs ESLint itself instead of consuming a JSON that a previous step wrote, so the number cannot come from a stale file. It gained an adoption score (share of clean `.vue`/`.astro` files, so a file with twenty findings weighs the same as one with a single finding), a baseline ratchet, `--format json`, and a coverage note that names what it did not look at. `--fail-on new` is the flip to blocking, once `.webkit-baseline.json` is committed. **style — reports.** stylelint over the stylesheets; clean today. `Webkit gate` aggregates them and is the single check to mark required: it passes when every stage succeeded *or was cleanly skipped*, so a stage can be disabled without changing which check is required. The report step is removed from `pr-checks.yml` — it lives in the new workflow now, so the measurement does not run twice. Two findings from wiring this up, both in `scripts/lib/webkit-lint.mjs`: - The scripts shell out to `node_modules/.bin/eslint`, not the Node API and not the resolved bin file. Measured here with eslint 9.39.5: the API returned a fatal "Unexpected token interface" for every `.astro` file while the CLI linted them fine, and `node <bin>` reproduced the failure. pnpm's shim exports NODE_PATH before exec'ing node; without it ESLint cannot resolve astro-eslint-parser and silently uses the default one. All three paths succeed — two just report a smaller number. - `pnpm run` prints "Already up to date" and "Done in Xms" on stdout, so the CI step uses `pnpm --silent` to keep that out of the Summary. These scripts are a stand-in. `webkit report` and `webkit canary` are being added to @aziontech/webkit itself (aziontech/webkit#964); they only reach a consumer through an npm release, and this repo needed the gate before that release. When the published package carries them, this workflow becomes one `uses:` of the design system's reusable workflow and all three scripts go away.
isaque-bock-azion
requested review from
a team,
bruno-andrade-azion,
marcus-souza-azion and
pedro-ribeiro-azion
as code owners
September 10, 2026 14:04
The gate went green with 72 violations, which is fair to ask about: it was green *because* nothing was checking the count. The adoption stage ran the report with no `--fail-on` and no baseline, so it exited 0 whatever it found — including if a PR added a 73rd. So green meant only "the canary fired". It said nothing about the number growing. This commits `.webkit-baseline.json` — the 72 keys as they stand — and adds a `No new violations` step to the adoption stage. The effect: - the 72 stay frozen and green; nobody is asked to clean up the past before shipping - a violation the PR *introduces* fails the stage, naming the file and the rule - fixing a frozen one is reported, never punished — `--update` prunes it, so the number can only ratchet down Verified locally, all three directions: clean run → exit 0 with "0 new since the baseline"; a hardcoded colour added to a new component → exit 1, "1 new", and the report lists `src/components/__NewViolation.vue — no-hardcoded-color`; a baseline entry whose violation is gone → exit 0 plus the prune hint. The report is still written to the Summary in its own step, before the ratchet runs, so the number reaches the Summary whatever the gate decides. Also drops `.eslint-report.json` from .gitignore — the report runs ESLint itself now, so there is no intermediate file to ignore.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Since #2335 the adoption report has been a single non-blocking step. That measures — but nothing verifies the measurement, and nothing checks the wiring it depends on. This turns it into a gate with four stages, deliberately not equally strict.
webkit doctorWebkit gateis the single check to mark required: it passes when every stage succeeded or was cleanly skipped, so a stage can be disabled without changing which check is required.The canary is the point of this PR
scripts/webkit-canary.mjswrites five fixtures that violate a rule on purpose and asserts each is still flagged by that exact rule:It exists because every way of losing the rules is silent:
filesIn all three cases the adoption number reads better, not worse. Two of the three actually happened while this was being built. That is why this is the stage that blocks — a red canary means the number below is meaningless, not merely worse.
Wiring reports instead of blocking, and why
webkit doctordecides whether webkit is registered with Tailwind by looking only atsrc/webkit.css. This project does it correctly insrc/styles/main.css:3, so the check returns a false FAIL — 1 fail / 2 warn / 6 ok, where the fail is wrong and the two warns are the husky wiring we deliberately skipped.The job carries
continue-on-error: truewith that explanation inline. It flips to blocking as soon as the doctor reads the project's real CSS entry. Filed upstream as part of aziontech/webkit#964's notes.The report became self-contained
It used to consume a JSON that a previous step wrote. Now it runs ESLint itself, so the number cannot come from a stale file. It also gained:
.vue/.astrofiles, so a file with twenty findings weighs the same as one with a single finding, and the number moves when a file is finished;--update,--fail-on new) with multiset keys, so a second violation of an already-baselined rule in the same file counts as new;--format json, uploaded as an artifact;Current baseline: 72 violations in 30 files, 51 of 79 UI files clean (65%) — same number the previous step reported, arrived at without the intermediate file.
Two findings worth reading
Both live in
scripts/lib/webkit-lint.mjs, next to the code they explain.The ESLint Node API and the ESLint CLI disagree. Measured here with eslint 9.39.5, same files:
.astrofindingsnew ESLint().lintFiles(['.'])Unexpected token interfaceper filenode node_modules/…/eslint/bin/eslint.jsnode_modules/.bin/eslintpnpm's shim exports
NODE_PATHinto its.pnpmdirectories before exec'ing node; without it ESLint cannot resolveastro-eslint-parserand silently falls back to the default parser. All three paths succeed — two just report a smaller number. So the scripts use the same entry pointpnpm lint:eslintdoes.pnpm runpollutes stdout with "Already up to date" and "Done in Xms", which would land in the Summary. The CI step usespnpm --silent.Verification
Every stage run locally in this branch:
pnpm canary:webkit→ 5/5, exit 0pnpm report:webkit-adoption→ 72 / 30 / 65%, Markdown clean on stdout (first line is## Webkit adoption)pnpm lint:style→ clean, exit 0pnpm doctor:webkit→ 1 fail / 2 warn / 6 ok, exit 1 (the false FAIL above — hence non-blocking)pnpm build:local→ green, 1628 pages, frontmatter test passesThe canary was also confirmed to fail correctly: run in a project the rules do not reach, all five report FAIL. It removes its fixture folder in a
finally.This is a stand-in, on purpose
webkit reportandwebkit canaryare being added to@aziontech/webkititself — aziontech/webkit#964. Those commands only reach a consumer through an npm release, and this repo needed the gate before that release.When the published package carries them,
.github/workflows/webkit-gate.ymlbecomes oneuses:of the design system's reusable workflow and all three scripts here are deleted. Both files say so, at the top, so nobody has to guess whether they are permanent.Note
.github/workflows/is owned by@aziontech/team-uxein CODEOWNERS, so this needs a UXE review alongside DevRel..github/GOVERNANCE.md§ What CI checks is updated in the same PR.Why green with 72 violations — and what changed
Fair question, and the first version of this PR had a bad answer: the adoption stage ran the report with no
--fail-onand no baseline, so it exited 0 whatever it found. Green meant only "the canary fired" — it said nothing about the number growing, and a PR adding a 73rd violation would also have been green.Fixed by committing
.webkit-baseline.json(the 72 keys as they stand) and adding aNo new violationsstep:pnpm report:webkit-adoption --updateprunes it, so the number can only ratchet down.Verified in all three directions locally:
0 new since the baseline1 new, and the report listssrc/components/__NewViolation.vue — no-hardcoded-colorAnd in CI on this branch:
72 webkit/* violation(s) in 30 file(s); adoption 65%; 0 new since the baseline— all five jobs green.The report is still written to the Summary in its own step, before the ratchet runs, so the number reaches the Summary whatever the gate decides.
Also dropped
.eslint-report.jsonfrom.gitignore: the report runs ESLint itself now, so there is no intermediate file to ignore.