Skip to content

chore(webkit): run the design-system gate in four stages - #2336

Closed
isaque-bock-azion wants to merge 2 commits into
release/new-azion-docsfrom
chore/webkit-gate
Closed

isaque-bock-azion wants to merge 2 commits into
release/new-azion-docsfrom
chore/webkit-gate

Conversation

@isaque-bock-azion

@isaque-bock-azion isaque-bock-azion commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Why

Since #2335 the adoption report has been a single non-blocking step. That measures — but nothing verifies the measurement, and nothing checks the wiring it depends on. This turns it into a gate with four stages, deliberately not equally strict.

Stage Blocks? Why
Wiringwebkit doctor not yet See the note below: the doctor returns a false FAIL on this project.
Canary yes The only stage that fails when the measurement itself stops working.
Adoption report on new only Score and tables in the run Summary, JSON as an artifact, then a ratchet — see below.
Tokens in CSS — stylelint no Clean today.

Webkit gate is the single check to mark required: it passes when every stage succeeded or was cleanly skipped, so a stage can be disabled without changing which check is required.

The canary is the point of this PR

scripts/webkit-canary.mjs writes five fixtures that violate a rule on purpose and asserts each is still flagged by that exact rule:

OK    denied-import.vue — webkit/no-deep-internal-import
OK    unknown-export.vue — webkit/valid-import-path
OK    hardcoded-color.vue — webkit/no-hardcoded-color
OK    foreign-library.vue — webkit/prefer-webkit-component
OK    hardcoded-color.astro — webkit/no-hardcoded-color

5 of 5 canaries fired — the rules reach this project.

It exists because every way of losing the rules is silent:

What breaks What you see
The catalog does not resolve one stderr line; 8 of the 12 rules quietly disabled
An extension is missing from the preset's files nothing at all for those files
A config edit drops the preset a lint that still passes

In all three cases the adoption number reads better, not worse. Two of the three actually happened while this was being built. That is why this is the stage that blocks — a red canary means the number below is meaningless, not merely worse.

Wiring reports instead of blocking, and why

webkit doctor decides whether webkit is registered with Tailwind by looking only at src/webkit.css. This project does it correctly in src/styles/main.css:3, so the check returns a false FAIL — 1 fail / 2 warn / 6 ok, where the fail is wrong and the two warns are the husky wiring we deliberately skipped.

The job carries continue-on-error: true with that explanation inline. It flips to blocking as soon as the doctor reads the project's real CSS entry. Filed upstream as part of aziontech/webkit#964's notes.

The report became self-contained

It used to consume a JSON that a previous step wrote. Now it runs ESLint itself, so the number cannot come from a stale file. It also gained:

  • an adoption score — share of clean .vue/.astro files, so a file with twenty findings weighs the same as one with a single finding, and the number moves when a file is finished;
  • a baseline ratchet (--update, --fail-on new) with multiset keys, so a second violation of an already-baselined rule in the same file counts as new;
  • --format json, uploaded as an artifact;
  • a coverage note that names what it did not look at.

Current baseline: 72 violations in 30 files, 51 of 79 UI files clean (65%) — same number the previous step reported, arrived at without the intermediate file.

Two findings worth reading

Both live in scripts/lib/webkit-lint.mjs, next to the code they explain.

The ESLint Node API and the ESLint CLI disagree. Measured here with eslint 9.39.5, same files:

Entry point .astro findings
new ESLint().lintFiles(['.']) 0 — fatal Unexpected token interface per file
node node_modules/…/eslint/bin/eslint.js 0 — same failure
node_modules/.bin/eslint 3 — correct

pnpm's shim exports NODE_PATH into its .pnpm directories before exec'ing node; without it ESLint cannot resolve astro-eslint-parser and silently falls back to the default parser. All three paths succeed — two just report a smaller number. So the scripts use the same entry point pnpm lint:eslint does.

pnpm run pollutes stdout with "Already up to date" and "Done in Xms", which would land in the Summary. The CI step uses pnpm --silent.

Verification

Every stage run locally in this branch:

  • pnpm canary:webkit → 5/5, exit 0
  • pnpm report:webkit-adoption → 72 / 30 / 65%, Markdown clean on stdout (first line is ## Webkit adoption)
  • pnpm lint:style → clean, exit 0
  • pnpm doctor:webkit → 1 fail / 2 warn / 6 ok, exit 1 (the false FAIL above — hence non-blocking)
  • pnpm build:local → green, 1628 pages, frontmatter test passes

The canary was also confirmed to fail correctly: run in a project the rules do not reach, all five report FAIL. It removes its fixture folder in a finally.

This is a stand-in, on purpose

webkit report and webkit canary are being added to @aziontech/webkit itself — aziontech/webkit#964. Those commands only reach a consumer through an npm release, and this repo needed the gate before that release.

When the published package carries them, .github/workflows/webkit-gate.yml becomes one uses: of the design system's reusable workflow and all three scripts here are deleted. Both files say so, at the top, so nobody has to guess whether they are permanent.

Note

.github/workflows/ is owned by @aziontech/team-uxe in CODEOWNERS, so this needs a UXE review alongside DevRel. .github/GOVERNANCE.md § What CI checks is updated in the same PR.


Why green with 72 violations — and what changed

Fair question, and the first version of this PR had a bad answer: the adoption stage ran the report with no --fail-on and no baseline, so it exited 0 whatever it found. Green meant only "the canary fired" — it said nothing about the number growing, and a PR adding a 73rd violation would also have been green.

Fixed by committing .webkit-baseline.json (the 72 keys as they stand) and adding a No new violations step:

  • the 72 stay frozen and green — nobody is asked to clean up the past before shipping;
  • a violation the PR introduces fails the stage, naming the file and the rule;
  • fixing a frozen one is reported, never punished — pnpm report:webkit-adoption --update prunes it, so the number can only ratchet down.

Verified in all three directions locally:

Scenario Result
Clean run against the baseline exit 0 — 0 new since the baseline
A hardcoded colour added in a new component exit 1 — 1 new, and the report lists src/components/__NewViolation.vue — no-hardcoded-color
A baseline entry whose violation is gone exit 0 + a prune hint

And in CI on this branch: 72 webkit/* violation(s) in 30 file(s); adoption 65%; 0 new since the baseline — all five jobs green.

The report is still written to the Summary in its own step, before the ratchet runs, so the number reaches the Summary whatever the gate decides.

Also dropped .eslint-report.json from .gitignore: the report runs ESLint itself now, so there is no intermediate file to ignore.

The adoption report has been a single non-blocking step since #2335. That measures,
but nothing verifies the measurement, and nothing checks the wiring it depends on.
This makes it a gate with stages that are deliberately not equally strict.

**canary — blocks.** `scripts/webkit-canary.mjs` writes five fixtures that violate a
rule on purpose and asserts each is still flagged by that exact rule. It exists
because every way of losing the rules is silent: an unresolved catalog disables 8 of
the 12 with one stderr line, an extension missing from the preset reports nothing at
all, and a config edit that drops the preset leaves a lint that still passes. In all
three cases the adoption number reads *better*. Two of the three happened while this
was being built, which is why this is the one stage that blocks.

**wiring — reports, for now.** `webkit doctor` decides whether webkit is registered
with Tailwind by looking only at `src/webkit.css`; this project does it correctly in
`src/styles/main.css`, so the check returns a false FAIL. It becomes blocking as soon
as the doctor reads the project's real CSS entry — the job carries that note.

**adoption — reports.** The script is now self-contained: it runs ESLint itself
instead of consuming a JSON that a previous step wrote, so the number cannot come
from a stale file. It gained an adoption score (share of clean `.vue`/`.astro` files,
so a file with twenty findings weighs the same as one with a single finding), a
baseline ratchet, `--format json`, and a coverage note that names what it did not
look at. `--fail-on new` is the flip to blocking, once `.webkit-baseline.json` is
committed.

**style — reports.** stylelint over the stylesheets; clean today.

`Webkit gate` aggregates them and is the single check to mark required: it passes
when every stage succeeded *or was cleanly skipped*, so a stage can be disabled
without changing which check is required.

The report step is removed from `pr-checks.yml` — it lives in the new workflow now,
so the measurement does not run twice.

Two findings from wiring this up, both in `scripts/lib/webkit-lint.mjs`:

- The scripts shell out to `node_modules/.bin/eslint`, not the Node API and not the
  resolved bin file. Measured here with eslint 9.39.5: the API returned a fatal
  "Unexpected token interface" for every `.astro` file while the CLI linted them
  fine, and `node <bin>` reproduced the failure. pnpm's shim exports NODE_PATH before
  exec'ing node; without it ESLint cannot resolve astro-eslint-parser and silently
  uses the default one. All three paths succeed — two just report a smaller number.
- `pnpm run` prints "Already up to date" and "Done in Xms" on stdout, so the CI step
  uses `pnpm --silent` to keep that out of the Summary.

These scripts are a stand-in. `webkit report` and `webkit canary` are being added to
@aziontech/webkit itself (aziontech/webkit#964); they only reach a consumer through
an npm release, and this repo needed the gate before that release. When the published
package carries them, this workflow becomes one `uses:` of the design system's
reusable workflow and all three scripts go away.
The gate went green with 72 violations, which is fair to ask about: it was green
*because* nothing was checking the count. The adoption stage ran the report with no
`--fail-on` and no baseline, so it exited 0 whatever it found — including if a PR
added a 73rd.

So green meant only "the canary fired". It said nothing about the number growing.

This commits `.webkit-baseline.json` — the 72 keys as they stand — and adds a
`No new violations` step to the adoption stage. The effect:

- the 72 stay frozen and green; nobody is asked to clean up the past before shipping
- a violation the PR *introduces* fails the stage, naming the file and the rule
- fixing a frozen one is reported, never punished — `--update` prunes it, so the
  number can only ratchet down

Verified locally, all three directions: clean run → exit 0 with "0 new since the
baseline"; a hardcoded colour added to a new component → exit 1, "1 new", and the
report lists `src/components/__NewViolation.vue — no-hardcoded-color`; a baseline
entry whose violation is gone → exit 0 plus the prune hint.

The report is still written to the Summary in its own step, before the ratchet runs,
so the number reaches the Summary whatever the gate decides.

Also drops `.eslint-report.json` from .gitignore — the report runs ESLint itself now,
so there is no intermediate file to ignore.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant