Repository navigation
Conversation
The baseline is loaded with
grep -v '^\s*#' "$BASELINE" | grep -v '^\s*$' | sort
grep exits 1 when it selects no lines, so for a baseline that contains
only the header comments, pipefail makes the pipeline fail and set -e
aborts the script with exit status 1 and no message. Such a baseline is
what --update-baseline (or the initial-baseline path) writes after a run
with no failures, so once every known failure is fixed, every following
check fails without saying why.
Strip comments and blank lines with sed instead, which exits 0 whether
or not it deletes anything. `{ grep ... || true; }` would also work but
would hide real grep errors such as an unreadable file.
If the GNU test run produced no results, the script found no failures
and reported every baseline entry as fixed, so the check passed. This
happens if run.sh is missing or exits early (e.g. on an unknown test
component), or if every test of a component is skipped. In the same
situation, --update-baseline recorded an empty baseline.
Use the diff directory to tell whether tests ran. run-tests.sh deletes
.test-helpers/tests/diff before the run, and the GNU harness
(ts_gen_diff in tests/functions.sh) creates diff/<component> whenever it
compares a test's output, deleting only the diff files of tests that
pass. So:
* If the diff directory has no component directories, no test produced
results: exit 2 with an error, also in --update-baseline mode.
* If a component with baseline entries has no diff directory, none of
its tests produced results: exit 2 rather than report its known
failures as fixed.
Alternatives considered:
* run.sh's exit status: it exits 1 both when tests fail and when it
cannot run them (unknown component, xargs error), so it cannot tell
the two apart.
* Parsing run.sh's summary ("N tests of M FAILED", "All M tests
PASSED"): run-tests.sh would have to capture the output, and the check
would depend on its wording.
* tests/output/<component>/: the harness creates an output file for
every test that starts, including skipped ones, and run-tests.sh does
not delete tests/output, so results from an earlier run would satisfy
the check.
* Requiring results for every utility in .test-helpers/utils.list: some
utilities have every test skipped in some environments (e.g. dmesg
and rev when test helpers are not built), so this would fail runs
that are fine.
The GNU harness (ts_gen_diff in tests/functions.sh) fails a test if either its stdout or its stderr differs from the expected output. It writes the stdout diff to tests/diff/<util>/<test> and the stderr diff to tests/diff/<util>/<test>.err, and removes each when there is no difference. The script collected failures by listing diff files and skipping *.err, so a test whose stdout matches but whose stderr differs was never counted: a change that only breaks an error message passed the check even though run.sh reports the test as FAILED. For example, invoking the multicall binary through a link by full path makes rename print the full path instead of `rename:` in its error messages. run.sh then reports rename/exit_codes and rename/symlink as FAILED, but both only have .err diffs, so the script did not see them. Map both kinds of diff file to the test name and de-duplicate. The baseline format is unchanged (`util/testname`, one per line), but it can now contain tests that fail only on stderr. A baseline written by the old script lacks those, so they are reported as new failures until the baseline is updated with --update-baseline.
The GNU test workflow never detected a regression. It tried to fetch the baseline with actions/download-artifact@v8 using only `name` and `path`. The action README documents the `github-token` input as "Optional. If unspecified, the action will download artifacts from the current repo and the current workflow run.", and `run-id` as applying only "If github-token is specified". Without a token it can only see artifacts uploaded earlier in the same run, but the baseline was uploaded by earlier runs, so the step always failed with "Artifact not found for name: gnu-test-failures" (hidden by `continue-on-error`). The check script then found no baseline, wrote one from the current results and exited 0. This happened on pull requests and on main alike: main also never compared, it only re-uploaded the new file, which no later run could download. Both are visible in the logs of main run 39571991580 and pull request run 39609494316. Fixing the download would need a token with actions:read and a lookup of the latest successful main run ID, since artifacts are addressable only by run; uutils/coreutils keeps its reference results on a separate branch instead. A file in the repo is simpler, versioned with the code it describes, and lets a pull request that fixes or knowingly breaks tests update the baseline in the same change: * Add tests/gnu-test-failures.txt. A non-hidden path next to the other test data is easier to find and review than .reference/, which was named for the downloaded artifact; stop ignoring .reference. * Remove the download, main-only update and upload steps. * Make a missing baseline an error in check-new-gnu-failures.sh instead of silently creating one and passing. This is not limited to CI: with the file checked in, a missing baseline means a broken checkout, and recreating it from local results would hide regressions and record results from a different util-linux version than CI uses. --update-baseline still creates it. * Note in the baseline header that it must be generated against the util-linux version the workflow checks out. The initial file has the entries of the gnu-test-failures artifact from main run 39571991580 (commit c05b0e9, util-linux v2.41.4, ubuntu-24.04), the latest successful main run, written with --update-baseline from a diff directory containing those entries. That artifact was produced before stderr-only failures were counted, so it was checked against the run log: the 45 tests run.sh reported as FAILED are exactly the 45 entries, so none were missing.
Each push to a pull request starts a new GNU test run while the previous one keeps building util-linux and our binary, although only the latest result matters. Add the concurrency settings used by uutils/coreutils GnuTests.yml: group runs by workflow and ref, cancel an in-progress run when its pull request or branch is updated, but not on main. Per the GitHub Actions docs, a newer queued run on main still replaces an older pending one in the same group. Also set the default GITHUB_TOKEN permissions to `contents: read`, as coreutils does. Since the previous commit removed the artifact steps, the job only checks out code (this repository and util-linux), restores the Rust cache (Swatinem/rust-cache, which coreutils also uses under `contents: read`), installs packages and runs scripts, so it needs no other permissions.
sylvestre
reviewed
Oct 24, 2026
| contents: read | ||
|
|
||
| # Cancel superseded runs of a pull request or branch when it is updated. | ||
| # On main, let a run in progress finish (GitHub still replaces a pending |
Contributor
There was a problem hiding this comment.
3 lines comment for the concurrency, please make it shorter
sylvestre
reviewed
Oct 24, 2026
| done | ||
|
|
||
| # Collect current failures (exclude .err companion files) | ||
| # run-tests.sh deletes DIFF_DIR before running the GNU tests, and the GNU |
Contributor
There was a problem hiding this comment.
nobody will read an 8 lines comment :)
please make it shorter, 2 lines max
sylvestre
reviewed
Oct 24, 2026
| exit 2 | ||
| fi | ||
|
|
||
| # Collect current failures. The GNU harness (ts_gen_diff in |
Contributor
There was a problem hiding this comment.
same here, could be one line, no?
sylvestre
reviewed
Oct 24, 2026
| exit 0 | ||
| fi | ||
|
|
||
| # The baseline is checked in, so a missing file is an error rather than a |
Contributor
There was a problem hiding this comment.
do we need this comment? the error message already says it
Contributor
|
great patch :) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The GNU test workflow never actually compared results against a baseline.
run-gnu-test.ymldownloads thegnu-test-failuresartifact withactions/download-artifactbut withoutgithub-token/run-id,and per the action's README it then only looks in the current workflow run. So the download always failed (
continue-on-error),check-new-gnu-failures.shcreated a fresh baseline from the current resultsand exited 0, and on main that fresh baseline was uploaded again. No run, on main or in a PR, ever reported a new failure.
check-new-gnu-failures.sh: accept a baseline with no entries: a comments-only baseline made the script exit 1 silently (grepselects nothing underset -e -o pipefail).check-new-gnu-failures.sh: fail if the GNU tests produced no results: ifrun.shdidn't run (missing diff directory, or a component with known failures has no results), exit 2 instead of reportingevery known failure as fixed.
check-new-gnu-failures.sh: count stderr-only failures: util-linux's harness fails a test on a stdout or a stderr diff, but the script ignored*.errdiffs, so stderr-only failures were never counted.ci: check the GNU test baseline into the repo:tests/gnu-test-failures.txt(45 entries, taken from the latest main run's artifact and cross-checked against that run's log). Remove the artifactdownload/upload steps; a missing baseline is now an error. Update the baseline in a PR with
./scripts/check-new-gnu-failures.sh --update-baseline, so changes are reviewable.run-gnu-test.yml: add concurrency and read-only permissions: sameconcurrencyandpermissions: contents: readsettings as coreutils' GnuTests.yml.