Conversation
Same shape as the Arsenal PoC: Yarn pinned via packageManager (with its integrity hash) rather than a committed release binary. - .yarnrc.yml: nodeLinker node-modules, Scality-scoped approvedGitRepositories, and networkConcurrency: 1 (Git dependencies with Yarn 1 lockfiles are bootstrapped by Yarn Classic, which races on its shared cache when run in parallel -- see YN0058 in Docker builds). npmMinimalAgeGate is left at Yarn's 1-day default. - Build scripts stay disabled globally; the 12 packages that need compiling are allowlisted via dependenciesMeta. Yarn 1 by contrast lets every package run install scripts. - fcntl carries no checksum: it is a Git dependency with a node-gyp build, so Yarn repacks it and the archive is not reproducible. Of the 17 Git-sourced entries it is the only unstable one -- publishing scality/node-fcntl to npm would remove the need for this.
- corepack enable must precede actions/setup-node: its cache: yarn probe shells out to Yarn to find the cache folder, and Yarn 1 refuses once packageManager is set (actions/setup-node#1027). - --frozen-lockfile -> --immutable, and drop --network-concurrency (rejected on the CLI; now a .yarnrc.yml setting). - Berry removed the global --silent flag, so 'yarn run --silent X' becomes 'yarn X'; Berry is quiet by default.
- yarn install --production exits 1 under Berry; the replacement is yarn workspaces focus --production, which needs no plugin. - workspaces focus cannot enforce lockfile immutability -- verified: with a deliberately stale lockfile and enableImmutableInstalls it still exits 0 and installs the drift. A preceding 'yarn install --immutable --mode=skip-build' restores that guarantee cheaply, since it resolves without building. - yarn global add was removed in Berry; nyc now comes from npm, matching the node-gyp/typescript installs already in this stage. - .yarnrc.yml has to be COPYed in for any of the above to apply. DockerfileMem is left alone: it is FROM node:6-slim, unreferenced anywhere, and last touched in 2022 -- Berry needs Node 18+.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| @@ -99,7 +101,7 @@ jobs: | |||
| run: yarn global add typescript@4.9.5 | |||
There was a problem hiding this comment.
yarn global add was removed in Yarn 4 — this will error out. The Dockerfile correctly switched nyc to npm install -g; same treatment needed here. Same issue at line 175.
| run: yarn global add typescript@4.9.5 | |
| run: npm install -g typescript@4.9.5 |
| @@ -169,7 +175,7 @@ jobs: | |||
| run: yarn global add typescript@4.9.5 | |||
There was a problem hiding this comment.
Same yarn global add issue as line 101 — needs npm install -g.
| run: yarn global add typescript@4.9.5 | |
| run: npm install -g typescript@4.9.5 |
| "prettier": "prettier", | ||
| "prettier:diff": "bash scripts/prettier-diff.sh" | ||
| }, | ||
| "packageManager": "yarn@4.18.0+sha512.fcb8716fe7cd0eece141ffc18b92193a9df9204c1ba83189c288835223fc0bbe64af473bab0d5e9927a7daeb5caf2bb07eb2787cc9338ca040ea125f2a1f2f7e", |
There was a problem hiding this comment.
Yarn Berry doesn't auto-run pre/post lifecycle scripts for custom scripts. The postcover script (line 167: nyc report --report-dir ./coverage/test --reporter=lcov) won't execute after yarn run cover, silently breaking the coverage pipeline in CI. Either fold the report command into cover or call postcover explicitly in the CI workflow after cover.
| # `workspaces focus` cannot enforce lockfile immutability (it silently | ||
| # re-resolves), so validate the lockfile first with a cheap build-less | ||
| # install. Together these preserve what --frozen-lockfile used to give us. | ||
| RUN corepack enable \ |
There was a problem hiding this comment.
corepack enable runs only in the builder stage. The production stage (CMD [ "yarn", "start" ] at line 64) inherits from node:${NODE_VERSION} directly — yarn there falls back to the globally pre-installed Yarn 1, not Yarn 4. This works today but is fragile (Node images may drop bundled Yarn 1 in favor of corepack). Consider adding RUN corepack enable to the production stage, or switching the CMD to node index.js directly.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## development/9.5 #6293 +/- ##
================================================
Coverage 86.54% 86.54%
================================================
Files 213 213
Lines 14615 14615
================================================
Hits 12649 12649
Misses 1966 1966
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
- yarn global add was removed in Berry, and CI still used it for typescript in two jobs (tests.yaml:101 and :175). This is why every test job failed within ~20s. Now npm install -g, matching the Dockerfile. - Berry does not run pre/post lifecycle scripts for custom scripts. Verified: with pre/main/post defined, Yarn 1 runs all three and Berry runs only the main one. postcover writes the lcov report, so coverage was silently not being produced; CI now calls it explicitly. - The production image would not have started. It does COPY . (so it has package.json with packageManager: yarn@4.18.0) and runs CMD ["yarn","start"], but Corepack was only enabled in the builder stage, so the image's bundled Yarn 1 refused to run at all. Verified in a node:22.14.0-bookworm-slim container both ways: it fails without corepack enable and prints the script output with it, including against production-pruned node_modules.
.github/actions/setup-ci is used by 11 of the 14 test jobs and was never
migrated -- I had only grepped .github/workflows. It had no corepack
enable, used 'yarn global add', and passed --ignore-engines,
--frozen-lockfile and --network-concurrency, all of which Berry rejects.
This is why every functional-test job failed.
tests/functional/{report,raw-node,healthchecks} have a package.json but
are not workspaces, so Berry hard-errors on 'cd X && yarn test' ('the
nearest package directory doesn't seem to be part of the project').
That broke ft_management, ft_node, ft_node_routes, ft_gcp and
ft_healthchecks. They now invoke mocha directly, like the 20 sibling
ft_* scripts already do.
The production image enabled Corepack but did not install Yarn, so every
container start fetched the CLI over the network -- fatal for air-gapped
deployments. Yarn is now baked in, with COREPACK_HOME shared and
world-readable so images/federation (which runs as 'scality') can use
it. Verified offline as a non-root user.
Also: drop aws-sdk, es5-ext, level and protobufjs from the build
allowlist (their install scripts only print notices), disable telemetry,
and add the missing !.yarn/releases gitignore exception.
| fetch-depth: 0 | ||
| - name: Enable Corepack | ||
| run: corepack enable | ||
| - uses: actions/setup-node@v4 |
There was a problem hiding this comment.
Missing second corepack enable after setup-node. The setup-ci/action.yaml correctly runs corepack enable twice — before and after setup-node — with a comment: "setup-node installs its own Node, so re-enable Corepack against it." Without it, yarn resolves to the runner's pre-installed Yarn 1, which doesn't understand --immutable.
Same issue in the async-migration-report (line 138), unit-tests (line 160), and cleanup-gcp-buckets (line 1183) jobs.
| fetch-depth: 0 | ||
| - name: Enable Corepack | ||
| run: corepack enable | ||
| - uses: actions/setup-node@v6 |
There was a problem hiding this comment.
Same issue as in tests.yaml: missing second corepack enable after setup-node. The setup-ci/action.yaml runs it twice because setup-node installs its own Node, invalidating the earlier shims. Without the second call, yarn falls back to the runner's global Yarn 1.
| "ft_checksums_disabled": "cd tests/functional/checksumsDisabled && mocha --reporter mocha-multi-reporters --reporter-options configFile=$INIT_CWD/tests/reporter-config.json -t 120000 *.js --exit", | ||
| "ft_mixed_bucket_format_version": "cd tests/functional/metadata && mocha --reporter mocha-multi-reporters --reporter-options configFile=$INIT_CWD/tests/reporter-config.json MixedVersionFormat.js --exit", | ||
| "ft_management": "cd tests/functional/report && yarn test", | ||
| "ft_management": "cd tests/functional/report && mocha -t 40000 *.js", |
There was a problem hiding this comment.
Missing --exit flag. Every other functional test script uses --exit to prevent CI hangs from dangling handles.
| "ft_management": "cd tests/functional/report && mocha -t 40000 *.js", | |
| "ft_management": "cd tests/functional/report && mocha -t 40000 *.js --exit", |
…ists Telemetry was disabled on review advice, but that was not asked for and Berry's default is fine here. approvedGitRepositories is not optional hardening as the comment implied: Yarn's default is an empty allowlist, which blocks every Git fetch. Verified by removing it -- the install fails with YN0080 on fcntl. The real choice is only between Yarn's migration-written "**" and a scoped list. All 11 Git resolutions here use the .git clone form, which is the path that enforces the allowlist, so scoping it to Scality is meaningful rather than decorative.
1 participant
Draft / WIP — not for merge. Second PoC for ZENKO-4051 (open since 2022), after Arsenal#2706. Cloudserver was chosen because it exercises what a library could not: a Docker image build, a production install, and 7 Git dependencies.
Measured: Berry is ~2x faster for developers
Same machine, same cold/warm state, both serialised (Yarn 1 already used
--network-concurrency 1everywhere):The warm number is the day-to-day one — every branch switch or dependency change. There is further upside: Berry at default concurrency was 107 s cold, so once the dependency repos are themselves on Berry, cold approaches 3.4x.
Security posture improves
Berry disables build scripts by default. The 12 packages that genuinely need compiling are allowlisted in
dependenciesMeta; under Yarn 1 every package may run install scripts.yarn installsilently wrotenpmMinimalAgeGate: 0andapprovedGitRepositories: ["**"]into.yarnrc.ymlduring migration — both reverted. Re-read that file after migrating, before committing.What Docker forced us to learn
yarn install --productionexits 1 under Berry — not deprecated-but-working, broken. Replacement isyarn workspaces focus --production(no plugin needed).workspaces focuscannot enforce lockfile immutability. Verified: with a stale lockfile andenableImmutableInstalls: trueit still exits 0 and installs the drift. A precedingyarn install --immutable --mode=skip-buildrestores what--frozen-lockfilegave us, without building anything.yarn global addwas removed;nycnow comes from npm.networkConcurrency: 1is load-bearing. Dropping Yarn 1's flag as obsolete was wrong. Git dependencies that still carry Yarn 1 lockfiles get bootstrapped by Yarn Classic, and concurrent Classic installs corrupt each other on its shared cache (YN0058, ENOENT while unpacking). This only reproduces in Docker. Berry rejects the CLI flag but honours the setting. It can be dropped once those repos move to Berry.Checksums
Of 17 Git-sourced lockfile entries, exactly one is unstable across a cold reinstall:
fcntl. It is node-gyp-built, so Yarn repacks it and the archive is not byte-reproducible. Deps that merely runtscinprepare(arsenal,scubaclient,utapi) are perfectly stable.Its checksum is therefore omitted. This is not durable — Yarn re-adds one on every install, so any developer who commits after installing reintroduces a machine-specific hash. The real fix is to publish
scality/node-fcntlto npm asfcntl@0.3.0: the repo is public and Apache-2.0, the bare name is unclaimed, andpackage.jsonalready declaresname: "fcntl", so consumers change one line with no code change and no registry auth. Only Arsenal and Backbeat consume it.State
Verified locally: install,
--immutable, Docker image build (--target builder), eslint (0 errors), prettier. The test suite has not been run locally — CI is the check for that.DockerfileMemuntouched:FROM node:6-slim, unreferenced anywhere, last modified 2022, and Berry needs Node 18+.