Skip to content

NuGet: read a package's zip directory one entry at a time, and take up to a million entries - #621

Merged
schmitch merged 1 commit into
mainfrom
schmitch/epic-goodall-815gln
Oct 28, 2026
Merged

schmitch merged 1 commit into
mainfrom
schmitch/epic-goodall-815gln

Conversation

@schmitch

Copy link
Copy Markdown
Member

Fixes #619

A Nexus import failed some NuGet items with invalid package: invalid package: limit: 29198 entries exceeds max 10000: nuget.max_entries (default 10 000) refuses a larger .nupkg whether it is published, proxied or imported. Raising the limit was no real way out. To read the one entry Forge needs, the .nuspec, it opened the archive with the zip crate, which indexes every entry: about 400 bytes each, measured, on top of the central directory, which find_manifest also read into memory whole. A million entries took about half a GiB per package being parsed.

What changes

  • find_manifest already checked every central header itself. It now reads the directory from the spooled package through a 64 KiB buffer and keeps the manifest's header alone: its method, CRC-32, sizes and local header offset, with ZIP64 values read in the order the existing checks already make .NET agree with.
  • read_manifest (new) inflates that entry straight from the file. As before, only stored and deflated manifests are read. Its local header is read as .NET reads it (signature, name and extra lengths), its output is capped at the declared size, and the size and CRC-32 must match. The size and compression-ratio caps are unchanged.
  • The zip crate is a dev-dependency only (test packages). flate2 and crc32fast were in the tree already through it.
  • The read budget (BudgetReader, VettedDirectory) that bounded the zip crate's own directory parsing is gone, along with the second open of the archive. Every refusal stays: ZIP64 markers NuGet reads otherwise, Unicode Path fields, two ZIP64 fields, values .NET reads at other places, and repeated or escaped manifests. Their comments now name the zip crate as the other reader they guard against.
  • Defaults: max_entries goes from 10 000 to 1 000 000, and max_central_directory_bytes from 16 MiB to 256 MiB. Both bound time now, not memory. forge.toml.example, the config docs and NupkgLimits say so.
  • Changelog: docs/changes/2.2.0/619.fixed.md, and 619.upgrade.md, because a forge.toml copied from the example pins the old values.

Measurements

These are from a throwaway probe that was not committed: release build, counting allocator, synthetic packages with stored entries.

entries package parse peak heap
30 000 5.9 MB 3 ms 192 KiB
999 999 198 MB 119 ms 192 KiB
3 000 000 594 MB 332 ms 192 KiB

The same probe parsed five nuget.org packages (Newtonsoft.Json 13.0.3, Microsoft.Extensions.Logging.Abstractions 3.1.0 with its repeated packageIcon.png, Microsoft.TypeScript.MSBuild 5.6.2, Microsoft.NETCore.App.Ref 8.0.0 and FSharp.Core 8.0.400). Each manifest came out byte-identical to what the zip crate extracts.

Tests

  • cargo test -p forge-format-nuget: 95 pass. New tests cover a 30 000-entry package (and its refusal at a 10 000 cap), plus a manifest read as its central header declares it: stored and deflated, a CRC-32 mismatch, a stream that does not inflate, compression method 12, a wrong declared size, a corrupt or out-of-range local header, and data past the end of the file. The entry-count and directory-size tests now give explicit limits or claim more than the new defaults.
  • cargo clippy -p forge-format-nuget --all-targets -D warnings, cargo fmt --check, cargo xtask changelog --check.

No admin API changes, so no web UI screen is needed. Retrying a Nexus import's failed items (Retry failed items on the import's page) already existed, and copies the items this limit refused once this ships.

The issue is in Backlog for now: there is no milestone yet for the next release. The fragments sit under 2.2.0; move both if the next release is 2.1.1.


Generated by Claude Code

A NuGet package of more than 10 000 entries was refused (`limit: 29198
entries exceeds max 10000` in a Nexus import), and raising
`nuget.max_entries` was no way out: to read the manifest, Forge opened
the archive with the zip crate, which indexes every entry (about 400
bytes each, on top of the central directory find_manifest read whole),
so a package of a million entries took about half a GiB to parse.

find_manifest already checks every central header itself; it now reads
the directory through a 64 KiB buffer and keeps the manifest's header
alone, and read_manifest inflates that entry straight from the file
(stored or deflated, its local header read as .NET reads it, its size
and CRC-32 checked). The zip crate is a test dependency only. Memory no
longer depends on the entry count: 192 KiB of heap at 30 000, one
million or three million entries, about 120 ms per million.

The defaults go up accordingly: max_entries 1 000 000 (was 10 000),
max_central_directory_bytes 256 MiB (was 16 MiB). An upgrade note says
to drop the old values a forge.toml copied from the example still sets.

Fixes #619

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012e6DJbkEQw3FgWbbuDgPzX
@schmitch
schmitch enabled auto-merge October 28, 2026 11:11
@schmitch
schmitch added this pull request to the merge queue Oct 28, 2026
Merged via the queue into main with commit f6fdacd Oct 28, 2026
19 checks passed
@schmitch
schmitch deleted the schmitch/epic-goodall-815gln branch October 28, 2026 11:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

NuGet: a package of more than 10 000 entries is refused (a Nexus import fails on one of 29 198), and the cap cannot go much higher

2 participants