Repository navigation
NuGet: read a package's zip directory one entry at a time, and take up to a million entries - #621
Merged
Merged
Conversation
A NuGet package of more than 10 000 entries was refused (`limit: 29198 entries exceeds max 10000` in a Nexus import), and raising `nuget.max_entries` was no way out: to read the manifest, Forge opened the archive with the zip crate, which indexes every entry (about 400 bytes each, on top of the central directory find_manifest read whole), so a package of a million entries took about half a GiB to parse. find_manifest already checks every central header itself; it now reads the directory through a 64 KiB buffer and keeps the manifest's header alone, and read_manifest inflates that entry straight from the file (stored or deflated, its local header read as .NET reads it, its size and CRC-32 checked). The zip crate is a test dependency only. Memory no longer depends on the entry count: 192 KiB of heap at 30 000, one million or three million entries, about 120 ms per million. The defaults go up accordingly: max_entries 1 000 000 (was 10 000), max_central_directory_bytes 256 MiB (was 16 MiB). An upgrade note says to drop the old values a forge.toml copied from the example still sets. Fixes #619 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012e6DJbkEQw3FgWbbuDgPzX
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #619
A Nexus import failed some NuGet items with
invalid package: invalid package: limit: 29198 entries exceeds max 10000:nuget.max_entries(default 10 000) refuses a larger.nupkgwhether it is published, proxied or imported. Raising the limit was no real way out. To read the one entry Forge needs, the.nuspec, it opened the archive with thezipcrate, which indexes every entry: about 400 bytes each, measured, on top of the central directory, whichfind_manifestalso read into memory whole. A million entries took about half a GiB per package being parsed.What changes
find_manifestalready checked every central header itself. It now reads the directory from the spooled package through a 64 KiB buffer and keeps the manifest's header alone: its method, CRC-32, sizes and local header offset, with ZIP64 values read in the order the existing checks already make .NET agree with.read_manifest(new) inflates that entry straight from the file. As before, only stored and deflated manifests are read. Its local header is read as .NET reads it (signature, name and extra lengths), its output is capped at the declared size, and the size and CRC-32 must match. The size and compression-ratio caps are unchanged.zipcrate is a dev-dependency only (test packages).flate2andcrc32fastwere in the tree already through it.BudgetReader,VettedDirectory) that bounded thezipcrate's own directory parsing is gone, along with the second open of the archive. Every refusal stays: ZIP64 markers NuGet reads otherwise, Unicode Path fields, two ZIP64 fields, values .NET reads at other places, and repeated or escaped manifests. Their comments now name thezipcrate as the other reader they guard against.max_entriesgoes from 10 000 to 1 000 000, andmax_central_directory_bytesfrom 16 MiB to 256 MiB. Both bound time now, not memory.forge.toml.example, the config docs andNupkgLimitssay so.docs/changes/2.2.0/619.fixed.md, and619.upgrade.md, because aforge.tomlcopied from the example pins the old values.Measurements
These are from a throwaway probe that was not committed: release build, counting allocator, synthetic packages with stored entries.
The same probe parsed five nuget.org packages (
Newtonsoft.Json13.0.3,Microsoft.Extensions.Logging.Abstractions3.1.0 with its repeatedpackageIcon.png,Microsoft.TypeScript.MSBuild5.6.2,Microsoft.NETCore.App.Ref8.0.0 andFSharp.Core8.0.400). Each manifest came out byte-identical to what thezipcrate extracts.Tests
cargo test -p forge-format-nuget: 95 pass. New tests cover a 30 000-entry package (and its refusal at a 10 000 cap), plus a manifest read as its central header declares it: stored and deflated, a CRC-32 mismatch, a stream that does not inflate, compression method 12, a wrong declared size, a corrupt or out-of-range local header, and data past the end of the file. The entry-count and directory-size tests now give explicit limits or claim more than the new defaults.cargo clippy -p forge-format-nuget --all-targets -D warnings,cargo fmt --check,cargo xtask changelog --check.No admin API changes, so no web UI screen is needed. Retrying a Nexus import's failed items (Retry failed items on the import's page) already existed, and copies the items this limit refused once this ships.
The issue is in Backlog for now: there is no milestone yet for the next release. The fragments sit under
2.2.0; move both if the next release is 2.1.1.Generated by Claude Code