Skip to content

Raw repositories as a built-in format plugin over a host file store - #441

Merged
schmitch merged 3 commits into
mainfrom
claude/431-raw-plugin
Oct 24, 2026
Merged

schmitch merged 3 commits into
mainfrom
claude/431-raw-plugin

Conversation

@schmitch

@schmitch schmitch commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

Fixes #431

Stacked on #440, which is stacked on #435. It retargets to main once those merge.

Raw moves under plugins/ next to NuGet. To make that possible, the forge:format contract gets a second storage model, files, alongside package versions. artifact-keeper's format-plugin.wit is path-centric (every artifact has a path, a version is optional). This PR takes that idea and keeps Forge's division of labour: the plugin never sees credentials, never touches storage and never holds a file in memory. Like every plugin after #440, raw is built in.

Contract (forge:format@0.4.0)

  • storage: a new plugin.toml key, storage = "files"; "packages" is the default.
  • New queries: a files format can ask for
    • File(path) and FetchFile(path); FetchFile also asks a proxy's upstream;
    • Directory { directory, after, limit }: a page of a directory;
    • Form: receive a multipart body, with file parts staged;
    • Store(files): from the body or form parts, all or none, under the deployment policy;
    • Remove(path).
  • New action: ServeFile(path).
  • Change queries: Form, Store and Remove each run alone in their round. They need the method's permission, a hosted repository and a package-client credential.
  • render now returns an Action, like handle, so a request can load data and then serve a file. This replaces Rendered.
  • Upstream File(path) operation: the plugin names the URL. The host fetches it, following redirects and sending credentials only to the remote's origin, re-validates it with conditional requests, and keeps the negative cache. The URL is held to the same origin rule as package URLs (ProxyRepo::file_url).
  • Package rules (canonical_id … manifest_summary) get fail-safe defaults, so a files format implements only the protocol.
  • Repository::inline: the repository's content disposition, used for the checksum sidecars.
  • Request path: empty when the URL ends at the repository name, so /raw/<repo> can redirect to /raw/<repo>/.
  • Response headers: link and vary may pass, link only within the repository. Path-absolute redirects are allowed under a public base URL that has a path of its own. A 204 or 304 gets no Content-Length.
  • Path rules: forge_plugin_api::files holds them, shared by the host and the plugin.

Raw

  • plugins/raw holds the protocol: request paths, routing by method, checksum sidecars, HTML and JSON listings, Nexus' raw.directory / raw.assetN.filename form fields, and upstream URLs.
  • crates/server/src/files (formerly crates/server/src/raw, moved with git mv) is the host's file store: storage, deployment policy, ranges and validators, proxies, groups, listings, retention, plus the web UI's upload/delete and the admin API's directory delete. It serves every files format.
  • is_raw() / RAW_FORMAT are gone. Host checks use FormatRegistry::stores_files instead (admin API, retention, members, proxy cache, browse). The retention worker asks Format::stores_files.
  • storage field: GET /admin/formats, the repositories API and /browse/repos now report it (packages, files or images).

Web UI

No screen changes. Its raw screens still key on the format id raw, which is unchanged, and the tree browser reads raw's own JSON listing, which is the raw plugin's protocol rather than the host's. The new storage field is informational.

Upgrade

No upgrade note. Since #440 there are no third-party plugins, and the contract version is checked only against plugin.toml files built with the server. The feature fragment is in 2.0.0. #431 sits in Backlog and moves to 2.0.0 when this merges.

Docker / OCI

Docker is still a native format. Making it a built-in plugin is a separate issue.

Testing

Run on this branch (rebased onto #440):

  • crates/server/tests/raw.rs: 7/7, test file unchanged.
  • .github/scripts/raw-e2e.sh: all checks passed, including the pdfium step against GitHub. This was run before the rebase onto Remove WebAssembly format plugins and Wasmtime: built-in plugins only #440; the files store code is unchanged by it.
  • Server suites plugin_contract, admin, admin_retention, browse, proxy_cache, dispatch, registry and the forge-server unit tests: all pass.
  • Core retention_worker, proxy_read: all pass.
  • forge-plugin-api, forge-plugin, NuGet and raw tests (raw: 12 unit + 8 protocol tests): all pass.
  • cargo fmt --check, cargo clippy --workspace --all-targets -D warnings (and forge-e2e --features forge-e2e) and cargo xtask changelog --check: all clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Kb5ECL544V6eRz8Va8KRHk

The plugin contract (forge:format@0.4.0) gets a second storage model:
a format whose plugin.toml says `storage = "files"` holds files under
paths. Its plugin asks the host for files (`file`, `fetch-file`),
directory pages (`directory`), receives forms and stores and deletes
files (`form`, `store`, `remove`), and serves a file with the
`serve-file` action; the upstream side gets a `file` operation that
names a path's URL. `render` returns an action like `handle`, and the
package rules default to refusing everything, so a files format
implements only the protocol. A request path is empty when the URL ends
with the repository's name.

Raw is now plugins/raw, built in like NuGet: paths, routing by
method, checksum sidecars, HTML and JSON listings, Nexus' form fields
and upstream URLs. crates/server/src/raw becomes the host's file store
(crates/server/src/files): storage, deployment policy, ranges and
validators, proxies, groups, listings, retention, the web UI's upload
and delete and the admin API's directory delete, for every files
format. Host checks for the raw format id become checks for files
storage, and /admin/formats, the repositories API and /browse/repos
report what a repository holds (`storage`).

/raw answers as before: crates/server/tests/raw.rs passes unchanged,
and so does .github/scripts/raw-e2e.sh against GitHub.

Fixes #431

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kb5ECL544V6eRz8Va8KRHk
@schmitch
schmitch force-pushed the claude/431-raw-plugin branch from 92e84b8 to d9d483c Compare October 24, 2026 06:04
@schmitch
schmitch changed the base branch from main to claude/437-remove-wasm-plugins October 24, 2026 06:05
…into claude/431-raw-plugin

# Conflicts:
#	crates/core/src/repo/proxy.rs
#	crates/server/src/admin/retention.rs
Base automatically changed from claude/437-remove-wasm-plugins to main October 24, 2026 06:21
@schmitch
schmitch enabled auto-merge October 24, 2026 06:22
@schmitch
schmitch added this pull request to the merge queue Oct 24, 2026

Copy link
Copy Markdown
Member Author

audit fails on this PR, and the failure isn't from this change. RUSTSEC-2026-0387 was published yesterday for crypto-bigint 0.5.5, which main's Cargo.lock already has: aws-sigv4 pulls it in for the S3 SDK's sigv4a feature. This PR doesn't touch either.

#446 fixes it by dropping sigv4a. I ported the same change here (Cargo.toml, the lockfile as cargo prunes it, and #446's fragment) and checked it locally: crypto-bigint is gone, clippy is clean, and the S3 and raw suites pass. The push was refused because this PR is already in the merge queue. audit is a required check, so the queue run will fail on the same advisory unless #446 merges first.

Either merge #446 before this one, or take this PR out of the queue and I'll push the port. It becomes a no-op once #446 is in.


Generated by Claude Code

Merged via the queue into main with commit dcd1e41 Oct 24, 2026
16 of 17 checks passed
@schmitch
schmitch deleted the claude/431-raw-plugin branch October 24, 2026 06:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Raw repositories as a format plugin: a file store in the plugin contract

2 participants