Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1317,6 +1317,17 @@ Registered when `hosted.enabled` is set. Pushing and deleting require `Authoriza
| `GET /api/hosted/{ecosystem}/{name}` | List the archives pushed for a package |
| `DELETE /api/hosted/{ecosystem}/{name}/{version}` | Delete a pushed version |

### Projects API

Registered when `projects.enabled` is set. Uploading and deleting require `Authorization: Bearer <projects.token>`. See [Project inventory](docs/configuration.md#project-inventory).

| Endpoint | Description |
|----------|-------------|
| `PUT /api/projects/{project}/sbom` | Replace a project's recorded package versions with the ones the SBOM in the body lists |
| `GET /api/projects` | List projects, or with `?purl=pkg:npm/lodash@4.17.20` the projects using a package or version |
| `GET /api/projects/{project}` | A project's recorded package versions |
| `DELETE /api/projects/{project}` | Delete a project |

### Cache API

Registered when `cache_delete` is set. It has no authentication of its own.
Expand Down Expand Up @@ -1520,6 +1531,7 @@ The proxy serves a web UI under `/ui`. No separate frontend build is needed -- t
- **Version detail** (`/ui/package/{ecosystem}/{name}/{version}`) -- per-version metadata, integrity hash, artifact cache status, and hit counts.
- **Source browser** (`/ui/package/{ecosystem}/{name}/{version}/browse`) -- browse files inside cached archives with syntax highlighting for text files and image previews.
- **Version diff** (`/ui/package/{ecosystem}/{name}/compare/{v1}...{v2}`) -- side-by-side diff of two cached versions showing added, removed, and changed files.
- **Projects** (`/ui/projects`) -- with the [project inventory](docs/configuration.md#project-inventory) enabled, the package versions each repository's uploaded SBOM lists, marked when outdated. Package pages then show which projects use each version.

## Monitoring

Expand Down
5 changes: 4 additions & 1 deletion cmd/proxy/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
// @securityDefinitions.apikey BearerAuth
// @in header
// @name Authorization
// @description "Bearer " followed by hosted.token
// @description "Bearer " followed by hosted.token, or projects.token for /api/projects
//
// The proxy caches package artifacts from upstream registries (npm, cargo, etc.)
// providing faster, more reliable access for development teams.
Expand Down Expand Up @@ -93,6 +93,9 @@
// PROXY_HOSTED_MAX_UPLOAD_SIZE - Max size of one pushed archive (default "100MB")
// PROXY_HOSTED_NPM_SCOPES - npm scopes served from pushed packages only
// PROXY_HOSTED_COMPOSER_VENDORS - Composer vendors served from pushed packages only
// PROXY_PROJECTS_ENABLED - Enable the project inventory (/api/projects, /ui/projects)
// PROXY_PROJECTS_TOKEN - Bearer token for uploading and deleting project SBOMs
// PROXY_PROJECTS_MAX_AGE - Remove projects not uploaded for this long (e.g. "90d")
// PROXY_URL - Proxy URL for proxy hosted
//
// Example:
Expand Down
9 changes: 9 additions & 0 deletions config.example.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -442,6 +442,15 @@ gradle:
# npm_scopes: ["@acme"]
# composer_vendors: ["acme"]

# Project inventory (disabled by default). Repositories upload an SBOM to
# PUT /api/projects/{project}/sbom, and the UI shows which project uses
# which package versions. Project names are visible on the unauthenticated UI.
# projects:
# enabled: true
# token: "${PROXY_PROJECTS_TOKEN}"
# # Drop projects that have not uploaded for this long ("" keeps them)
# max_age: "90d"

# Re-mirror container image tags on a schedule, so a moved tag is picked up
# even when nobody pulls it. An unchanged tag costs one HEAD request.
# mirror:
Expand Down
4 changes: 4 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -367,6 +367,10 @@ Selective package mirroring for pre-populating the proxy cache. Supports multipl

The package also provides a `MetadataCache` for storing raw upstream metadata blobs so the proxy can serve metadata responses offline. The `JobStore` manages async mirror jobs exposed via the `/api/mirror` endpoints.

### `internal/inventory`

Turns an SBOM uploaded to `/api/projects` into the package versions recorded for the project. Package URLs are mapped to the ecosystem names and unversioned PURLs the handlers record packages under, so a project's components join with `packages` by `purl`. Also compares versions for the outdated marker on the project pages.

### `internal/config`

Configuration loading.
Expand Down
83 changes: 83 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -1564,6 +1564,89 @@ Generic files never come from an upstream: anything not pushed answers 404. Whil

Pushed archives are stored like cached ones but marked as local. The cache size limit and storage retention never remove them; only a delete does. Scanning, when enabled, checks pushed archives like fetched ones.

## Project inventory

The project inventory answers "which repository uses which version of a package?". Each repository uploads an SBOM of its default branch, and the proxy records the package versions it lists. The UI then shows:

- `/ui/projects`: every project, and per project its package versions, marked when the proxy knows a newer version.
- On a package page: which projects use each version, including versions the proxy hasn't cached.

It is disabled by default. Uploads and deletes require a token:

```yaml
projects:
enabled: true
token: "${PROXY_PROJECTS_TOKEN}"
max_age: "90d"
```

| Config | Environment | Description |
|--------|-------------|-------------|
| `projects.enabled` | `PROXY_PROJECTS_ENABLED` | Register the `/api/projects` endpoints and the UI pages (default `false`) |
| `projects.token` | `PROXY_PROJECTS_TOKEN` | Bearer token required to upload and delete; required when enabled, supports `${VAR}` |
| `projects.max_age` | `PROXY_PROJECTS_MAX_AGE` | Remove projects that haven't uploaded for this long, e.g. `90d` or `720h`; empty or `0` keeps them (default) |

Removing old projects runs hourly. With several instances on one database, one of them does it.

### Uploading an SBOM

`PUT /api/projects/{project}/sbom` takes a CycloneDX (JSON or XML) or SPDX (JSON or tag-value) document of up to 32 MiB and replaces everything recorded for the project with what it lists. The optional `ref` and `branch` query parameters are stored and shown with the project.

The project name is a path such as `group/sub/repo`: segments of letters, digits, `.`, `_` and `-`, at most 255 characters. It can be sent as path segments (`/api/projects/group/sub/repo/sbom`) or URL-encoded as one segment (`/api/projects/group%2Fsub%2Frepo/sbom`).

```bash
syft dir:. -o cyclonedx-json > sbom.json
curl -X PUT --data-binary @sbom.json \
-H "Authorization: Bearer $PROXY_PROJECTS_TOKEN" \
"https://proxy.example.com/api/projects/acme/shop/web/sbom?ref=$(git rev-parse HEAD)&branch=main"
```

The response counts the recorded package versions and the skipped entries:

```json
{"project":"acme/shop/web","components":9,"skipped":6}
```

Entries are recorded by their package URL, with the package named the way the proxy caches it: NuGet IDs lowercased, PyPI names normalized as in PEP 503 (`zope.interface` becomes `zope-interface`), the distribution dropped from `deb`, `rpm` and `apk` package URLs. Entries without a package URL (files, the operating system), package types the proxy doesn't serve (such as `pkg:github`), entries without a version, and conda entries without a `channel` qualifier (the proxy records conda packages as `channel/name`) are skipped. So is the project's own package: the root of a lock file such as `package-lock.json`, and with syft, the Go module its binaries were built from. Duplicates are recorded once.

A GitLab CI job on the default branch could look like this:

```yaml
sbom:
image: alpine:3.20
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
script:
- apk add --no-cache curl
- curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /usr/local/bin
- syft dir:. -o cyclonedx-json > sbom.json
- >
curl -sSf -X PUT --data-binary @sbom.json
-H "Authorization: Bearer $PROXY_PROJECTS_TOKEN"
"https://proxy.example.com/api/projects/$CI_PROJECT_PATH/sbom?ref=$CI_COMMIT_SHA&branch=$CI_COMMIT_BRANCH"
```

### Querying

```bash
curl https://proxy.example.com/api/projects # all projects
curl https://proxy.example.com/api/projects/acme/shop/web # one project's package versions
curl 'https://proxy.example.com/api/projects?purl=pkg:npm/lodash' # projects using lodash
curl 'https://proxy.example.com/api/projects?purl=pkg:npm/lodash@4.17.20' # projects using lodash 4.17.20
curl -X DELETE -H "Authorization: Bearer $PROXY_PROJECTS_TOKEN" \
https://proxy.example.com/api/projects/acme/shop/web
```

Reading needs no token, like the rest of `/api`. With `?purl=`, each project carries the `version` it uses and no `component_count`. A delete names the project itself: `DELETE /api/projects/acme/shop/web/sbom` is answered with 400 unless a project `acme/shop/web/sbom` exists.

A project's page in the UI also has a delete button. It asks for the token and sends it to `DELETE /ui/api/projects/{project}`, which works like the API endpoint and also requires the token; the token is kept for the browser session. Like `/ui/api/cache`, it sits under `/ui`, so rules for the UI at an ingress cover it.

### What it shows

The inventory reflects what the projects' SBOMs list, not what was downloaded through the proxy: a project that installs from another registry still shows up, and the proxy may not have cached any of its packages. The outdated marker compares the version a project uses with the latest version the proxy has recorded for the package; a package the proxy has no record of has none. Versions are compared as the SBOM gives them: a version spelled differently from how the proxy caches it, such as Composer's `v6.4.1` for `6.4.1` or a Debian version with or without its epoch, shows as "not cached" on the package page.

Project names and the packages they use are shown on `/ui` and `/api/projects` without authentication. If that is sensitive, restrict both at your ingress or reverse proxy.

## Mirror API

The `/api/mirror` endpoints are disabled by default. Enable them to allow starting mirror jobs via HTTP:
Expand Down
Loading
Loading