Skip to content

Add optional security-scan receipt _meta extension (v1) — resolves #1273 - #1404

Open
eeee2345 wants to merge 4 commits into
modelcontextprotocol:mainfrom
eeee2345:feat/security-scan-meta-extension
Open

eeee2345 wants to merge 4 commits into
modelcontextprotocol:mainfrom
eeee2345:feat/security-scan-meta-extension

Conversation

@eeee2345

Copy link
Copy Markdown

Resolves the converged v1 proposal in #1273. Thanks to @JinNing6 and @HarperZ9 for the design discussion in that thread; the shape here follows what the three of us converged on, scoped to the small v1 cut @HarperZ9 outlined.

What this adds

An optional io.modelcontextprotocol.registry/security-scan extension under the existing _meta reverse-DNS namespace, holding an array of evidence-scoped, scanner-neutral security scan receipts. It sits next to io.modelcontextprotocol.registry/publisher-provided and follows the same convention, rather than adding a bare top-level field.

Each receipt binds a verdict to the exact evidence that produced it: scanner, scanner_version, rule_set_ref, policy_profile, scanned_artifact_ref, scanned_artifact_digest, scan_scope, verdict, scanned_at, freshness_expires_at, evidence_ref, evidence_digest, and attestation. The point is that clean only ever means clean under this scanner version, rule set, policy profile, and artifact digest, for the listed scope, not a server-level safety property.

The client invariant @HarperZ9 named is enforced in the schema, not just prose:

  • scanned_artifact_digest is required, so a verdict binds to exact bytes and a client can reject a receipt that does not join to the current package or artifact.
  • scan_scope is a required, non-empty array describing what was actually evaluated, so dependency or package coverage is machine-distinguishable from handler-side validation.
  • verdict includes inconclusive as a first-class value, and an inconclusive verdict requires a machine-readable inconclusive_reason (artifact_digest_mismatch, unsupported_package_type, scope_excludes_handler_validation, evidence_unavailable, stale_scan). So a scan that leaves handler-side validation unassessed stays representable rather than collapsing into clean.

attestation (publisher-asserted, registry-attested, third-party-attested) keeps a self-asserted receipt distinct from an attested one. Signatures and capability posture are intentionally out of scope for this version; the docs note capability posture as a possible future sibling under the same namespace, and attestation plus evidence_digest carry verifiability for now without pulling in key distribution.

scanner and rule_set_ref are open strings so any community scanner can populate them; the registry does not endorse any particular one.

Scope and conventions

  • Schema is defined in docs/reference/api/openapi.yaml and regenerated with make generate-schema, per docs/reference/server-json/CONTRIBUTING.md. server.schema.json is in sync (make check-schema passes).
  • The receipt schema uses additionalProperties: true so it can grow without breaking existing validators.
  • The change is additive and optional; existing server.json documents remain valid.
  • official-registry-requirements.md notes that the official registry preserves only the publisher-provided key today, so a publisher-asserted receipt is carried nested under publisher-provided, with an example. registry-attested and third-party-attested are reserved for when the registry supports them. This keeps the doc honest about current behavior versus the format-level shape.

Files

  • docs/reference/api/openapi.yaml: SecurityScanReceipt component plus the new _meta key
  • docs/reference/server-json/draft/server.schema.json: regenerated
  • docs/reference/server-json/generic-server-json.md: format documentation and example
  • docs/reference/server-json/official-registry-requirements.md: current preservation behavior and publishing example
  • docs/reference/server-json/CHANGELOG.md: Draft (Unreleased) entry

Validation

make generate-schema, make check-schema, and make validate (schema validity, sync, and all doc examples against both the JSON schema and the Go validator) pass locally. go build ./..., go vet, and go test ./internal/validators/... pass. The change touches only schema and docs, no Go code, API handlers, or database, so existing server behavior is unaffected.

Happy to adjust naming or field shapes to match registry conventions, and to split anything out if a smaller first cut is preferred. Reviews welcome async whenever it suits.

Adds an optional io.modelcontextprotocol.registry/security-scan extension
under the existing _meta reverse-DNS namespace, holding an array of
evidence-scoped, scanner-neutral security scan receipts. Resolves the
converged v1 proposal in modelcontextprotocol#1273.

Each receipt binds a verdict (clean | warnings | findings | inconclusive)
to a specific scanner, rule_set_ref, policy_profile, and
scanned_artifact_digest, with an explicit machine-readable scan_scope of
what was actually evaluated. The schema enforces the client invariant
named in-thread: scanned_artifact_digest is required so a clean verdict
binds to exact bytes, scan_scope is a required non-empty array, and
inconclusive verdicts require a machine-readable inconclusive_reason
(artifact_digest_mismatch, unsupported_package_type,
scope_excludes_handler_validation, evidence_unavailable, stale_scan) so a
dependency or package scan that leaves handler-side validation unassessed
stays representable rather than collapsing into clean.

attestation (publisher-asserted | registry-attested | third-party-attested)
keeps self-asserted receipts distinct from attested ones. Signatures and
capability posture are intentionally out of scope for this version.

scanner and rule_set_ref stay open strings so any community scanner can
populate them; the registry does not endorse any particular one.

Schema is defined in openapi.yaml and regenerated via make generate-schema.
Documented in generic-server-json.md and official-registry-requirements.md,
with a CHANGELOG entry. Additive and optional; existing server.json
documents remain valid.

Signed-off-by: Adam Lin <adam@agentthreatrule.org>
@JinNing6

Copy link
Copy Markdown

Thanks for turning the discussion into a concrete PR.

This looks like the right v1 boundary to me: optional, scanner-neutral, _meta-scoped, and evidence-bound rather than a server-level safety badge. I especially like that scanned_artifact_digest, non-empty scan_scope, attestation, and first-class inconclusive make the client behavior checkable instead of relying on scanner-specific prose.

Keeping capability_posture out of this PR also seems right. The receipt should answer “what evidence was checked?”, while posture can remain a future sibling field for “what authority does this server expose?”

From my side, I would support this as a small additive schema/docs change. Happy to review wording or field-shape adjustments if maintainers prefer a narrower first cut.

@HarperZ9

Copy link
Copy Markdown

This is the right PR to converge on from #1273. I opened #1405 before noticing this one, but closed mine to avoid splitting review.

The important invariant is preserved here: clean stays evidence-scoped, because it has to bind to scanned_artifact_digest and a non-empty scan_scope, while inconclusive remains first-class with a machine-readable reason. The official-registry note is also useful because it keeps current preservation behavior separate from the generic format-level extension.

One downstream client test I would add when this lands: a receipt with verdict: clean and matching artifact digest can render only with the displayed scan_scope; a digest mismatch or stale receipt must degrade to inconclusive rather than a clean badge.

@eeee2345

Copy link
Copy Markdown
Author

@HarperZ9 thanks — and for closing #1405 to keep review in one place. Agreed on the invariant: the digest binding plus a non-empty scan_scope is what keeps clean from drifting into a server-level claim, and splitting the official-registry preservation note from the format-level extension was the cleaner cut.

Good call on the downstream client test — a receipt with verdict: clean and a matching scanned_artifact_digest accepted, versus a mismatched digest or empty scope rejected. I'll add that so the binding is covered by a test, not just the schema shape. And happy to tighten scan_scope to an enum if the maintainers would rather a smaller v1 surface.

eeee2345 added 2 commits June 30, 2026 02:33
…iant

Adds a downstream-client test for the
io.modelcontextprotocol.registry/security-scan extension (modelcontextprotocol#1404) and a
named render-invariant note in the format spec.

The test compiles the canonical draft server.schema.json (the same file
tools/validate-examples compiles) and validates _meta security-scan
receipts the way a client would read them:
- clean receipt with a well-formed scanned_artifact_digest and a
  non-empty scan_scope is accepted
- mismatched/malformed scanned_artifact_digest (not algorithm:hex) is
  rejected, and a missing digest is rejected
- empty scan_scope is rejected (minItems 1)
- verdict inconclusive without inconclusive_reason is rejected, and is
  accepted once the machine-readable reason is present

generic-server-json.md gains a Render invariant subsection making the
client rule explicit: do not surface clean unless the receipt binds to
the current artifact digest and the displayed claim names the covered
scan_scope; an unbound digest or empty scope must be treated as
inconclusive rather than clean.

Signed-off-by: Adam Lin <adam@agentthreatrule.org>
golangci-lint flagged cloneReceipt as unused (the table cases build maps
inline, no clone needed). Removing it; the schema test is unchanged.

Signed-off-by: Adam Lin <adam@agentthreatrule.org>
@eeee2345

eeee2345 commented Jul 8, 2026

Copy link
Copy Markdown
Author

@HarperZ9 following up when you have a moment — the security-scan receipt _meta extension is ready and I believe we're aligned on the digest-binding + non-empty scan_scope invariant. Happy to adjust anything before review.

@HarperZ9

HarperZ9 commented Jul 8, 2026

Copy link
Copy Markdown

@eeee2345 looks good from my end. :)

@Xander-Xai

Copy link
Copy Markdown

Hi maintainers — I’m building an independent downstream consumer for the experimental security-scan receipt shape in this PR: mcp-evidence-gate.

I pinned a compatibility profile to the current PR head SHA (20747d3253ba8638161dd95f1cec70df02993c22) and implemented a local verifier plus a self-contained GitHub Action. The consumer currently:

  • validates the pinned receipt structure;
  • binds scanned_artifact_digest to the exact artifact under review;
  • treats digest mismatch and policy-configured staleness as inconclusive, rather than rewriting the scanner verdict or making a server-level safety claim;
  • requires a machine-readable reason when the receipt verdict is inconclusive.

The Action is currently referenced at immutable release SHA 13bd12875a2d9381b518c0b543549ca89cbc42b8 and has been exercised from a separate dogfood repository with both PASS and expected INCONCLUSIVE runs. This is a downstream consumer experiment, not an official Registry implementation.

Could you confirm whether these are still the intended consumer-facing invariants for v1?

  1. Should digest binding and non-empty scan_scope remain the minimum conditions for interpreting a clean receipt?
  2. Should digest mismatch or stale evidence remain a consumer/policy-level inconclusive outcome rather than a mutation of the receipt’s original verdict?
  3. If the proposal changes after this PR, would a versioned compatibility profile be the preferred way for downstream clients to track the change?

I’m happy to adjust the consumer or add focused fixtures based on maintainer feedback.

@Xander-Xai

Copy link
Copy Markdown

Downstream follow-up with implemented edge cases:

I added consumer-side checks in mcp-evidence-gate#1 and executable cases in dogfood#1:

  • scanned_at more than five minutes in the future is invalid;
  • freshness_expires_at < scanned_at is invalid;
  • a strict consumer policy caps scan age at seven days, so a publisher-selected far-future expiry cannot bypass max age.

Should these remain downstream consumer-policy concerns, or should any become documented receipt invariants? The verifier also treats attestation as declarative metadata because the proposal does not define a trusted issuer/signature contract.

@Nikolife2016

Copy link
Copy Markdown
Contributor

One observation from an angle this thread has not covered, offered as data rather than as a design objection — the digest binding looks right to me and I am not asking for it to change.

The digest binds bytes. Some risk moves without the bytes moving.

scanned_artifact_digest makes clean fall out of scope the moment the artifact changes, which is exactly right for anything a scanner reads inside the package. But a class of supply-chain events leaves the scanned bytes byte-identical and changes who controls the next bytes:

  • package ownership transferred to a different maintainer set,
  • repository link removed, so the source behind a still-valid digest can no longer be reviewed,
  • repository moved to a different origin,
  • package unpublished, freeing a name that carries accumulated installs,
  • a previously removed name republished by someone else.

A receipt bound to digest D stays perfectly joinable through all of those. Nothing in the current shape is wrong; the axis simply is not artifact-scoped.

How often, measured rather than asserted. We run a daily external diff of the whole registry population (25,030 entries, series from 2026-07-30). Across the events of that class so far:

packages with an ownership/source event      106
of those events, no version published same day  53   (44.9%)
alongside a version publish                     65

So roughly 45% of ownership and source changes arrive with no artifact change on the same day — a receipt with verdict: clean would still join, and freshness_expires_at would still be in the future. Recent concrete ones: @doist/todoist-mcp, @fabric-harness/sdk, @edgescaleai/cube-mcp (maintainer set changed), ai.bowmark/bowmark (repository moved).

Two things this is not. It is not an argument for widening v1 — the scope discipline in this PR is the reason it is reviewable. And it is not a case for a drift field: drift is an event stream with a publisher, not a property of a server record, and stuffing it into a point-in-time receipt would recreate exactly the server-level-safety-badge failure mode this design avoids.

What I think it does argue. If scan_scope is the machine-readable answer to what was evaluated, then the values a scanner may put there are where this belongs — a receipt whose scope covers package contents at digest D is saying nothing about custody of the name, and a consumer should be able to see that distinction rather than infer it. Whether that is worth a documented scope value in v1 or a note in generic-server-json.md is a call for the people who set the boundary here.

Happy to contribute the ownership-change series as public evidence if it is useful — it is free, needs no key, and the format is whatever is convenient. Not asking for anything in return, and not proposing our scanner as a value for scanner; the point is the axis, not us.

Content/digest-scoped receipts assert nothing about who controls the name the
artifact is published under; a clean receipt bound to a digest stays joinable
through ownership transfer, source-repo removal/relocation, unpublish, or name
republication. Custody is a distinct axis, not scanner-assessable, so it is a
documented scope boundary (clients MUST NOT infer custody from a content scan)
rather than a new scan_scope value or inconclusive_reason. Motivated by the
registry-population ownership-change discussion on the PR; no schema change.

Signed-off-by: Adam Lin <adam@agentthreatrule.org>
@eeee2345

Copy link
Copy Markdown
Author

@Nikolife2016 — this is the kind of contribution the thread needed, and framing it as data rather than a design objection is what makes it land. The measurement carries it: across your daily registry diff, ~45% of the ownership/source events you tracked arrived with no artifact change the same day, so a verdict: clean receipt bound to digest D stays joinable and freshness_expires_at stays in the future while custody of the name has already moved. That's a real axis this design doesn't cover — a content/digest-scoped receipt says nothing about who controls the next bytes.

I want to agree explicitly with the two things you ruled out, because they're the same discipline that keeps this PR reviewable: not widening v1, and no drift field. The second especially — drift is a publisher-owned event stream, not a property of a point-in-time receipt, and folding it in would recreate exactly the server-level-safety-badge failure mode v1 exists to avoid.

Where this lands is scan_scope semantics, and I've pushed it (#### Scope boundary: name custody in generic-server-json.md): a note making explicit that a receipt whose scope covers package contents at a digest asserts nothing about name custody, ownership continuity, or source-link availability, and that a consumer MUST NOT infer custody from a content scan — v1 field shapes unchanged. inconclusive_reason already models a scan that reached no verdict; custody is a different axis — not scanner-assessable at all — so the honest home is a documented scope boundary, not a verdict value.

Yes to the offer — please share the ownership-change series; I'd like to cite it as the measured motivation for that boundary, since measured beats asserted. On whether this is a reserved scope value in v1 or prose I'll defer to whoever owns the boundary here; my lean is the docs note now, with an optional scope vocabulary reserved for a later cut so the v1 schema doesn't grow. (And +1 to @yandexuanxuan's consumer-side invariants upthread.)

@Nikolife2016

Copy link
Copy Markdown
Contributor

Series published, and one correction to the number before you cite it.

The artifact: https://pulsefeed.dev/evidence/name-custody.json — free, no key, CORS-enabled, stable URL. It carries the full event list, not just the counts, so the figure can be recomputed rather than taken from me.

The number moved, and you should cite the current one. I quoted 44.9% upthread. Today the same computation reads:

custody events                     177   across 172 distinct packages
no version published same day       82   (46.3%)
version published same day          95

by type: repo_changed 68 · unpublished 50 · maintainer_changed 38 · repo_removed 21

Nothing was restated — the series simply grew by four days. If a stable figure is wanted for a docs note, "roughly half" survives both readings and will survive the next one; a precise percentage in prose will be stale within the month, which is the same reason I would keep the number out of the schema.

Four limits, and they are inside the file rather than in this comment, because the file is what gets cited:

  • The series begins 2026-07-30. Anything earlier is invisible to us and is not counted as absence — this is a floor on history, not a claim about it.
  • Resolution is same calendar day (UTC), not same instant. A version published hours after a handover on the same day counts as accompanied. So 46.3% is a lower bound on silent handovers, not an estimate of them.
  • Coverage is the MCP registry population plus npm packages reachable from it. Sellers outside those surfaces are outside the measurement — a blind spot we found by checking our own instrument against a live endpoint we could not see, and reported publicly rather than discovering it later.
  • We are an interested party: we publish a drift feed. That is why the raw events ship with the counts.

Each day's dataset manifest is hashed and timestamped into Bitcoin via OpenTimestamps, so the series can be shown to have existed on a given date without trusting us. Current anchored digest is in the file's verify block. Timestamping proves existence before a block and nothing about correctness — worth saying plainly if it appears near a standard.

On your placement call: the docs note now, with a reserved scope vocabulary deferred, reads right to me — and I say that as someone with an obvious interest in custody becoming a first-class field, which is a reason to discount my agreement rather than to weight it. inconclusive_reason genuinely does not fit: custody is not a scan that failed to reach a verdict, it is a question no content scanner is positioned to ask.

If the wording of the boundary section would be better with the measurement stated differently, say so and I will produce whatever cut is most useful — including one that makes our own coverage limits explicit, since a boundary note that overstates its evidence is worse than none.

@Nikolife2016

Copy link
Copy Markdown
Contributor

Correction to the figure I posted upthread (44.9% → 46.3%). It pools four event types, and two of them cannot be "accompanied by a version" by construction, so the pooled share overstates silent handovers. Recomputed today from the same file, split by type:

type                 events   no same-day version
maintainer_changed       42        32   (76.2%)
repo_changed             77         1   ( 1.3%)
unpublished              60        60   (100% — the package is gone, nothing to publish)
repo_removed             25         0   (a repo field disappears with a release, every time)

The honest statements are narrower and, I think, more useful for the scope-boundary section:

  • Maintainer handovers are usually silent: 32 of 42 carried no same-day release. That is the case a digest-bound receipt stays joinable across.
  • Repository moves are almost never silent (1 of 77). A repo change is normally part of a publish, not a custody event in disguise.
  • Pooled over transfers only (maintainer + repo): 33 of 119, 27.7%, not "roughly half".

/evidence/name-custody.json now carries the split (counts.maintainerChanges, counts.transfers, counts.byTypeSilent) and a caveats array saying exactly this. The pooled field stays for continuity of earlier citations but should not be quoted as the headline. The text of the PR does not carry the number, so nothing in the spec needs to move — this is for the record, since I asked for it to be cited.

@Nikolife2016

Copy link
Copy Markdown
Contributor

The custody series is now frozen and citable: https://doi.org/10.5281/zenodo.22251006 (CC BY 4.0). It carries the event list as JSON and CSV, a stdlib recount script, the corrected per-type figures (maintainer handovers silent in 32 of 42; repository moves in 1 of 77), the signed daily manifest and the OpenTimestamps proofs for the deposit day, with the coverage limits stated in the README (deep npm subset for maintainer events, whole population for registry-field events). If the scope-boundary section ever wants a citation, that DOI is stable; the live file at /evidence/name-custody.json keeps moving.

@Nikolife2016

Copy link
Copy Markdown
Contributor

One more correction to my own numbers, and then I will stop moving them. Upthread I called the same-day shares lower bounds. That was wrong: both the custody change and a release are inferred by diffing consecutive daily snapshots, so their true times are interval-censored, and a change and a release that fall on different sides of a snapshot boundary were counted as unaccompanied even if they were hours apart. External review (I. Santos-Grueiro) caught it.

Same 204 events, now with a second flag that tolerates a release in an adjacent observation (±1 day):

maintainer-set changes      42   no release, same observation 32 (76.2%)   same or adjacent 23 (54.8%)
repository URL changes      77                                 1 ( 1.3%)                     1 ( 1.3%)

So the honest statement is "at daily observation resolution", with the ±1 column showing how much the boundary matters — and a maintainer-set change is not an established transfer of control; the layers (package authority, registry-name authority, repository provenance) are now reported separately. Dataset v2 with both flags per event and the recount script: https://doi.org/10.5281/zenodo.22268322 (concept DOI for all versions 10.5281/zenodo.22251005). The live file at /evidence/name-custody.json carries the same two readings under counts.maintainerChanges and counts.adjacentObservation.

@ciceroyang

Copy link
Copy Markdown

Independent data point for this design, plus one concrete signal I would suggest the receipt carry. I run a small census of the registry (read-only: registry metadata + published npm documents + the referenced hook scripts fetched from the published package).

Census, September 2026 (6,000 registry entries, reproduced weekly):

  • 2,142 unique servers; 297 declare a package (npm 245, pypi 38, oci 8, mcpb 6); 1,845 are remote-only.
  • 16 of the 245 npm packages run code at install time (preinstall / install / postinstall); 11 of those download and then execute a platform binary (resolve archive from a GitHub release, fetch, verify a SHA-256 sidecar, extract with tar/unzip/PowerShell, install).
  • 52 package-declaring servers are in registries my scopes do not cover; I report them as not clean rather than clean.

Report: https://github.com/ciceroyang/mcp-supply-audit/blob/main/docs/census-2026-09.md · rolling artifact: https://github.com/ciceroyang/mcp-supply-audit/releases/tag/mcp-census · scanner (MIT, zero-dep): same repo.

Suggested signal: install_time_execution.

This is a property of the published artifact, not of handler-side validation, so it reads naturally as a scan_scope entry. Machine-checkable values with the evidence that produced them:

  • none — no install hook in the declared version.
  • hooks_present — hooks exist; receipts list the exact scripts entries.
  • hooks_fetch_or_spawn — a hook (or a script it references) fetches over the network, spawns a process, or decodes-and-executes. The receipt should point at the evidence: for node scripts/install.js the referenced file is fetched from the published package at the declared version, which binds the finding to scanned_artifact_digest exactly as your invariant requires.

Two properties of that pattern are worth a scope flag rather than a bare verdict, because they change what the verdict means: (a) the expected checksum is fetched from the same origin as the artifact, so verification covers transport corruption but not a compromised release origin; (b) the base URL, repository and version are overridable through environment variables (LABBY_RELEASE_BASE_URL, LABBY_REPO, LABBY_VERSION), so a redirected download is checked against a checksum from the same redirected location.

Why the receipt shape matters, from experience. My detector's critical tier changed three times in one day: v0 flagged any node -e hook and had precision 0/3 on real artifacts; v0.1 added content inspection of referenced scripts and still flagged string literals (an install notice containing a URL; a warning containing curl ... | sh); v0.2 matched commands raw and scripts after stripping strings/comments, guarded by a hand-labeled corpus. A verdict without scanner_version + rule_set_ref + scan_scope is not stable enough to publish — which is exactly what this proposal requires. A consumer that had cached the v0 verdict would have cached three false positives.

If the field lands I am happy to emit receipts in this shape from the census; the scanner already binds findings to the declared version and the fetched script bytes.

@ciceroyang

Copy link
Copy Markdown

Following the census comment above with one concrete failure mode for the receipt to cover, because it is the case scan_scope exists to catch.

A scanner that crashes can still produce a clean-looking report. In AgentAuditKit 0.6.5 a scanner exception becomes an AAK-INTERNAL-SCANNER-FAIL finding at INFO severity, which is below the default reporting floor and below anything --fail-on accepts. A single file with invalid UTF-8 takes down four scanners, including the MCP configuration one. The summary reads {"info":4,"total":4,"reported":0} with an empty findings array, --score still returns 100 A, the SARIF output has zero results, and --ci exits 0. Reported at sattyamjjain/agent-audit-kit#743.

The design implication I would draw: a receipt that says clean should be able to say which components actually ran. If scan_scope is a free-form string, a scanner that never executed looks the same as one that ran and found nothing. Two things would close it — a set of scanners/rules that completed, and an explicit incomplete state that cannot coexist with clean. The digest binding stops clean from drifting into a server-level claim; this stops it from drifting into a claim about work that did not happen.

Copy link
Copy Markdown

Follow-up with executable downstream evidence for the incomplete-scanner failure mode raised by @ciceroyang above.

We implemented and promoted a fail-closed scanner-execution-completeness path across an independent Producer -> Core -> Dogfood chain, without changing the v1 SecurityScanReceipt schema or the pinned compatibility profile.

Promoted immutable identities:

  • Producer: Xander-Xai/mcp-evidence-producer-trivy@3b4862245ce1778d52d6a3b58f8b1b8cb4906dfb (PR #4)
  • Core consumer: Xander-Xai/mcp-evidence-gate@1c5a6cfae2901b97fc0925d0b102710d9a73cb82 (PR #14)
  • Dogfood acceptance: Xander-Xai/mcp-evidence-gate-dogfood@2c7819214e9d1c4295c3d6fc9d3c3b23a2d81221 (PR #12)

What the experiment now proves:

  • Producer-owned evidence.json carries a project-defined scanner_execution record with required/completed/failed components, process/exit state, output presence/parseability, and semantic-consistency state.
  • clean is eligible only when all required scanner work is complete. A parseable Trivy report with Results: [] is treated as trivy_result_sections_missing and cannot become clean.
  • Incomplete, failed, missing, malformed, or contradictory execution evidence cannot produce a passing strict admission decision. A digest mismatch makes execution evidence unverified and the downstream admission inconclusive rather than clean.
  • Core consumes the exact same detached evidence bytes for digest binding and scanner-execution validation, so a path-replacement A/B race cannot make it hash one report and interpret another.
  • Dogfood reran the promoted Producer and promoted Core together, including real Trivy, OSV, OCI exact-identity, multi-receipt composition, Results=[], incomplete/failed/missing/malformed/contradictory/tampered cases, and the TOCTOU fixture. Post-merge exact-main hosted runs completed successfully; the aggregate run was 35112191883 and scanner-completeness acceptance was 35112191533.

The boundary is intentional: this does not add scanner_execution to Registry v1, does not add install_time_execution, and does not ask the Registry to standardize the producer-specific completeness contract. The compatibility profile remains pinned to registry-pr-1404@20747d3253ba8638161dd95f1cec70df02993c22; the current PR-head drift remains non-contract for this consumer.

So the downstream result is simply: absence of findings is not accepted as evidence that required scanner work completed. This is project-owned executable evidence, not a claim of Registry adoption or maintainer approval.

@ciceroyang

Copy link
Copy Markdown

Thanks — the executable chain is what makes this checkable rather than asserted. Two things I would take from it:

  1. The completion set has to travel in the receipt. scan_scope as free text cannot separate "ran and found nothing" from "never ran"; your scanner_execution record (required / completed / failed, process and exit state, output presence and parseability, semantic consistency) is the minimum that closes it.
  2. incomplete must be a state that cannot coexist with clean. Your Results: []trivy_result_sections_missing case is the failure I raised, and digest mismatch → inconclusive rather than clean is the right default.

For the v1 extension I would keep the schema unchanged and ask only that a producer publishing a receipt either includes such a completion record or leaves clean unavailable — optional receipt, non-optional honesty about coverage. I will map my census output onto that shape so the two can be compared field by field.

@ciceroyang

Copy link
Copy Markdown

One correction to my comment of 15 September: the scanner is AGPL-3.0, not MIT. The repository is the extract that became agentgate's collection package, and it ships under one licence. Sorry for the noise.

ciceroyang added a commit to ciceroyang/agentgate that referenced this pull request Sep 17, 2026
Every index entry now carries a scanExecution block: which scanners were required,
which completed, and whether their output was present, readable and self-consistent.
A required component that failed, was skipped or produced unreadable output makes
the record incomplete, and build-index enforces that rather than leaving it to
convention: the verdict is incomplete however clean the findings look.

The mapping is the one I said I would do in modelcontextprotocol/registry#1404.
Field names inside scanner_execution - required, completed, failed, status,
output_present, output_parseable, semantic_consistency - follow that thread's
shape on purpose, so the two records can be diffed field by field instead of
argued about in prose. Everything outside the block stays in the local style.

Also in this change: a validateScanExecution reference implementation with no
dependencies, a JSON Schema for other consumers, and a test that asserts the two
agree on the required fields so the schema cannot drift from the code that writes
the records. The spec is docs/spec/scan-execution-v1.md.

12 tests in packages/collect/test/execution.test.mjs, including that nothing
required is not a pass, that an unknown status is treated as failed, and that the
index refuses to call an unfinished scan clean. Full suite 536 pass / 0 fail.
@ciceroyang

Copy link
Copy Markdown

Implemented on my side, mostly so the comparison stops being rhetorical: each evidence block in my census output is now one component in exactly this shape — required, status, output_present, output_parseable, semantic_consistency, per-severity counts — and the index builder refuses to emit clean when any required component did not finish.

The field names match this thread's on purpose, so a producer and a consumer can diff two records field by field rather than in prose. Spec and the mapping table from our block statuses: https://github.com/ciceroyang/agentgate/blob/main/docs/spec/scan-execution-v1.md

@ciceroyang

Copy link
Copy Markdown

Coming at this from an implementation that already consumes this shape, so a concrete gap rather than a naming opinion.

We record a sibling block per index entry — scanner_execution — with the fields you list plus four that this receipt cannot currently express:

  • required / completed / failed roll-ups, and per component: status, exit_code, output_present, output_parseable, semantic_consistency (ok / unverified / …), and per-severity finding counts.

Why those earn their place: the failure we built around is a scanner that crashes or emits an unparseable report while the wrapper still produces a low-severity finding and exit code 0 — the receipt then reads like a clean scan. inconclusive covers the cases you enumerated (evidence_unavailable, stale_scan, digest mismatch), but not "the process did not finish", "the output was not parseable", or "the counts contradict the listed components". In our index that is the majority state, not an edge case: of 2,055 records, 12.8% have a fully measured coverage state and 87.2% are incomplete — and for most of those the reason is that no scanner produced usable output for the record (a registry-only server with nothing published to inspect), not that a finding was raised.

Two smaller things we learned that are cheap to state normatively:

  1. A consumer that meets an unknown status, state or semantic_consistency must treat it as unmeasured, never as clean. With additionalProperties: true the schema can grow, and that one rule is what keeps growth from silently upgrading a record to a pass. The same applies to an unknown severity value in the per-severity counts.
  2. scan_scope works better as a denominator when the receipt also states how many scanners were required versus completed — otherwise a receipt that omits a scanner is indistinguishable from one where that scanner was never required.

Spec we implemented against (frozen; the field names were aligned to this PR's shape so the two records can be compared field by field): https://github.com/ciceroyang/agentgate/blob/main/docs/spec/scan-execution-v1.md

It runs over ~2,000 registry records a day; the coverage distribution is public at https://xn--5kvo87g.com/evidence.html

Happy to open a follow-up PR adding an optional execution/coverage object to the receipt if that is useful to the thread. If you would rather keep v1 minimal, the consumption rule in (1) is a one-paragraph addition to the docs.

@Xander-Xai

Copy link
Copy Markdown

Governance evidence refresh (2026-09-21):

  • Producer promoted main: 12dc756ddda7789081d58854b9b2c45587457513; required verify-producer + verify-oci; hosted promotion checks PASS.
  • Core promoted main: 8c6d19b9ad90d6f066fa6fd1f41d0a8da3b017f3; required verify; hosted CI/profile-drift checks PASS.
  • Dogfood promoted main: ce70eaf4a69e250d20d88784c80ac4809125d17f; required promotion-gate + adversarial-matrix; full refresh PR checks PASS.
  • Governance promoted main: 12c62e4407309229182d2e7f4537815130f66319; required governance-self-test; promoted-main run PASS.

All four main branches now require pull requests, strict required checks, and disallow force-push/deletion. This is project evidence for the pinned compatibility profile; it does not assert Registry adoption or third-party adoption.

Copy link
Copy Markdown

@ciceroyang thanks — the independent implementation is useful evidence, especially because it exercises the same failure mode from a separate producer/consumer path rather than only mirroring our own tests.

From the downstream-consumer side, the two interoperability rules you identified match what we have learned:

  1. Unknown execution/status/semantic-consistency values must remain unmeasured or inconclusive; they must never be interpreted as clean.
  2. Coverage needs an explicit denominator — required versus completed work — otherwise omission is indistinguishable from “not required”.

Our current scanner_execution contract is still project-defined evidence attached outside the Registry v1 receipt shape, so I would not treat our implementation as an argument that Registry v1 must expand.

For interoperability, a small docs-level consumption rule covering unknown execution states would already be valuable. An optional execution/coverage object could be explored separately if the PR owner or maintainers want that additional surface.

The independent implementation is also useful corroboration for the narrower claim we can actually make today: execution completeness is implementable across separate producer/consumer systems without turning clean into a server-level safety claim.

Thanks for publishing the mapping and implementation.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants