infra: agent-friendly docs spec v0.5.0 compliance gaps

[marc0olo]

Background

Ran npx afdocs check https://docs.internetcomputer.org against spec v0.5.0 (2026-04-25).
Result: 14 passed, 1 warning, 6 failed out of 23 checks.

Full spec: https://agentdocsspec.com/spec/

CLI output

content-discoverability
  ✓ llms-txt-exists
  ✓ llms-txt-valid
  ✓ llms-txt-size: 26,425 chars
  ⚠ llms-txt-links-resolve: 48/50 resolve (2 broken)
  ✓ llms-txt-links-markdown: 50/50
  ✓ llms-txt-directive-html: found in all 49 sampled pages
  ✗ llms-txt-directive-md: not found in any of 50 sampled pages

markdown-availability
  ✓ markdown-url-support: 50/50
  ✗ content-negotiation: server ignores Accept: text/markdown (0/50)

page-size
  ✓ rendering-strategy
  ✗ page-size-markdown: 1 page exceeds 100K chars (max 480K)
  ✗ page-size-html: 1 page converts to 524K markdown (79% boilerplate)
  ✗ content-start-position: 34/50 pages have content starting past 50% (worst 91%)

content-structure
  ✓ tabbed-content-serialization
  ○ section-header-quality: skipped
  ✓ markdown-code-fence-validity: 691 fences ok

url-stability
  ✓ http-status-codes
  ✓ redirect-behavior

observability
  ✓ llms-txt-coverage: 100% of 152 sitemap pages
  ✗ markdown-content-parity: 1 page has substantive differences
  ✓ cache-header-hygiene

authentication
  ✓ auth-gate-detection
  ○ auth-alternative-access: n/a

Action items

P1 — Easy wins (code changes in plugins/astro-agent-docs.mjs and a new public/_headers)

• llms-txt-directive-md — inject > For the complete documentation index, see [llms.txt](/llms.txt) near the top of every generated .md file (inside cleanMarkdown()). New in spec v0.5.0.
• content-negotiation — add a public/_headers file that sets Content-Type: text/markdown; charset=utf-8 for /*.md responses. Fixing this may also resolve content-start-position as a side effect: once the checker gets text/markdown responses it will use our clean .md files (which start with content immediately) instead of converting Starlight's nav-heavy HTML.

P2 — Identify and fix the oversized page

• page-size-markdown / page-size-html — find the page producing 480K chars and either split it or trim content. Likely candidates: IC interface spec or management canister reference.

P3 — Investigate after P1 is shipped

• content-start-position — re-run checker after fixing content-negotiation. If the checker then uses .md files for this check, the 34 failures should drop to 0. If not, needs further investigation into Starlight's HTML structure.
• markdown-content-parity — identify which page has content drift and fix the stripMdx() output.
• llms-txt-links-resolve (warning) — find and fix the 2 broken .md links in llms.txt.

Notes

• content-start-position failing for 34/50 pages is most likely because the checker is converting Starlight's HTML (which has full sidebar nav before <main>) rather than using our clean .md files. Fixing content-negotiation is the right lever to pull first.
• Branch convention for this work: infra/agent-docs-spec-compliance

[marc0olo]

Updated status after re-run (2026-04-30)

Before: 14 passed / 1 warning / 6 failed
After PR #182: 17 passed / 0 warnings / 4 failed

Fixed

• llms-txt-directive-md ✓ — directive now in all 50 sampled .md files
• llms-txt-links-resolve ✓ — was likely sampling noise
• markdown-content-parity ✓ — resolved as a side effect of the directive fix

Still failing — findings

content-negotiation — not fixable with IC asset canister hosting. The check requires the server to return markdown for the same URL when Accept: text/markdown is sent (true HTTP content negotiation). The IC asset canister routes by URL path only. .ic-assets.json5 already sets Content-Type: text/markdown on .md URLs, but that's not what this check measures. Fixing it would require an edge worker or CDN middleware. The public/_headers file added in #182 was a mistake (Cloudflare Pages convention); removed in #183.

content-start-position — improved from 34/50 to 29/50 pages failing (worst 91%). Likely caused by the checker converting Starlight's nav-heavy HTML rather than using our clean .md files. Would probably be resolved by content-negotiation working, but that's blocked by hosting constraints.

page-size-markdown / page-size-html — identified the two large pages:

• reference/ic-interface-spec.md — 482K chars (the checker's 480K max). Synced from portal.
• languages/motoko/reference/changelog.md — 101K chars (borderline). Auto-synced from caffeinelabs/motoko.

Both are synced files. Options: exclude from .md endpoint generation, split into sub-pages, or truncate the .md output at 100K with a continuation note. Needs a decision before acting.

Remaining action items (updated)

• page-size-markdown/html — decide approach for ic-interface-spec.md (482K) and motoko/reference/changelog.md (101K): split, truncate, or exclude from .md generation
• content-negotiation / content-start-position — requires edge worker or CDN middleware in front of the IC asset canister; out of scope for now unless hosting changes

[marc0olo]

Deep-dive: content-negotiation — what the check requires and a path to fixing it

What the check actually tests

afdocs sends a GET request to each sampled page URL without any path change, but with the header Accept: text/markdown. It expects the server to return the markdown representation of that page from the same URL, with Content-Type: text/markdown.

This is true HTTP content negotiation — not .md URL support. The checker is asking:

GET /guides/backends/hello-world
Accept: text/markdown
→ expected: markdown body, Content-Type: text/markdown

GET /guides/backends/hello-world
Accept: text/html
→ expected: HTML body, Content-Type: text/html

We already pass markdown-url-support (50/50) because /guides/backends/hello-world.md exists. But content-negotiation is testing the same-URL, different-representation path.

Why the _headers file didn't work

public/_headers is a Cloudflare Pages convention. The IC asset canister ignores it entirely — it was a no-op, correctly removed in #183.

The core constraint: the IC asset canister routes by URL path only. It has no mechanism to inspect request headers and serve a different file for the same path.

Why this is not fixable at the CDN/hosting layer

We don't sit behind a CDN or reverse proxy that we control. Requests go directly to the IC boundary nodes, which forward to the asset canister. There is no Cloudflare Worker, Nginx, or equivalent middleware layer where we could intercept Accept: text/markdown and rewrite the request to <path>.md.

The good news: the asset canister already does this for Accept-Encoding

The ic-asset-certification crate (in dfinity/response-verification) already implements content negotiation for compression. When you request /foo with Accept-Encoding: gzip, the asset canister serves the gzip-compressed variant. It doesn't use the HTTP Vary header to do this — instead it uses IC response certification:

1. At upload time, it pre-computes and certifies separate responses for each encoding of the same URL.
2. The CEL expression for each certified response includes the expected request headers (e.g., Accept-Encoding: gzip).
3. The HTTP gateway verifies the response matches its certification before forwarding.

The internal RequestKey struct is keyed by (path, encoding, range). The Accept-Encoding parser in asset_router.rs extracts and ranks encodings by quality factor (br > zstd > gzip > deflate > identity), selects the best match, and routes to the pre-certified response for that key.

What adding Accept: text/markdown would take

The pattern is identical. The changes needed:

In ic-asset-certification (dfinity/response-verification):

• Extend RequestKey with an accepted_type: Option<String> field (alongside the existing encoding field)
• Add an Accept header parser mirroring the existing Accept-Encoding parser
• When an asset at /foo has a sibling at /foo.md, register a second certified response for /foo keyed to accepted_type: Some("text/markdown")
• Include Accept in the CEL expression for the markdown-variant certified response

In the asset canister (dfinity/sdk):

• At upload time: for each HTML asset /foo, if /foo.md exists in the asset store, register the markdown content as a content-negotiated response for the path /foo
• At request time: check the Accept header for text/markdown and route to the markdown-certified response if available

Why this site would benefit immediately

The agentDocs() plugin already generates /foo.md alongside every /foo HTML page. The asset store already contains both representations. The only missing piece is the routing logic to serve /foo.md's content when /foo is requested with Accept: text/markdown.

Once this works, content-start-position should also resolve as a side effect: afdocs currently falls back to converting Starlight's nav-heavy HTML (because content negotiation fails), which puts content well past the 50% mark. With clean .md responses, that check should pass.

Recommended next step

File a feature request (or PR) on dfinity/response-verification to extend the existing Accept-Encoding content negotiation to also handle Accept: text/markdown. The infrastructure is already there — it is the same certification pattern, applied to a different request header. Every IC-hosted documentation site would benefit.

A thin proxy canister in front of the asset canister could work as a stopgap but adds a round-trip hop and complicates response certification — the upstream change is the correct path.

[marc0olo]

Summary of improvements

Ran npx afdocs check https://docs.internetcomputer.org after the recent batch of PRs. Score moved from 14 passed / 1 warning / 6 failed (original report) to 17 passed / 4 failed.

Fixed

Check	PR
llms-txt-directive-md ✗ → ✓	#182 — inject directive into every .md endpoint
llms-txt-links-resolve ⚠ → ✓	#194 — fix 16 missing reference pages + broken links
markdown-content-parity ✗ → ✓	#198 — include hero.title / hero.tagline from frontmatter in .md output (Starlight splash pages render these in HTML but they were absent from the markdown endpoint)
page-size-markdown / page-size-html (partial)	#188 — split ic-interface-spec from one 480K page into 7 focused sub-pages; worst case down from 480K to 212K

Remaining failures — all blocked or impractical

Check	Status
content-negotiation	Platform limitation: IC asset canister does not support server-side content negotiation. public/_headers was tried in #182 and reverted in #183. Not fixable without changing the hosting platform.
content-start-position (33/50 pages)	Directly depends on content-negotiation. The checker converts Starlight's nav-heavy HTML when it doesn't receive text/markdown — sidebar nav appears before <main>, pushing content past the 50% mark. Not fixable without either resolving content-negotiation or deeply customising the Starlight layout.
page-size-markdown / page-size-html	Two pages remain over the 100K threshold: references/ic-interface-spec/abstract-behavior.md (212K — a dense formal spec section that would need another split) and languages/motoko/reference/changelog.md (101K — synced from upstream, not editable directly). The check is also sampling-based (50 of 158 pages per run), so results vary between runs depending on whether these pages are sampled.

Closing as the actionable items are done. The three remaining failures are either inherent to the hosting platform or require upstream changes.