fix: negative-cache property/Connect JSON fetch failures

[JakeSCahill]

Problem

Two of the top 404 sources on docs.redpanda.com come from tooltip data fetches that retry a missing file on every page view:

• 19-property-tooltips.js caches successful properties-JSON fetches for 24h in localStorage, but failures are never cached — when the referenced attachment doesn't exist (release tag drifted ahead of the generated JSON), every streaming page view fires a guaranteed 404 (~17.6k/day for redpanda-properties-v26.1.14.json).
• 16-bloblang-interactive.js builds the Connect JSON URL from latest-connect-version and, on failure, walks a hardcoded fallback-version list — up to 6 404s per page view (~6.4k/day for connect-4.102.0.json), with no caching of failures.

Fix

• Property tooltips: missing-resource markers live in a per-URL map under their own localStorage key (redpanda-properties-missing), so a marker can never clobber the cached dataset and browsing multiple doc versions with missing JSON can't thrash a shared marker. Markers are written only for HTTP 404/410, expire after 1h (vs 24h for successes, so a fix deploy is picked up quickly), are additionally keyed by latest-redpanda-tag, and a successful fetch clears its own URL's marker. The dataset-cache and marker reads use separate try/catch blocks so a corrupt cache entry can't disable the marker check.
• Bloblang: per-URL failure timestamps in localStorage (connect-json-fetch-failures, 1h TTL); URLs that recently returned 404/410 are skipped. Transient failures (429, 5xx, network errors, parse errors) are never marked and retry on the next view. The Connect JSON is now fetched only from the connect-json-url meta tag, which the build resolves against the attachments that actually exist in the catalog (docs-extensions-and-macros#224). The client-side version guessing is removed entirely: the raw.githubusercontent.com antora.yml lookup 404s for all visitors now that rp-connect-docs is private, and the hardcoded fallback versions (4.75.0–4.79.0) name JSON files that are no longer hosted — both in 16-bloblang-interactive.js and in the playground's completion fetch (bloblang-playground.hbs), which now prefers the meta-tag URL and keeps only the GitHub latest-release lookup as a secondary.
• Preview mode (localhost, 127.0.0.1, docs-ui.netlify.app) is unaffected: failures there are never marked, and property-tooltip cache reads were already skipped in preview. Note that content-repo deploy previews (other *.netlify.app hosts) are treated as production.

This is defense-in-depth for the storm; the root cause (meta tags referencing JSON that was never generated) is fixed at build time by redpanda-data/docs-extensions-and-macros#224.

Testing

• New puppeteer suite tests/negative-cache/ (45 assertions, wired into npm run test:all and the Bloblang Tests workflow). It serves the page from a fake production hostname via request interception and verifies: 404/410 are negative-cached and not re-requested; markers expire after the TTL; 429/503/network/parse failures are never cached and retry on every view; success clears the marker, populates the cache, and actually attaches a tooltip in the DOM; preview mode (localhost) never negative-caches.
• node --check and npx eslint on both files (only pre-existing max-len warning remains)

🤖 Generated with Claude Code

[netlify]

✅ Deploy Preview for docs-ui ready!

Name	Link
🔨 Latest commit	aba67b7
🔍 Latest deploy log	https://app.netlify.com/projects/docs-ui/deploys/6a846fd9e38d670008f5e09e
😎 Deploy Preview	https://deploy-preview-407--docs-ui.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
Lighthouse	1 paths audited Performance: 26 (🔴 down 4 from production) Accessibility: 89 (no change from production) Best Practices: 92 (no change from production) SEO: 89 (no change from production) PWA: - View the detailed breakdown and full score reports
🤖 Make changes	Run an agent on this branch

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: ed862a30-4f19-4997-9b00-36287641f77e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

The changes add localStorage-backed failure caching for Connect JSON and property JSON fetches. Connect requests skip URLs recorded as recently failed and record non-OK responses outside preview mode. Property data cache entries now distinguish failed fetches from successful results, using a one-hour failure TTL instead of the 24-hour success TTL, and failed fetches store version and timestamp metadata.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant tryFetchConnectJSON
  participant localStorage
  participant ConnectJSONEndpoint
  tryFetchConnectJSON->>localStorage: Read recent URL failures
  localStorage-->>tryFetchConnectJSON: Return failure status
  tryFetchConnectJSON->>ConnectJSONEndpoint: Fetch URL when not recently failed
  ConnectJSONEndpoint-->>tryFetchConnectJSON: Return non-OK response
  tryFetchConnectJSON->>localStorage: Record URL failure

Loading

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The description clearly explains the negative caching changes, affected fetches, preview behavior, root cause, and validation.
Title check	✅ Passed	The title clearly and concisely summarizes the main change: negative caching for property and Connect JSON fetch failures.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/json-fetch-negative-cache

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[micheleRP]

Right idea — an uncacheable failure retried on every page view is the actual mechanism behind the 404 storm, and negative caching is the correct defense-in-depth alongside redpanda-data/docs-extensions-and-macros#224. The Bloblang half looks good. The property-tooltips half has a problem worth fixing before merge.

Fix before merge

The failure marker writes to the same key as the successful cache, and it's written for far more than 404s.

localStorage.setItem(
  CACHE_KEY,
  JSON.stringify({ version: cacheVersion, timestamp: Date.now(), failed: true })
)

Two things compound here:

1. It clobbers good data. CACHE_KEY is the same entry that holds parsed.data for successful fetches, so writing the marker discards a valid 24-hour cache.
2. It fires on transient failures. This sits in the .catch() of the fetch chain, and the chain throws new Error('HTTP ' + response.status) for any non-ok response. So the catch — and therefore the marker — is reached for a 503, a 429, a user who is briefly offline, and a malformed-JSON parse error, not just the deterministic 404 this PR is targeting.

Together that means one transient blip does this: valid 24h cache destroyed, and for the next hour every page sharing that latest-redpanda-tag resolves to propertiesData = {} — property tooltips silently dead site-wide for that user, with nothing in the UI to indicate why.

This also doesn't match the PR body, which says "Only deterministic HTTP errors are marked — transient network errors still retry." That's accurate for 16-bloblang-interactive.js (which checks !response.ok and never marks on a thrown error), but not for 19-property-tooltips.js.

Two changes fix it:

• Store the marker under its own key (e.g. redpanda-properties-fetch-failed) so a failure can't evict cached data. The Bloblang side already does the equivalent by keying failures per URL in a separate connect-json-fetch-failures entry.
• Mark only on statuses that won't fix themselves — 404/410, or status >= 400 && status < 500 — and let 5xx, 429, and thrown network errors retry. That needs the status plumbed into the catch, or the marking moved to where the response is still in hand.

Suggestion

Bloblang marks on any !response.ok too, which includes 429 and 5xx:

if (!response.ok) {
  if (!isPreviewMode()) markFetchFailure(url)
  return null
}

Less severe than above — the failure is scoped to one URL and can't evict anything else — but a transient CDN 5xx still disables Bloblang tooltips for an hour. Same narrowing to 404/410 applies.

Verified

• The hasRecentFetchFailure guard sits inside the non-preview branch after the meta-tag lookup and the fallback assignment, so it covers both the primary URL and the hardcoded fallback. That's the ~6.4k/day path, and it's correctly covered.
• Preview mode is excluded on both sides: Bloblang never marks and takes the static connect.json path; property tooltips already skipped cache reads in preview.
• readFetchFailures prunes expired entries on every read and markFetchFailure persists the pruned set, so the failure map self-cleans and can't grow without bound.
• Keying the failure TTL by latest-redpanda-tag means a fix that bumps the tag invalidates immediately, and the 1h-vs-24h split is a sensible asymmetry. Worth keeping.

Minor note

The Bloblang fallback URL is built as /redpanda-connect/components/_attachments/connect-${version}.json, but the site serves that content at /connect/... (head-meta.hbs uses component='connect'). Pre-existing and unrelated to this change, but it means the fallback path depends on a 301 — and with this PR that redirect chain now gets negative-cached under the pre-redirect URL. Worth a look while you're in the file.

[JakeSCahill]

Fixed per the review: the missing-resource marker now lives under its own key (redpanda-properties-missing) so it can never clobber a valid cached dataset, it's written only when the fetch failed with HTTP 404/410 (the error now carries status), transient failures (5xx, offline, parse) aren't cached at all and retry on the next view, and a successful load clears the marker. Bloblang half untouched.

[kbatuigas]

Medium — Transient Connect JSON failures are cached for one hour

src/js/16-bloblang-interactive.js:243-246 calls markFetchFailure(url) for every non-successful HTTP response. This includes temporary failures such as 429, 500, and 503.

After one transient response, subsequent pages skip that URL for an hour, leaving Connect/Bloblang documentation unavailable even after the service recovers. The property-data implementation in this PR correctly limits negative caching to permanent 404 and 410 responses.

Please apply the same 404/410 restriction to Connect JSON caching and add tests confirming that temporary failures are retried.

[JakeSCahill]

@kbatuigas Fixed in 08f011c: markFetchFailure is now gated on response.status === 404 || response.status === 410, matching the property-tooltips side, so a transient 429/500/503 is never cached and retries on the next page view.

Tests added as requested: a new puppeteer suite (tests/negative-cache/, npm run test:negative-cache, wired into the Bloblang Tests workflow). Because the negative cache is disabled on localhost, the runner uses request interception to serve the page from a fake production hostname and controls each JSON response's status. It covers: 404 and 410 negative-cached (requested once, then zero on subsequent views), markers expiring after the 1h TTL, 429/503/network-error/parse-error all retried on every view with no marker written, success clearing the marker and populating the cache, and preview mode never caching.

A follow-up review pass surfaced two more issues, fixed in 3082683:

• The properties missing-marker was a single slot, so users browsing multiple doc versions with missing JSON would thrash it and keep firing 404s — it's now a per-URL map like the Connect side.
• With the connect-json-url meta tag set, the 5-version fallback loop could only ever re-request the same URL (up to 6 identical requests per view during a transient failure); it's now skipped in that case, so transient failures fetch exactly once per view.

[JakeSCahill]

Follow-up in 45bd2d7, prompted by the observation that the site already hosts the source of truth for valid versions: the connect-json-url meta tag is resolved at build time against the attachments that actually exist in the catalog (docs-extensions-and-macros#224, merged). Verified against production: the live meta tag points at connect-4.103.1.json (200 OK), and only that newest file is hosted.

That made the client-side version guessing pure 404 fuel, so it's removed:

• getConnectVersion() fetched rp-connect-docs/antora.yml from raw.githubusercontent.com — that repo is now private, so the fetch 404s for every visitor and always returned null.
• The hardcoded fallback lists (4.79.0–4.75.0 in 16-bloblang-interactive.js, 4.78.0–4.75.0 in bloblang-playground.hbs) name versions whose JSON no longer exists — every walk was a guaranteed 404 chain.

16-bloblang-interactive.js now fetches only the meta-tag URL (or the static preview path). The playground's completion fetch prefers the meta-tag URL and keeps the GitHub latest-release lookup as a secondary, since that release's JSON is the one that's hosted. All suites pass: negative-cache (45/45), interactive, and playground.