[devhims]

Implements the video index retrieval guideline (.scratch/video-index-retrieval/PRD.md) behind AGENT_TRANSCRIPT_RETRIEVAL, which defaults to full. Production behavior is unchanged until the var is switched, and the comparison below says not to switch yet.

What it does

• Index (shared, built once per transcript version): caption passages (about 100 to 160 tokens) embedded with Qwen3 Embedding 8B, plus DeepSeek V4.1 Flash chapters with embedded titles and summaries. They are stored as transcript_index and transcript_chapters in VIDEO_CATALOG/VIDEO_ASSETS, pinned to the transcript content_hash, with float16 vectors. Payloads hold no transcript text.
• Background build: TranscriptIndexWorkflow uses a deterministic instance ID, so a duplicate start is a no-op and a failed instance is restarted. Step 1 is saved before step 2. It is started from the coordinator after a word-level transcript is published, and by any question that finds no index. Answering never waits for it.
• Question time (inspect_video): the classifier gains an optional transcriptScope field and sees chapter titles when an index is ready.
  • topic: passage search with a 2,000-token budget.
  • one_part: the same, plus the best-matching chapter for up to half the budget.
  • whole_video: parallel per-chapter point extraction.
  • Index not ready, or any failure: today's full-transcript path.
• Evidence: excerpts are always caption text with the complete transcript's IDs, so citations are unchanged. The finalizer budget drops to 60k characters for indexed evidence.
• Deletion boundary: the index is never attached to sessions or users. Session deletion, account deletion and refresh leave it intact (existing deletion tests extended).
• Optional Jev router: AGENT_TRANSCRIPT_ROUTER=jev plus a JEV_API_KEY secret.
• Docs: platform-internals.md, VIDEO_CATALOG.md, SESSION_EVIDENCE.md.

Deviation from the PRD

Readiness is decided at the first get_video_transcript call in a run, not at the start of research, because the transcript's catalog version is only known then. It still holds for the rest of the run. Whole-video extraction is not priced as a separate data operation.

Evaluation

Offline (test/transcript-retrieval.eval.test.ts, real Fireworks and Jev calls, research-repo transcripts):

• Keynote targeted questions: plain search located 19/24 facts, matching the research result exactly. With the chapter add-on, 23/24.
• Whole-video routing: the GLM classifier with chapter titles got 18/18 decisions right, and so did Jev. This closes the PRD's open router risk.
• Extraction: React creator-chapter recall 1.00; keynote announcement locations near points 21/22.
• Index build: step 1 took 2.3 to 4.5 s (gate ≤7 s). Step 2 took 12 to 17 s.

Agent path: a local Worker answered the same 18 questions on each path. The automatic metric is fact locations cited; manual blind grading is still to do (scripts/eval-transcript-retrieval.ts sheet).

	full	index	index, with chapter titles for first questions
Keynote targeted (24)	23	19	20
Keynote "everything announced" (22)	22*	18	19
React targeted (12)	12*	7	10
Typical run time	~27 s	~18 s	~18 s

* Today's path failed on the first attempt for both of these long-context questions and succeeded on retry. The index path had no failures.

Conclusion: the index path is faster and more reliable, but today's path still cites more fact locations, so keep full (PRD Phase 0 gate). The largest remaining misses are Q1 (the chapter choice) and the agent skipping the transcript on one React question.

Tests

• Unit: 2,035 passed, including 54 new tests.
• Integration: catalog (14) and Durable Object/session (356) suites pass.
• docs:check passes; OpenAPI regenerated for the new route field.
• Live test (AGENT_INDEX_LIVE=1) passes.

Before enabling

This needs a deploy that creates the transcript-index-workflow Workflow, then AGENT_TRANSCRIPT_RETRIEVAL=shadow first.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
video2ctx-web	Ready	Preview	Oct 20, 2026 2:45am UTC

[devhims]

Independent review — adbf9b0

Verdict: CHANGES NEEDED. Reviewed the current description, merge-base diff against target main 44933d2, surrounding code and checks. Default full gating passed; the findings below affect the new enabled shadow/index paths. Do not enable them yet. Green CI and author-reported evaluations do not cover these reproduced edges.

P2 — Optional query embedding can prevent full-transcript fallback

platform/src/agents/runtime/transcript-retrieval.ts:160–166, with lib/video-index/embeddings.ts:7–8,43–57, uses three 30-second attempts under a 40-second nonvisual research deadline. A stalled embedding retries after 30 seconds; the research signal then aborts before fallback can return the already-fetched transcript. Production retrieval/tool/deadline probes reproduce this in both shadow and index modes: transcript fetch completes immediately, two embedding requests start, and no transcript packet is returned before Research phase timeout. This does not prove every eventual answer fails, but discards usable context. Give optional retrieval a short independent deadline; shadow observation must not block full evidence.

P2 — The optional chapter preload's first lookup is unbounded

runtime/transcript-retrieval.ts:88–99 awaits currentReference() before starting its 1.5-second race. The caller at research/research-agent.ts:171–184 has already started the 20-second classification budget, but this await is outside its deadline wrapper. A stalled catalog lookup remains pending after 21 seconds in a production-function fake-timer probe. Bound the entire optional preload, including version resolution, and honor cancellation so an optional hint cannot consume classification availability.

P2 — Chapter-density changes reuse the wrong completed workflow

lib/video-index/build.ts:23–25,38–40,55 omits chapterMinutes from the workflow identity, while storage.ts:82–84 includes it in the chapter artifact key. Reproduction: finish a 3.5-minute-density build, then request density 2. The artifact keys differ, but enqueue returns the old completed instance, whose payload remains 3.5; no new chapter artifact is built. Existing transcripts can permanently lose chapter enrichment/fall back until separately recovered. Include the requested derived variant in identity or detect and rebuild missing required artifacts; restarting an old failed instance also retains its old payload.

P2 — Whole-video budget reduction removes almost all citation support

runtime/model-evidence.ts:547–552 and research/research-agent.ts:941–947: whole-video chapter output can exceed the new 60,000-character budget. Generic packet reduction then retains one caption while preserving all chapter point text; the new reference filter strips the other point IDs.

Actual-function fixture: a supported 80.5-minute video, 23 chapters × 4 points and 85-character captions yields 276 caption excerpts before budgeting, one afterward, with 91 of 92 points uncitable. A 119-minute fixture similarly loses support for 135/136 points. prompt_evidence_cut is disclosed, but the remaining input cannot support the requested chapter-grounded answer. Budget points and supporting captions together, preserve alignment, and disclose which coverage was omitted instead of retaining unsupported summaries.

P2 — Invalid nonempty chapter citations silently disappear

providers/youtube/chapter-extractor.ts:79–82 filters out-of-range caption IDs but treats the resulting empty points as successful. A two-chapter production retrieval/tool fixture has valid first-chapter points and a nonempty second response citing caption 99 outside its range. It remains in indexed mode with only first-chapter evidence, failedChapters:[] and warnings:[]. chapterPointCoverage also omits the now-empty chapter. Distinguish intentional empty output from a nonempty output rejected by validation; mark/repair the latter and disclose missing coverage. All-invalid output already falls back to full, but mixed-validity output does not.

P2 — Schema-invalid completed calls bypass usage accounting

providers/youtube/chapter-extractor.ts:77–87 records usage only after generateText returns. With the installed AI SDK, a schema-invalid completed response throws AI_NoObjectGeneratedError carrying usage before return. The catch discards that usage. Probe: valid output records 100 input/20 output tokens; invalid caption type with the same provider usage records nothing and marks the chapter failed. This undercounts the model-cost ledger/guard. Record available error usage once, avoiding double counting errors after successful recording. Existing transcript-analyst.ts:226–230 handles this SDK case.

Independent verification

• Full offline platform unit suite: 2,035 passed, 1 skipped; live/evaluation provider suites excluded.
• Catalog integration: 14 passed. Session-catalog integration, including changed deletion cases: 86 passed.
• 2,135 passed, 1 skipped across those distinct suites. The focused 59-test run overlaps the unit suite and is not added again.
• Four independent retrieval/deadline/workflow reproduction tests passed; additional production projection/SDK probes and one five-case fallback test passed. These assert edge behavior and are separate from suite totals.
• TypeScript and generated API docs checks passed; tracked checkout clean.
• Verify CI 39293503722 and Vercel passed for the exact SHA.

Boundaries verified and limits

Default full mode avoids new index reads/builds; no default-full blocker was found. Shared derived indexes remain outside session deletion ownership; changed deletion tests passed. Ordinary failed chapter calls preserve other chapters with PARTIAL_EVIDENCE, and all-failed/all-empty extraction falls back to full. Citation aliases work when packets fit.

No live Fireworks/Jev calls, paid evaluations, production outages/workflow executions, deployment or manual blind grading were performed. Fixtures use production functions with synthetic provider/catalog/workflow responses; the usage probe exercises the installed SDK error behavior. Author-reported speed/quality results were not independently reproduced and do not establish broad reliability. No code edits, push, merge or deployment by this review.

[devhims]

Independent re-review — 85c425f

Verdict: CHANGES NEEDED. The deadline, workflow-identity, invalid-citation and error-usage fixes pass independent regression probes. The chapter-budget fix works for the original short-caption cases but still has a minimum-fit failure; a separate large-partial-result edge also needs correction. Default full mode remains unchanged.

P2 — Minimum-fit overflow still leaves chapter points without citations

platform/src/agents/runtime/model-evidence.ts:339–364: when even one point plus its starting caption per chapter cannot fit, fitChapterPoints returns undefined and the packet falls through to the generic one-excerpt reducer. All chapter point text survives, then the finalizer strips the missing citation IDs.

Production-function reproductions at the actual 60,000-character budget:

• Supported 119-minute/34-chapter input, one 1,800-character caption and 100-character point per chapter: 34 points remain, 33 have no reference, with no packet truncation warning.
• Supported dense setting, 120 chapters with 300-character captions and 120-character points: 119/120 points become uncitable.

The earlier 23/34-chapter short-caption fixtures now pass. Do not fall through to generic reduction for a chapter packet when its minimum aligned form cannot fit. Select a bounded set of point-plus-caption groups and explicitly disclose omitted chapters/points, or use an appropriate fallback; preserve citation alignment for everything retained.

P2 — More than 50 failed chapters reject usable partial evidence

providers/youtube/tools/get-video-transcript.ts:287–289 emits one warning per failed chapter; evidencePacketSchema caps warnings at 50. The actual unmodified extractor → retrieval → executeGetVideoTranscript path with a valid 119-minute/68-chapter plan produces one successful point and 67 failed chapters, then throws ZodError at :126 because warnings exceed 50. Sixty-eight chapters are allowed by the default builder's 2× target cap. Successful evidence is lost instead of returned as partial.

Aggregate the warning count/ranges into a bounded representation while preserving detailed failed-chapter metadata. This issue was present in the first head and is newly tested here; it is not caused by the deadline fix.

P3 — Shadow cancellation leaves an unobserved rejected promise

runtime/transcript-retrieval.ts:55–56: bounded(work, signal) returns a rejection for an already-aborted signal without attaching a handler to the already-created work promise. Immediate shadow return followed by normal phase cleanup reproduces an unhandled “Agent run finished.” rejection in Vitest and standalone Node using production functions. Full evidence still returned successfully; no Cloudflare response failure is claimed. Observe/settle the work promise even on early cancellation, or avoid starting it when already aborted.

Fixes verified

• Index fallback returns the complete transcript after the independent 8-second budget; shadow returns full evidence immediately.
• Full preload, including initial version lookup, is bounded to 1.5 seconds.
• Different chapter densities create distinct workflow instances.
• Invalid nonempty citation output retries once then becomes failed/PARTIAL_EVIDENCE; intentional empty output stays valid.
• Schema-invalid completed model calls record reported usage.
• Original long-video projection fixtures retain aligned points/captions with reduction disclosure.

Fresh validation for this head

• Offline units: 2,047 passed, 1 skipped.
• Catalog integration: 14 passed; session-catalog integration: 86 passed.
• 2,147 passed, 1 skipped across those suites; TypeScript and generated docs checks passed.
• Four regression assertions pass, but their separate Vitest process exits 1 due to the reproduced unhandled rejection above. This is not reported as a clean passing probe run.
• Additional production projection/SDK probes and the actual 68-chapter tool-path probe reproduced the two remaining P2s. No live provider calls.
• Verify CI 39296400101 passed.

No source edits, push, merge, deployment or paid/live evaluation. Kiro was asked to review this exact head in a fresh session; no substantive verdict has returned yet, so this comment does not claim its agreement. The synthetic fixture and local test limits from the prior review still apply.

[devhims]

Re-review — 5c724bc

Verdict: APPROVE with the existing default full mode retained. No remaining blocking findings identified. This is code-review approval, not approval to enable indexed retrieval or a claim of live quality parity.

Prior findings independently verified fixed

• platform/src/agents/runtime/model-evidence.ts:339–344,378–415: minimum-fit overflow retains aligned chapter points and captions with explicit omission disclosure. The prior 34/120-chapter fixtures retain 27/85 supported points within 60k, with zero uncitable retained points.
• providers/youtube/tools/get-video-transcript.ts:287–304: the actual extractor → retrieval → tool fixture with 67 failed chapters preserves the successful chapter and emits one bounded warning instead of failing packet validation.
• runtime/transcript-retrieval.ts:55–62: standalone shadow lifecycle returns full evidence with zero unhandled rejections. Earlier deadline/workflow regression probes also exit cleanly.

P3, nonblocking — spread-selection size is not monotonic

runtime/model-evidence.ts:399–415 binary-searches the number of evenly spread chapters, but changing that number changes which caption lengths are selected. Independent production-function fixture: 12 alternating 1,200/100-character captions, 4,500 characters available; three selected chapters cost 5,201, while four cost 4,401. The search retains only two. This unnecessarily reduces indexed-answer coverage under tight budgets, although citation alignment and omission disclosure remain correct. Consider checking the bounded candidate counts without assuming monotonicity. Choosing cheapest chapters instead would be a separate coverage-policy decision, not required by this finding.

Review comparison

A separate Kiro review of this exact head agrees all three prior findings are fixed and identifies no P1/P2. Its independent same-selector counterexample also confirms the P3 above. The chapters_uncited metric uses persisted source coverage (research-agent.ts:592,873–882), so budget-omitted chapters legitimately count under its documented definition; it cannot alone distinguish budget omissions from model omissions. Treat that as an observability limitation, not another correctness defect.

Fresh verification

• Offline platform units: 2,051 passed, 1 skipped.
• Catalog integration: 14 passed; session-catalog integration: 86 passed.
• Total 2,151 passed, 1 skipped, plus 10 independent regression/boundary tests and standalone lifecycle checks.
• TypeScript, generated docs and diff checks passed; tracked review checkout clean. The npm docs wrapper hit local IPC EPERM; equivalent node --import tsx invocation passed.
• Verify CI 39357733973 and Vercel passed for the exact SHA.

Kiro's package installation was blocked by registry access; its corroboration used standalone extracted-function probes, not a passing full suite. My independent validation used the installed project dependencies and production functions. No paid/live provider evaluation, deployment, production outage simulation, or manual blind quality grading was performed. Keep the PR's separate quality/rollout gate before enabling shadow/index. No implementation, push, merge or deployment performed by this review.

[devhims]

Re-review — 9022b64

Verdict: APPROVE, retaining default full mode. The remaining nonblocking selector finding is fixed; no new findings in this delta.

platform/src/agents/runtime/model-evidence.ts:407–413 now checks descending candidate counts rather than assuming monotonic size. It preserves the existing evenly spread selection policy. The independent counterexample now retains four chapters instead of two within 4,500 characters. An exhaustive oracle over 286 heterogeneous budget cases confirms selection of the largest fitting existing spread, with supported citations and omission disclosure preserved.

Fresh independent validation: 2,052 units + 14 catalog + 86 session-catalog tests = 2,152 passed, 1 skipped. Eleven additional regression/boundary tests passed, including the oracle. Standalone shadow cleanup reports zero unhandled rejections. TypeScript, generated docs and diff checks passed; checkout clean. Verify CI 39360587515 and Vercel passed for this SHA.

Kiro independently agrees on this exact head: its 40-chapter counterexample now retains the largest fitting 32 chapters instead of 29, and 400 randomized extracted-function trials passed. Its local full suite remained unavailable because of registry access; those probes are corroboration, not an additional full-suite pass. Descending enumeration does more serialization than binary search but is bounded by candidate count; no production Worker latency benchmark was performed.

This supersedes the prior P3 selector follow-up. The documented source-coverage metric limitation is unchanged. No paid/live provider evaluation or rollout-quality revalidation was performed; approval does not authorize enabling indexed retrieval. No implementation, push, merge or deployment performed by this review.