feat: send prompt-attached videos directly with the prompt#1999
Conversation
Videos attached to a prompt (pasted in the TUI, uploaded in the web UI)
previously reached the model only after it opened the file with
ReadMediaFile — an extra tool round trip that could leave the video
unseen when the model never made that call. They are now uploaded
through the model provider's file channel and embedded directly in the
user message as the provider-issued reference; ReadMediaFile stays as
the fallback and as the model's own way to open video files.
- agent-core: new uploadVideo agent RPC and Session.uploadVideo in the
SDK; the TUI uploads pasted videos at submit time, falling back to
the file-tag form on failure
- kap-server: inline file-source video prompt parts at the REST edge
and map provider file ids back to local uploads behind
GET /files/llm/{llm_id}
- kimi-web: play provider-referenced prompt videos after reload/resume
commit: |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 128faf145f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
ReadMediaFile only used the provider's video upload channel when one was bound, and surfaced a hard tool error when the upload itself failed — on providers without a files endpoint (or a transient upload failure) the model was told the video could not be read at all. Now a missing or failing upload falls back to delivering the video inline (base64), the same shape providers without an upload channel already get. Both engines (agent-core and agent-core-v2) are fixed.
Stress coverage for the prompt video pipeline: - agent-core uploadVideo RPC: extension/magic classification (.txt with video bytes accepted, extension trusted when magic is absent), exact 100MB boundary, directory and nonexistent paths - TUI: mixed per-video outcomes (one inlined, one tag fallback), submission order behind an in-flight upload, queued video messages carrying final uploaded parts - kap-server: per-video provider-id mappings, Range requests through the llm redirect, mapping persistence across a server restart
🦋 Changeset detectedLatest commit: 57af423 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ead47c6bc6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
# Conflicts: # apps/kimi-code/src/tui/kimi-tui.ts # packages/kap-server/src/routes/prompts.ts # packages/kap-server/test/prompts.test.ts
The base64 fallback for a failing video upload must not mask auth rejections: a 401/403 (surfaced as provider.auth_error) drives the credential force-refresh and a clear auth error, while an inline payload would just be rejected again by the next request. Only a missing or broken upload channel (no files endpoint, network/server errors) falls back to inline delivery.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 64d95efe00
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Main's contract for a provider with no video upload hook is an honest "does not support video upload" tool error — an inline payload would be dropped on that protocol's wire anyway. The base64 fallback now only applies to an upload channel that exists but failed at runtime. The no-hook throw gets a stable type (VideoUploadUnsupportedError) so the two cases are told apart without matching message text.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3a4e720c60
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The provider file id is used as a blob-store key, and the node-fs backend joins scope and key into a storage path. An id containing an encoded path separator (%2F) could address a different storage path than the intended llm-video mapping; the route now only accepts the provider-id alphabet.
…t blocks The package's comment convention keeps comments solely in the top-of-file block; relocate the new notes (url-source id pairing, video delivery fallback, VideoUploadUnsupportedError) from functions and schema fields into their module headers.
The uploadVideo RPC only checked the provider's upload channel, so an SDK caller on a text-only or unknown-capability model could obtain a valid-looking video_url part the current model is not supposed to accept. The TUI capability guard does not cover this public path, so the agent now gates on video_in itself.
The inline branch trusted the upload-time content type; now the bytes are sniffed first, and anything magic-confirmed as a non-video kind (e.g. an image mislabeled as video/mp4) falls back to the file-tag form instead of being uploaded. Like the image gate, bytes are authoritative where the container format allows it — an MPEG-PS lookalike still rides the extension, matching ReadMediaFile.
An immediately-answered 404 ends the turn (non-retryable) before the test's abort call, racing the cleanup into a 409; the stub now hangs generation until the abort cancels it.
A stale file_id failed inside media resolution — after the model, thinking and permission overrides had already been applied — so a rejected prompt still changed the session's controls. File references are now checked up front, keeping failed submits side-effect free.
A slow provider video upload let a later text-only request reach the queue ahead of an earlier video one, silently reordering the conversation for REST clients and multiple tabs. Submissions to the same session now chain, matching the ordering the TUI already guarantees locally.
A video recovered from an ms:// reference carried only the bare redirect URL, which 401s under daemon auth when loaded natively. The attachment now keeps the provider file id (llmFileId) end to end, and AuthMedia fetches the bytes with the Bearer credential through the daemon's llm redirect — the same blob-URL path uploaded files already use.
A slow paste-upload left the TUI idle, so /new, the session picker or /model could fire mid-upload; the continuation then dispatched the old session's provider reference into the newly selected session or a model that cannot resolve it. The dispatch now re-checks that the session and model are unchanged and asks for a resend instead.
Trimming the path changed the filesystem target before validation and upload, so a name that legitimately starts or ends with whitespace resolved to the wrong file; the trim is now only the emptiness check.
The shared url-source schema accepts id for image and video parts, but only the video path forwarded it, dropping provider-keyed image ids between prompt acceptance and the model request.
…sage The loose user-message matcher counted media parts and <video path> tags but not the [video:ms://…] text shape, so a racing server echo of an inlined upload slipped through as a duplicate user bubble.
A bash-mode submit could start while a pasted video was still uploading, recording shell context before the earlier prompt was dispatched and reordering user actions; it now chains behind the upload like normal submits.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9d40cd551d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
A stale file_id posted to a fresh or cold session materialized the main agent (registering it in session metadata and igniting agent-scoped services) before the request was rejected. The file check now runs first, so failed submits create nothing and mutate nothing.
Projections and clients read the provider id from the ms:// URL, not the id field, so an upload that returns an id-less or mismatched reference would inline fine but 404 on playback. The mapping now derives its key from the URL, falling back to the explicit id.
The read route got the safe-alphabet guard, but recordLlmVideoRef still used the provider-returned id verbatim as a blob-store key; a crafted provider response could write the mapping outside the llm-video namespace. Out-of-alphabet ids are now dropped on both put and get.
The preflight only proved a referenced file exists; a real upload used with the wrong kind (e.g. a PDF submitted as video) still passed it and mutated session controls before assertMediaFile rejected the request. The kind assertion now runs up front with the existence check.
The media preview returned early for every kind except image, so a user-turn video chip's play action was a no-op even with llmFileId threaded through. The preview now handles video: bytes come from the authenticated file/llm fetch into a blob URL and render in a native player.
The llm re-upload branch fetched bytes with auth but kept the protected redirect URL as the chip preview, which 401s as a native video src; the fetched blob now becomes the preview URL, mirroring the fileId branch.
…eader Same package comment convention as before: rationale lives in the top-of-file block, not beside the registration call.
The chained bash callback ran runShellCommandFromInput against whatever session was active at dispatch time, so a command submitted in session A could execute in session B's workspace and be recorded there after a mid-upload switch. The originating session is now captured at submit and re-verified at dispatch, like the prompt and skill paths.
video_upload is an agent-level event requiring ambient agent identity, but the uploader was built with the Core-scoped session view, leaving prompt-upload events unattributable. The route now resolves telemetry from the target agent for the uploader while image compression keeps the session-scoped view.
…appers The url-source id accepted by the prompt schema was dropped again by the legacy message projection and the web wire mapper, so provider- keyed image references lost their id across messages, snapshots, and undo responses. It now flows through both directions.
A failed llm re-upload removed the attachment without revoking the freshly created preview blob URL, pinning the whole video in the browser blob store until page unload on every failed edit attempt.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a08db8ab60
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
/compact and /init started their turn while a pasted video was still uploading, so the earlier message landed after them — and compaction summarized the context without it. Prompt-producing builtin commands now share the same queueBehindPendingUploads chain (with the session/model dispatch fence) as skill, plugin, and bash submits.
Media resolution now runs before any profile/model/thinking/permission/ denylist mutation, with the uploader resolved transiently from the requested (or currently bound) model — a failed submission leaves the session's controls untouched. A concurrent model switch during preparation is rejected with session.busy instead of enqueueing a reference uploaded for the previous model.
Session.uploadVideo is new public API surface, not a patch-level tweak.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d6f64ef96d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
A slash command deferred behind a video upload ran the moment the video prompt dispatched, landing on an already-running turn: beginSessionRequest wiped the active turn's live pane, and /init or /compact started on top of it. The deferred callbacks for skills, plugin commands, /compact and /init now re-run the resolver's busy check at dispatch and show the same blocked message the user would get when typing while streaming.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0998978700
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
A first prompt carrying "profile" without "model" resolved the upload model from the still-unbound alias, so no uploader was installed and every attached video fell back to a tool-read tag even when the configured default model supports provider upload. The transient resolution now mirrors AgentProfileService.bind: an explicit body model wins, a profile bind falls back to the configured default model, and only otherwise does the currently bound alias apply.
dfb8307 to
78f5ca9
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 78f5ca9816
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Move prompt-video delivery out of the submission edge: the TUI submits synchronously with a local file:// part the v1 turn resolves before the message enters history, and kap-server carries an internal kimi-file:// reference the v2 requester resolves against the effective model with an app-scoped upload cache. History keeps the durable local file id, the /messages projection emits structured video parts, and the web plays videos back through the authenticated /files channel - deleting the submit fences, per-session serialization, provider-id reverse mapping, redirect endpoint, and the unreleased SDK upload API.
There was a problem hiding this comment.
💡 Codex Review
Key video upload cache by requester context
When a session reuses the same uploaded prompt video after switching to another configured provider entry/account, protocol, or model that shares this providerType, this key hits the old in-memory/blob cache and readCachedUpload returns the previous ms:// id before the resolver rechecks capabilities, protocol, endpoint, or auth. Provider file ids are scoped to the requester context, so the next request can either send video to a model that should have fallen back to a tag/inline form or reuse an id from a different account/base URL and fail every subsequent turn. Include the requester context (at least provider name/protocol/base URL/auth identity and relevant capability) in the cache key, or run those checks before honoring cached ids.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
A turn cancelled mid-upload used to be treated as an ordinary upload failure: v1 fell back to an inline base64 part and appended the degraded message to history, and the v2 resolver memoized the tag fallback for the rest of the agent's lifetime. Both catch sites now check the delivery signal itself - abort rejections vary in shape by provider - and re-throw so cancellation ends the turn (v1, classified as cancelled via the abort reason) or the request (v2, not memoized, so the next turn uploads).
…e video An OpenAI-family model configured with video_in but no provider upload channel used to receive prompt videos as an inline base64 part - which chat completions rejects and the Responses adapter degrades to an omitted-video placeholder, persisting ~4/3x the file size in history for bytes the model never sees. The prompt path now mirrors the v2 resolver's protocol gate and degrades to the <video path> tag instead; ReadMediaFile's own delivery is unchanged. Also merges the prompt-video changesets into a single user-facing entry.
The cache-key template literal contained a literal NUL byte instead of the \0 escape, which made Git classify the whole source file as binary - no inline diffs, unreliable text tooling. The escape produces the byte-identical runtime string, so hashed cache keys are unchanged.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 353210af30
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The no-uploader inline path (and the post-upload-failure fall-through) never consulted the delivery signal, so cancelling a turn while the video bytes were being read still base64-encoded the file and appended the degraded message to history. The inline branch now re-throws the abort reason first, matching the upload catch.
A generic upload failure used to memoize its tag fallback for the rest of the agent's lifetime, freezing a transient files-endpoint error into a permanently degraded video. The resolver now marks failure-born fallbacks as non-memoizable: the current request keeps the lightweight tag form and the next step retries the upload. Structural outcomes (successful uploads, capability and sniff fallbacks, no-hook inline) stay memoized for step-retry stability.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 57af423240
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| else if (part.type === 'image' && part.source.kind === 'url') parts.push({ type: 'image_url', imageUrl: { url: part.source.url, id: part.source.id } }); | ||
| else if (part.type === 'image' && part.source.kind === 'base64') parts.push({ type: 'image_url', imageUrl: { url: `data:${part.source.media_type};base64,${part.source.data}` } }); | ||
| else if (part.type === 'video' && part.source.kind === 'url') parts.push({ type: 'video_url', videoUrl: { url: part.source.url } }); | ||
| else if (part.type === 'video' && part.source.kind === 'url') parts.push({ type: 'video_url', videoUrl: { url: part.source.url, id: part.source.id } }); |
There was a problem hiding this comment.
When a REST client sends a video source.kind === 'url' whose URL starts with the new internal kimi-file:// scheme, this path forwards it unchanged into context; later AgentVideoResolverService treats every such URL as a trusted daemon upload reference, while assertPromptFileRefs only validates { kind: 'file' } sources. That means a stale or forged internal URL can bypass the new file-reference validation and be accepted/projected as a file-backed video (or resolved to the caller-provided fallback path) instead of being rejected or treated as an ordinary external URL. Reject or escape the internal scheme on client URL inputs before building the core part.
Useful? React with 👍 / 👎.
| const cachedLlmFileId = await this.readCachedUpload(cacheKey); | ||
| if (cachedLlmFileId !== undefined) { | ||
| return { | ||
| part: { type: 'video_url', videoUrl: { url: `ms://${cachedLlmFileId}`, id: cachedLlmFileId } }, | ||
| memoize: true, |
There was a problem hiding this comment.
For upload providers that return a non-ms:// videoUrl.url plus a provider id, the first request uses the correct uploaded part, but any later step/retry hits this cache branch and fabricates ms://<id>. That Kimi-specific URL can be sent to a different protocol that only accepts the original URL/id pair, so cached prompt videos stop resolving after the first request; cache and replay the full VideoURLPart (or at least the original URL with its id) instead of reconstructing an ms:// URL from the id.
Useful? React with 👍 / 👎.
1 participant
Related Issue
No linked issue — the problem is explained in the next section.
Problem
Videos attached to a prompt (pasted into the TUI, uploaded in the web UI) only reached the model indirectly: the client sent a
<video path="…">text reference and the model had to spend aReadMediaFiletool call to actually see the video. That costs an extra tool round trip every time, and if the model never makes the call, the video goes unseen entirely.A second, related gap: after reloading or resuming a session in the web UI, a previously uploaded video could no longer be played back, because the prompt carried only a server-local file path reference.
What changed
Prompt-attached videos are now uploaded through the model provider's file channel and embedded directly in the user message as the provider-issued
video_urlreference — the model receives the video with the prompt, no tool call required. The old path stays as an automatic fallback (no upload channel on the current model, or an upload failure), andReadMediaFileremains the model's own way to open video files.agent-core: newuploadVideoagent RPC that reads a local video (format sniff, 100 MB cap) and uploads it through the bound provider; exposed in the SDK asSession.uploadVideo(path). The TUI uploads pasted videos at submit time and inlines the returned reference; on failure it degrades to the previous file-tag form. Submits without videos keep their fully synchronous path, and submits chain behind an in-flight upload so a slow upload cannot be overtaken by a later message.kap-server: file-source video prompt parts are inlined at the REST edge using the session's effective model (prompt-time model overrides now apply before media resolution). A newGET /api/v1/files/llm/{llm_id}endpoint redirects a provider file id back to the local upload it came from (persisted one-key-per-mapping), so clients can play inlined videos back from daemon storage.kimi-web: user-turn videos carried as provider references (in either the structured or the flattened text projection form) render as playable attachments again after reload/resume; the previous path-tag form keeps working.Tests: new coverage at each layer (agent upload RPC + error branches, SDK passthrough, TUI segment pipeline and submit flow incl. fallback, kap-server inline + redirect + 404, web rendering), plus real-gateway smoke verification of both engine paths (upload → inline → model describes the video; redirect serves the original bytes).
Checklist
gen-changesetsskill, or this PR needs no changeset.gen-docsskill, or this PR needs no doc update.