Add video input support for vision-language models - #140
Open
stikves wants to merge 7 commits into
Open
Conversation
stikves
force-pushed
the
sukru/video-input
branch
2 times, most recently
from
August 3, 2026 21:18
fbb72c2 to
6b16086
Compare
stikves
commented
Aug 3, 2026
stikves
requested review from
carinapeng,
kevchengcodes and
tjia1818
and removed request for
carinapeng and
tjia1818
August 3, 2026 21:53
stikves
marked this pull request as ready for review
August 3, 2026 21:53
stikves
force-pushed
the
sukru/video-input
branch
from
August 4, 2026 18:16
9c5db2b to
a373446
Compare
Frame extraction pipeline using AVAssetImageGenerator with two sampling strategies (uniform and FPS-based). Frames are delivered lazily via a concrete VideoFrameSequence (AsyncSequence + Sendable) so peak memory stays at 1-2 decoded frames. VLM engine encodes each frame independently through the existing single-image vision encoder, concatenates embeddings along the sequence dimension, and scatter-merges into the token stream. Zero changes to exported models required. New types: FrameSamplingStrategy, VideoInput, VideoFrameSequence, VideoFrame, VideoFrameExtractor. VisionConfig extended with optional max_video_frames and tokens_per_frame fields (backwards compatible). CLI: --video, --video-frames, --video-sampling flags on llm-runner. Tested end-to-end with Qwen3-VL-2B on a screen recording (4 frames, 784 visual tokens, correct output).
M4: VLM export --num-frames flag for native temporal video encoding. StaticVisionEncoder now accepts num_frames parameter (default 1 for backwards compatibility). Multi-frame input: [1, 3*N, H, W] is reshaped into real temporal patches instead of duplicating a single frame. Position and rotary embeddings are computed with grid_t > 1. metadata.json gains max_video_frames, tokens_per_frame, and temporal_patch_size fields when exporting with --num-frames > 1. M5: Vectorize the RGBA-to-planar CHW transpose in ImagePreprocessor using vDSP_vsadd with stride-4 gather. Replaces the O(3*H*W) scalar loop with three vectorized passes.
- Remove unused num_visual_tokens_for_frames property from VLMSpec - Handle BFloat16 in video embedding concatenation (switch on scalarType) - Guard pixelCount > 0 in preprocessCHW to avoid force-unwrap on empty buffer - Respect maxFrames: 0 in FPS sampling (return empty instead of 1 frame) - Keep underlying Error in VideoInputError.frameExtractionFailed - Replace videoSampling String with VideoSamplingMode enum - Extract shared runVLMGeneration helper from image/video inference paths
stikves
force-pushed
the
sukru/video-input
branch
from
August 6, 2026 05:34
6a23617 to
065afaf
Compare
… video config - Rename FrameSamplingStrategy.timestamps() to sampleTimes() for clarity - Add withFrameCount() to FrameSamplingStrategy, make it ExpressibleByArgument - Remove VideoSamplingMode wrapper enum (de-dupe with FrameSamplingStrategy) - Add VisionConfig.supportsVideo computed property to clarify maxVideoFrames semantics
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
AVAssetImageGeneratorwith uniform and FPS-based sampling strategiesVideoInputdelivers frames lazily via a concreteVideoFrameSequence(AsyncSequence+Sendable) to keep peak memory at 1–2 decoded frames--video,--video-frames,--video-samplingflags onllm-runnerVisionConfigextended with optionalmax_video_framesandtokens_per_framefields (backwards compatible)--num-framesfor native multi-frame temporal encoding (grid_t > 1)ImagePreprocessorusingvDSP_vsaddstride-4 gatherNew types
FrameSamplingStrategy.uniform(count:)/.fps(rate:, maxFrames:)VideoInputVideoFrameSequenceVideoFrameSequenceAsyncSequence & SendableoverVideoFrameVideoFrameCGImage+ timestampUsage
swift run -c release llm-runner \ --model exports/qwen3_vl_2b.llmasset \ --video path/to/video.mp4 \ --video-frames 8 \ --prompt "Describe what happens in this video"Test plan
FrameSamplingStrategy: uniform/fps math, edge cases (single frame, clamp, zero duration, negative duration)VideoInput.fromFrames: wraps[CGImage], empty arrayVisionConfig: decodes with/without video fields (backwards compat)ImagePreprocessor: existing tests pass with vectorized transpose