A Vision-Language Framework and Benchmark for Detecting Any Shot Transitions
Accepted to ECCV 2026 (Poster)
- TransVLM was accepted to ECCV 2026 as a Poster.
- Model weights, inference code and evaluation code are released. See Release Progress.
Traditional Shot Boundary Detection (SBD) inherently struggles with complex transitions by formulating the task around isolated cut points, frequently yielding corrupted video shots. We address this fundamental limitation by formalizing the Shot Transition Detection (STD) task. Rather than searching for ambiguous points, STD explicitly detects the continuous temporal segments of transitions. To tackle this, we propose TransVLM, a Vision-Language Model (VLM) framework for STD. Unlike regular VLMs that predominantly rely on spatial semantics and struggle with fine-grained inter-shot dynamics, our method explicitly injects optical flow as a critical motion prior at the input stage. Through a simple yet effective feature-fusion strategy, TransVLM directly processes concatenated color and motion representations, significantly enhancing its temporal awareness without incurring any additional visual token overhead on the language backbone. To overcome the severe class imbalance in public data, we design a scalable data engine to synthesize diverse transition videos for robust training, alongside a comprehensive benchmark for STD. Extensive experiments demonstrate that TransVLM achieves superior overall performance, outperforming traditional heuristic methods, specialized spatiotemporal networks, and top-tier VLMs.
Qualitative results and the comparison against baselines are on the project page.
- Model weights
- Inference code
- Evaluation code
- Data engine code
- STD benchmark data
- Re-annotated dataset labels
- Leaderboard
🚧 The remaining items are being prepared for release.
You need Python 3.12, a CUDA GPU, and ffmpeg on PATH. Install first: setup has a few sharp
edges (a cuDNN version floor, mutually exclusive backend venvs, and FFmpeg library discovery),
so follow inference/README.md rather than guessing:
cd inference
uv venv && source .venv/bin/activate
uv sync --group cu130 --group dev
uv pip install nvidia-cudnn-cu13==9.16.0.29 # required, and every uv sync undoes itThen fetch the checkpoint and run detection on one video:
hf download HeyGenAI/TransVLM-Qwen3-VL-4B-Instruct --local-dir ./pretrained/TransVLM-v1
python infer_video.py \
--video /path/to/video.mp4 \
--ckpt-dir ./pretrained/TransVLM-v1 \
--output-jsonl out.jsonlImportant
This checkpoint takes 6-channel RGB + optical-flow input, so a plain transformers
Qwen3-VL pipeline will not produce correct results. Use the code in inference/.
Every option, the output format and the three backends are documented here:
| Topic | Document |
|---|---|
| Inference | inference/README.md |
| Evaluation | evaluation/README.md |
| Path | Contents |
|---|---|
inference/ |
Inference pipeline and its two entry points, plus the vendored NeuFlow v2 optical-flow code |
evaluation/ |
Evaluation code for the STD benchmark |
docs/ |
Project page sources (figures, videos, interactive comparison widgets) |
assets/ |
Logos used by this README |
NOTICE |
Vendored third-party components and their upstream attribution |
@inproceedings{chen2026transvlm,
title={TransVLM: A Vision-Language Framework and Benchmark for Detecting Any Shot Transitions},
author={Chen, Ce and Ren, Yi and Li, Yuanming and Goriachko, Viktor and
Ye, Zhenhui and Guo, Zujin and Hong, Zhibin and Gong, Mingming},
booktitle={European Conference on Computer Vision},
year={2026},
organization={Springer}
}The STD benchmark is intended as a living resource. We are actively exploring smaller and more efficient VLM backbones for the STD task, and the resulting checkpoints, training recipes, and evaluation results will be open-sourced here as they become available.
We welcome community submissions of smaller-model baselines on the STD benchmark. If you have a more compact or more efficient method for STD, please open an issue or pull request with: (i) the model description and total parameter count; (ii) per-method evaluation results following the format on the project page; and (iii) reproduction instructions. Approved submissions will be added to the leaderboard on the project page.
TransVLM builds on Qwen3-VL-4B-Instruct and uses NeuFlow v2 for optical-flow estimation. We thank both teams for releasing their work.
Released under the Apache License 2.0. Vendored third-party components and their
upstream attribution are listed in NOTICE. All of them are Apache-2.0, which is why
this repository carries a single license. The released weights derive from Qwen3-VL-4B-Instruct
and are distributed under Apache-2.0 as well.
TransVLM was developed as academic research and is published at ECCV 2026. It is not used in any HeyGen product; HeyGen's production system for this task is a different model. The training data behind these weights, and the licences of the public datasets in it, are listed on the model card.