feat(benchmarks): add TeleFuser AIPerf video benchmark support#5
Open
ActivePeter wants to merge 1 commit into
Open
feat(benchmarks): add TeleFuser AIPerf video benchmark support#5ActivePeter wants to merge 1 commit into
ActivePeter wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces an initial AIPerf-based benchmarking workflow for TeleFuser video serving, including batch /v1/videos configs/contracts and a WebRTC stream benchmark harness, plus a Diffusers Wan2.1 I2V baseline service for apples-to-apples comparisons.
Changes:
- Added TeleFuser AIPerf benchmark assets (configs, scripts, datasets) and benchmark contract manifests for TeleFuser + Diffusers baselines.
- Added a standalone Diffusers Wan2.1-I2V-14B-480P FastAPI service baseline and a fixed-workload TeleFuser 480p service example.
- Added small runtime/service updates for benchmark stability (OpenAI adapter size preservation, pipeline runner media aliasing, WebRTC ICE host allowlisting).
Reviewed changes
Copilot reviewed 34 out of 34 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/service/test_pipeline_runner.py | Adds unit test coverage for media alias injection in _select_kwargs. |
| tests/unit/openai/test_adapter.py | Adds unit test for preserving exact OpenAI WxH size fields in task requests. |
| telefuser/webrtc_ice.py | Adds ICE host candidate allowlisting via aioice monkeypatching. |
| telefuser/service/main.py | Calls ICE host allowlist configuration on stream server startup. |
| telefuser/service/core/pipeline_runner.py | Extends kwarg selection and injects media aliases for **kwargs pipelines. |
| telefuser/service/api/openai/adapter.py | Preserves openai_video_size plus parsed width/height in task extras. |
| telefuser/pipelines/lingbot_world_fast/pipeline.py | Makes fast checkpoint paths robust to absolute/relative + ~ usage. |
| telefuser/models/init.py | Makes LTX cross-file registration optional (but currently too broad in exception handling). |
| pyproject.toml | Adds runtime deps (uvicorn[standard], python-multipart) needed for serving endpoints. |
| mkdocs.yml | Adds Benchmarks nav entries and translations. |
| examples/wan_video/wan21_14b_image_to_video_480p_service.py | Adds fixed-workload Wan2.1 I2V 480p TeleFuser service example. |
| examples/lingbot/stream_lingbot_world_fast.py | Makes model zoo paths user/abs-path robust via Path. |
| docs/en/index.md | Adds benchmark doc entry. |
| docs/en/benchmark_aiperf.md | Adds English benchmark documentation. |
| docs/zh/index.md | Adds benchmark doc entries (including design doc link). |
| docs/zh/benchmark_aiperf.md | Adds Chinese benchmark runbook documentation. |
| docs/zh/benchmark_aiperf_design.md | Adds Chinese benchmark design notes/spec. |
| benchmarks/telefuser_aiperf/scripts/run_video_bench.sh | Adds batch benchmark runner script with health check + ulimit tuning. |
| benchmarks/telefuser_aiperf/scripts/run_stream_bench.sh | Adds stream benchmark runner wrapper for the WebRTC harness. |
| benchmarks/telefuser_aiperf/scripts/run_stream_bench.py | Adds WebRTC session benchmark harness (offer RTT, first-frame, FPS, control latency). |
| benchmarks/telefuser_aiperf/README.md | Documents benchmark layout and usage. |
| benchmarks/telefuser_aiperf/data/video_prompts.jsonl | Adds prompts + reference images dataset (currently hard-coded to /workspace/TeleFuser/...). |
| benchmarks/telefuser_aiperf/configs/video_generation_*.yaml | Adds TeleFuser AIPerf batch configs (quick/e2e/rate/compare). |
| benchmarks/telefuser_aiperf/benchmark_contract.yaml | Adds TeleFuser benchmark contract manifest. |
| benchmarks/baseline/diffusers_wan_i2v/service.py | Adds Diffusers baseline FastAPI service implementing async /v1/videos. |
| benchmarks/baseline/diffusers_wan_i2v/scripts/run_video_bench.sh | Adds baseline benchmark runner script. |
| benchmarks/baseline/diffusers_wan_i2v/scripts/run_service.sh | Adds baseline service launcher (currently doesn’t export the env vars it intends to set). |
| benchmarks/baseline/diffusers_wan_i2v/README.md | Documents the baseline service and how to run it. |
| benchmarks/baseline/diffusers_wan_i2v/configs/*.yaml | Adds baseline AIPerf configs (E2E + compare). |
| benchmarks/baseline/diffusers_wan_i2v/benchmark_contract.yaml | Adds baseline benchmark contract manifest. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+54
to
+57
| def _load_rgb_image(path: Any) -> Any: | ||
| from PIL import Image | ||
|
|
||
| return Image.open(path).convert("RGB") |
Comment on lines
118
to
122
| if name in ("image", "input_image"): | ||
| first_path = task_data.get("first_image_path") or task_data.get("image_path") | ||
| first_path = _first_image_path(task_data) | ||
| if first_path: | ||
| from PIL import Image | ||
|
|
||
| kwargs[name] = Image.open(first_path).convert("RGB") | ||
| kwargs[name] = _load_rgb_image(first_path) | ||
| continue |
Comment on lines
+43
to
+45
| allowed_host_ips = _normalize_host_ips(host_ips) | ||
| if not allowed_host_ips: | ||
| return |
Comment on lines
+47
to
+50
| try: | ||
| from aioice import ice as aioice_ice | ||
| except Exception: | ||
| return |
Comment on lines
+14
to
+20
| try: | ||
| from telefuser.core.model_registry import register_model_config | ||
| from telefuser.models.ltx_dit import LTXVideoTransformer | ||
| from telefuser.models.ltx_gemma_text_encoder import LTXEmbeddingsProcessor | ||
| from telefuser.models.ltx_video_vae import LTXVideoVAE | ||
| except Exception: | ||
| pass |
Comment on lines
+7
to
+11
| PYTHON_BIN="${PYTHON_BIN:-python3}" | ||
| SERVICE_HOST="${DIFFUSERS_WAN_I2V_HOST:-127.0.0.1}" | ||
| SERVICE_PORT="${DIFFUSERS_WAN_I2V_PORT:-8010}" | ||
|
|
||
| exec "${PYTHON_BIN}" benchmarks/baseline/diffusers_wan_i2v/service.py |
Comment on lines
+49
to
+53
| serverMetrics: | ||
| enabled: ${TELEFUSER_AIPERF_SERVER_METRICS:true} | ||
| urls: | ||
| - ${TELEFUSER_AIPERF_METRICS_URL:http://127.0.0.1:8000/metrics} | ||
| formats: [json, csv] |
Comment on lines
+1
to
+5
| {"text": "The character turns slightly toward the camera and breathes naturally, cinematic motion, consistent identity", "image": "/workspace/TeleFuser/examples/data/101235-video-720_0.png"} | ||
| {"text": "The subject begins walking forward with subtle parallax in the background, natural lighting, smooth motion", "image": "/workspace/TeleFuser/examples/data/101235-video-720_0.png"} | ||
| {"text": "A gentle push-in camera move with soft environmental motion, realistic dynamics, no sudden cuts", "image": "/workspace/TeleFuser/examples/data/101235-video-720_0.png"} | ||
| {"text": "The scene gains light wind and small secondary motion while preserving composition and subject details", "image": "/workspace/TeleFuser/examples/data/101235-video-720_0.png"} | ||
| {"text": "A short cinematic continuation with stable structure, coherent motion, and clean temporal consistency", "image": "/workspace/TeleFuser/examples/data/101235-video-720_0.png"} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds the first TeleFuser benchmark workflow for AIPerf-based video serving evaluation. It includes TeleFuser benchmark configs, a Diffusers Wan2.1 I2V baseline service, benchmark contracts, documentation, and small service/runtime fixes
needed for reproducible batch and stream benchmark runs.
Changes
/v1/videosand stream WebRTC runs.openai_video_size,width,height) for fixed-workload benchmarks.Validation
pytestinstalled.中文说明
本 PR 增加了 TeleFuser 第一版基于 AIPerf 的视频服务 benchmark 工作流，包括 TeleFuser benchmark 配置、Diffusers Wan2.1 I2V baseline 服务、benchmark contract、相关文档，以及用于稳定批处理和流式 benchmark 的少量服务端运行时修正。
主要改动
/v1/videos和 WebRTC stream benchmark。size的精确宽高字段，支持固定 workload benchmark。extra覆盖 endpoint 管理的媒体字段。验证
pytest。