feat(benchmarks): add TeleFuser AIPerf video benchmark support

benchmarks/baseline/diffusers_wan_i2v/README.md

@@ -0,0 +1,74 @@
+# Diffusers Wan2.1 I2V Baseline
+
+This directory contains a standalone benchmark baseline for the official
+`Wan-AI/Wan2.1-I2V-14B-480P-Diffusers` pipeline.
+
+The goal is to compare:
+
+- the same model
+- the same I2V workload
+- the same async `/v1/videos` HTTP semantics
+- while changing only the inference framework
+
+Current fixed workload:
+
+- model: `Wan-AI/Wan2.1-I2V-14B-480P-Diffusers`
+- resolution: `480p`
+- effective size: `832x480`
+- frames: `81`
+- denoising steps: `40`
+- guidance scale: `5.0`
+- output fps: `16`
+
+## Layout
+
+- `service.py`: standalone FastAPI server
+- `configs/`: AIPerf configs for the baseline
+- `scripts/run_service.sh`: launch helper
+- `scripts/run_video_bench.sh`: benchmark helper
+
+## Start The Service
+
+```bash
+python3 benchmarks/baseline/diffusers_wan_i2v/service.py
+```
+
+Or:
+
+```bash
+bash benchmarks/baseline/diffusers_wan_i2v/scripts/run_service.sh
+```
+
+Default address:
+
+- `http://127.0.0.1:8010`
+
+Health check:
+
+```bash
+curl http://127.0.0.1:8010/v1/service/health
+```
+
+## Run AIPerf
+
+Install the vendored AIPerf first:
+
+```bash
+pip install -e ./benchmarks/aiperf
+```
+
+Then run:
+
+```bash
+bash benchmarks/baseline/diffusers_wan_i2v/scripts/run_video_bench.sh
+```
+
+This reuses the same prompt dataset as the TeleFuser benchmark:
+
+- `benchmarks/telefuser_aiperf/data/video_prompts.jsonl`
+
+## Notes
+
+- The service supports both `multipart/form-data` and JSON requests.
+- `reference_url` is treated as a local filesystem path in the current benchmark dataset.
+- Remote HTTP image URLs are intentionally rejected in this first baseline to keep the benchmark focused on inference rather than network fetch behavior.

benchmarks/baseline/diffusers_wan_i2v/benchmark_contract.yaml

@@ -0,0 +1,40 @@
+contract_version: v1
+name: diffusers_wan_i2v
+mode: batch_video
+implementation: diffusers
+model_family: wan_video
+model: Wan2.1-I2V-14B-480P-Diffusers
+supported_tasks:
+  - i2v
+transport: http
+endpoint:
+  path: /v1/videos
+  protocol: openai_video_generation
+request_encoding:
+  content_type: multipart/form-data
+  media_inputs:
+    first_image_path: input_reference
+  parameters:
+    prompt: prompt
+    negative_prompt: negative_prompt
+    seed: seed
+    size: size
+    seconds: seconds
+result_delivery:
+  polling_path: /v1/videos/{id}
+  content_path: /v1/videos/{id}/content
+workload:
+  resolution: 832x480
+  frames: 81
+  inference_steps: 40
+  fps: 16
+  guidance_scale: 5.0
+metrics:
+  - request_latency
+  - completion_latency
+  - download_latency
+  - request_throughput
+  - success_rate
+artifacts:
+  config: benchmarks/baseline/diffusers_wan_i2v/configs/video_generation_compare.yaml
+  dataset: benchmarks/telefuser_aiperf/data/video_prompts.jsonl

benchmarks/baseline/diffusers_wan_i2v/configs/video_generation_compare.yaml

@@ -0,0 +1,54 @@
+# yaml-language-server: $schema=../../../aiperf/src/aiperf/config/schema/aiperf-config.schema.json
+
+schemaVersion: "2.0"
+
+randomSeed: 42
+
+benchmark:
+  model: diffusers-wan21-i2v-480p
+
+  endpoint:
+    url: ${DIFFUSERS_WAN_AIPERF_URL:http://127.0.0.1:8010}
+    type: video_generation
+    timeout: 7200.0
+    downloadVideoContent: true
+    extra:
+      size: ${DIFFUSERS_WAN_AIPERF_SIZE:480p}
+      seconds: ${DIFFUSERS_WAN_AIPERF_SECONDS:5}
+      seed: ${DIFFUSERS_WAN_AIPERF_SEED:42}
+      model: ${DIFFUSERS_WAN_AIPERF_MODEL:Wan-AI/Wan2.1-I2V-14B-480P-Diffusers}
+
+  tokenizer:
+    name: builtin
+
+  dataset:
+    type: file
+    path: ./benchmarks/telefuser_aiperf/data/video_prompts.jsonl
+    format: single_turn
+    sampling: sequential
+
+  warmup:
+    type: concurrency
+    concurrency: 1
+    requests: 1
+    excludeFromResults: true
+
+  profiling:
+    type: concurrency
+    concurrency: ${DIFFUSERS_WAN_AIPERF_CONCURRENCY:1}
+    requests: ${DIFFUSERS_WAN_AIPERF_REQUESTS:2}
+    duration: ${DIFFUSERS_WAN_AIPERF_DURATION:7200}
+    gracePeriod: ${DIFFUSERS_WAN_AIPERF_GRACE_PERIOD:900}
+
+  artifacts:
+    dir: ./artifacts/diffusers_wan_i2v/wan21_i2v_480p_compare
+    summary: [json]
+    records: [jsonl]
+    showTraceTiming: true
+    trace: true
+
+  serverMetrics:
+    enabled: ${DIFFUSERS_WAN_AIPERF_SERVER_METRICS:false}
+    urls:
+      - ${DIFFUSERS_WAN_AIPERF_METRICS_URL:http://127.0.0.1:8010/metrics}
+    formats: [json, csv]

benchmarks/baseline/diffusers_wan_i2v/configs/video_generation_e2e.yaml

@@ -0,0 +1,54 @@
+# yaml-language-server: $schema=../../aiperf/src/aiperf/config/schema/aiperf-config.schema.json
+
+schemaVersion: "2.0"
+
+randomSeed: 42
+
+benchmark:
+  model: diffusers-wan21-i2v-480p
+
+  endpoint:
+    url: ${DIFFUSERS_WAN_AIPERF_URL:http://127.0.0.1:8010}
+    type: video_generation
+    timeout: 7200.0
+    downloadVideoContent: true
+    extra:
+      size: ${DIFFUSERS_WAN_AIPERF_SIZE:480p}
+      seconds: ${DIFFUSERS_WAN_AIPERF_SECONDS:5}
+      seed: ${DIFFUSERS_WAN_AIPERF_SEED:42}
+      model: ${DIFFUSERS_WAN_AIPERF_MODEL:Wan-AI/Wan2.1-I2V-14B-480P-Diffusers}
+
+  tokenizer:
+    name: builtin
+
+  dataset:
+    type: file
+    path: ./benchmarks/telefuser_aiperf/data/video_prompts.jsonl
+    format: single_turn
+    sampling: sequential
+
+  warmup:
+    type: concurrency
+    concurrency: 1
+    requests: 1
+    excludeFromResults: true
+
+  profiling:
+    type: concurrency
+    concurrency: ${DIFFUSERS_WAN_AIPERF_CONCURRENCY:2}
+    requests: ${DIFFUSERS_WAN_AIPERF_REQUESTS:6}
+    duration: ${DIFFUSERS_WAN_AIPERF_DURATION:7200}
+    gracePeriod: ${DIFFUSERS_WAN_AIPERF_GRACE_PERIOD:900}
+
+  artifacts:
+    dir: ./artifacts/diffusers_wan_i2v/video_e2e
+    summary: [json]
+    records: [jsonl]
+    showTraceTiming: true
+    trace: true
+
+  serverMetrics:
+    enabled: ${DIFFUSERS_WAN_AIPERF_SERVER_METRICS:true}
+    urls:
+      - ${DIFFUSERS_WAN_AIPERF_METRICS_URL:http://127.0.0.1:8010/metrics}
+    formats: [json, csv]

benchmarks/baseline/diffusers_wan_i2v/scripts/run_service.sh

@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
+cd "${ROOT_DIR}"
+
+PYTHON_BIN="${PYTHON_BIN:-python3}"
+SERVICE_HOST="${DIFFUSERS_WAN_I2V_HOST:-127.0.0.1}"
+SERVICE_PORT="${DIFFUSERS_WAN_I2V_PORT:-8010}"
+
+exec "${PYTHON_BIN}" benchmarks/baseline/diffusers_wan_i2v/service.py

benchmarks/baseline/diffusers_wan_i2v/scripts/run_video_bench.sh

@@ -0,0 +1,23 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
+cd "${ROOT_DIR}"
+
+CONFIG_PATH="${1:-benchmarks/baseline/diffusers_wan_i2v/configs/video_generation_e2e.yaml}"
+SERVER_URL="${DIFFUSERS_WAN_AIPERF_URL:-http://127.0.0.1:8010}"
+HEALTH_URL="${DIFFUSERS_WAN_AIPERF_HEALTH_URL:-${SERVER_URL}/v1/service/health}"
+AIPERF_BIN="${AIPERF_BIN:-aiperf}"
+
+if ! command -v "${AIPERF_BIN}" >/dev/null 2>&1; then
+    echo "aiperf is not installed. Set AIPERF_BIN or run: pip install -e ./benchmarks/aiperf" >&2
+    exit 1
+fi
+
+if command -v curl >/dev/null 2>&1; then
+    echo "Checking baseline service health: ${HEALTH_URL}"
+    curl --fail --silent --show-error "${HEALTH_URL}" >/dev/null
+fi
+
+echo "Running AIPerf with config: ${CONFIG_PATH}"
+"${AIPERF_BIN}" profile --config "${CONFIG_PATH}"