Skip to content

Add a benchmark and a TL1 perf test for fn.experimental.readers.video - #6541

Open
jantonguirao wants to merge 21 commits into
NVIDIA:mainfrom
jantonguirao:janton/dali-4935-video-reader-bench
Open

jantonguirao wants to merge 21 commits into
NVIDIA:mainfrom
jantonguirao:janton/dali-4935-video-reader-bench

Conversation

@jantonguirao

@jantonguirao jantonguirao commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Category:

Other (e.g. Documentation, Tests, Configuration)

Description:

Adds internal_tools/video_reader_bench.py, a benchmark for fn.experimental.readers.video that uses DALI_extra data only, and a TL1 test that runs it. It is meant to measure, before and after, the planned work on making the reader scale with num_threads: decode lanes, NVDEC session pipelining, multiple batches in flight, the NVDEC decoder cache fix (#6540), and parallel indexing. It also checks that this work doesn't change the output, and compares the reader with PyNvVideoCodec and torchcodec on the same samples.

Each scenario isolates one aspect of the reader:

Scenario What it measures Data
lanes per-file cost plus decoding vs. num_threads: every sample is a different file, one keyframe-aligned sequence each, so nothing seeks 256 distinct files (symlinks to the 15 Sintel H.264 clips), batch 16
engines scaling with concurrent NVDEC sessions (K independent readers in one pipeline, each on its own thread) as above, disjoint per reader
single_stream per-stream throughput (per-frame serialization) 1253-frame trailer as H.264, HEVC 8-bit, HEVC 10-bit; a 512x512 clip
reader_depth the reader's own prefetch_queue_depth at batch 1 many distinct files; one sequential long-GOP file
cache_key per-switch cost: alternate (every sample switches files) vs. same (rewinds one file), one thread 800x600 pair (coded 800x608, display area cropped) and a 1280x720 pair (not cropped)
seq_longgop, random_access sequential and shuffled access (keyframe seeks), no-regression the 15 clips; random_access also the HEVC trailer (open GOPs)
cpu CPU backend vs. num_threads 128 distinct VP8/VP9 files, batch 8

Methodology:

  • Each configuration runs in a fresh subprocess, so the frame index and NVDEC decoder caches don't carry over. It runs --repeat times (default 3), round-robin, and the table shows the median and the spread.
  • Timed iterations run until both --iters and --min_time are reached. The first depth + reader_depth iterations (the pipeline's queue and the reader's own queue) are excluded, since they may have been prefetched off the clock. Frame rates count the actual output shapes. Pipeline.run() without cuda_stream returns host-synchronized outputs, so no extra copies are added to the timed loop.
  • NVML records per-process NVDEC utilization and memory. A run is flagged CONTAMINATED if any other process uses the GPU before, during or after it, or NVDEC is busy before it starts. The provenance (DALI version and path, GPU, driver, library versions, DALI_*/CUDA_* env) goes into the JSON.
  • --hash records per-sample checksums of frames, labels and frame numbers, for at least 40 samples per configuration so that the reader's prefetch queue wraps around. --compare A B [--ignore threads,reader_depth] checks that two builds, or two thread counts, produce identical output, and exits non-zero on a difference, or when a configuration failed or is missing on either side.
  • --hash_epochs E checksums at least E epochs and one more batch of every configuration, so that decoders are reused across batches and epochs. The lanes_depth scenario, run only with --hash, decodes batches of 16 on several decode lanes with the reader's prefetch_queue_depth 1 and 2.
  • --quick is a smoke mode that runs everything in a few minutes.

Other readers and libraries (-b/--backends, default dali only), for reference:

  • dali_legacy: DALI's legacy fn.readers.video, through the same pipeline as the reader. GPU only, and no cache_key (both pairs include a variable frame rate file) or seeking in the open-GOP HEVC trailer, which it does not support.
  • dali_decoder: DALI's video decoder, fn.readers.file + fn.decoders.video (fn.experimental.decoders.video in older DALI). Every sample is a whole file, decoded with a new decoder per sample; the samples of a batch are decoded in parallel on num_threads threads. Same samples as the reader, shuffled by the file reader.
  • pynvvideocodec: PyNvVideoCodec SimpleDecoder, RGB interleaved output.
  • torchcodec: torchcodec VideoDecoder with FFmpeg's NVDEC support (device="cuda"), or FFmpeg on the CPU (cpu scenario).
  • torchcodec_nvdec: torchcodec's own NVDEC decoder. The CUDA backend is selected explicitly, as torchcodec's default changed between versions.

The libraries are called through their own APIs and decode the same samples as the reader (same sequences of the same files, in the same order, or shuffled with a fixed seed). A batch is decoded by min(threads, batch) threads, each keeping its own decoder while consecutive samples come from the same file; batches are not prefetched. A batch's outputs are kept until the next batch is requested, as DALI's, so the process memory includes a batch (DALI's also its queued batches); PyNvVideoCodec's frames stay in its decoder's buffers, reused by its next call. They run the scenarios that mean the same for them, with two limits: cache_key runs at one thread (with more, each file of a pair stays on its own decoder), and PyNvVideoCodec runs only configurations whose decoders start every file at frame 0. Rows are flagged when torchcodec reports a CPU fallback, and PyNvVideoCodec and dali_decoder rows on clips with lead-in frames hidden by an MP4 edit list are flagged, as they return those frames.

TL1 test (qa/TL1_video_reader_perf): runs the benchmark the way TL1_decoder_perf runs hw_decoder_bench.py, with the same CPU pinning. The checks are in qa/TL1_video_reader_perf/check_results.py, and the test fails if:

  • any benchmark configuration fails;
  • the output depends on num_threads (1 vs 16) or on the reader's prefetch_queue_depth, compared by per-sample checksums over two epochs (lanes, lanes_depth, reader_depth, cache_key, cpu, seq_longgop, random_access);
  • throughput drops by more than 10% when num_threads (lanes, seq_longgop, random_access, cpu) or the reader's prefetch_queue_depth grow;
  • engines with 4 readers reaches less than 2.5x the throughput of one reader while NVDEC is less than 90% busy (with decode lanes, one reader already spreads its decoding over the NVDEC engines, so 4 readers can saturate NVDEC below 2.5x);
  • an absolute minimum is set and not met. The minimums (MIN_LANES, MIN_SINGLE_STREAM, MIN_CPU, MIN_ENGINES in test.sh) are empty until a few CI runs establish the baseline; the measured values are printed as REFERENCE lines;
  • on x86_64, installing or running the reference step fails: the legacy reader, DALI's video decoder, PyNvVideoCodec 2.0.2 and torchcodec 0.15.0 (with torch 2.11.0 from the PyTorch cu129 or cu130 index, matching the CI CUDA version). Their numbers are reported, not checked.

On failure, it saves the JSON results to core_artifacts; if the performance run or its checks failed, it also re-runs the performance scenarios under nsys and saves the profile. On success, the results are printed to the log.

Usage:

DALI_EXTRA_PATH=... python internal_tools/video_reader_bench.py --json before.json
DALI_EXTRA_PATH=... python internal_tools/video_reader_bench.py -s lanes,engines -j 1,2,4,7,8,14,16 --readers 1,2,4,7,8,14
DALI_EXTRA_PATH=... python internal_tools/video_reader_bench.py -s lanes,single_stream -b dali,dali_legacy,dali_decoder,pynvvideocodec,torchcodec,torchcodec_nvdec
DALI_EXTRA_PATH=... python internal_tools/video_reader_bench.py --hash --json before_hash.json
python internal_tools/video_reader_bench.py --compare before_hash.json after_hash.json

Additional information:

Affected modules and functionalities:

New files internal_tools/video_reader_bench.py and qa/TL1_video_reader_perf/{test.sh,check_results.py}. No changes to the library. The CI job is added separately in the CI configuration.

Key points relevant for the review:

  • Scenario design: whether each one isolates what it claims to. In particular, lanes uses distinct symlinked files, and cache_key uses a no-switch control.
  • The reference comparison: same samples and synchronization semantics as DALI, and the limits above.
  • DALI_extra has no 1080p content. The tool is meant for comparing builds, not datasets.
  • The engines scaling check assumes a GPU with several NVDEC engines, as on the CI machines.

Tests:

  • Existing tests apply

  • New tests added

    • Python tests
    • GTests
    • Benchmark
    • Other
  • N/A

  • New TL1 test qa/TL1_video_reader_perf, run in CI on H100 (CUDA 12 and 13).

  • Manual runs of the TL1 flow (benchmark, hash run and comparison, checker, reference run) with --quick.

  • --compare checked against a tampered checksum file (non-zero exit code).

Checklist

Documentation

  • Existing documentation applies
  • Documentation updated
    • Docstring
    • Doxygen
    • RST
    • Jupyter
    • Other
  • N/A

DALI team only

Requirements

  • Implements new requirements
  • Affects existing requirements
  • N/A

REQ IDs: N/A

JIRA TASK: DALI-4939

internal_tools/video_reader_bench.py measures the experimental video reader on
DALI_extra data only, with one scenario per aspect of its performance:

- lanes: many distinct files, one keyframe-aligned sequence each, so a batch
  holds `batch` different files: decode parallelism across samples vs.
  num_threads.
- engines: K independent readers in one pipeline: scaling with the number of
  concurrent NVDEC sessions.
- single_stream: per-stream throughput (H.264, HEVC 8/10-bit, 512x512).
- reader_depth: the reader's own prefetch_queue_depth at batch 1.
- cache_key: per-switch cost between two files of the same format, cropped
  (800x600 coded as 800x608) or not, against a no-switch control.
- startup: build time and first batch latency with many files.
- cpu: the CPU backend on many VP8/VP9 files vs. num_threads.
- seq_longgop, random_access, switch: no-regression checks for sequential,
  shuffled and mixed-resolution access.

Every configuration runs in a fresh process, --repeat times, round-robin; the
median and spread are reported. NVDEC utilization and memory are sampled per
process with NVML, and runs are flagged when other processes use the GPU.
--hash records per-sample checksums and --compare checks that two builds, or
two thread counts (--ignore threads), produce identical output.

Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
Copilot AI balanced review requested due to automatic review settings October 21, 2026 13:12
@copy-pr-bot

copy-pr-bot Bot commented Oct 21, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Comment thread internal_tools/video_reader_bench.py Fixed
Comment thread internal_tools/video_reader_bench.py Fixed
Comment thread internal_tools/video_reader_bench.py Fixed
Comment thread internal_tools/video_reader_bench.py Fixed

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Comparison can falsely pass failed configurations, and path handling and measurement accuracy need correction.

5 open issues
What changed in this PR

Adds a DALI_extra-based benchmark for fn.experimental.readers.video to measure planned scaling improvements and check output consistency, without changing the library.

Changes:

  • Adds scenarios for decoding parallelism, buffering, startup, and decoder-cache behavior.
  • Runs isolated benchmarks with GPU monitoring and JSON reporting.
  • Adds per-sample hashes and cross-run comparison.
File Description
internal_tools/​video_reader_bench.py Implements benchmark scenarios, execution, monitoring, and checksum comparison.

Review effort: Balanced

Comment thread internal_tools/video_reader_bench.py Outdated
Comment on lines +672 to +674
if "hashes" not in r or "hashes" not in other:
print(f"skip (failed run): {label(r)}")
continue

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2d49b4e: a configuration that failed or is missing in either file is counted as a failed comparison (FAILED in <file> / MISSING in <file>), and --compare exits with 1.

Comment thread internal_tools/video_reader_bench.py Outdated


def resolve_files(dali_extra, name):
root = os.path.join(dali_extra, VIDEO_DIR)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2d49b4e: --dali_extra / DALI_EXTRA_PATH is made absolute after parsing, and resolve_files() builds an absolute root, so symlink targets and file_list entries are absolute. Tested with --dali_extra ../DALI_extra.

Comment thread internal_tools/video_reader_bench.py Outdated
Comment on lines +448 to +451
# the first `depth` batches may have been produced off the clock
skip = min(cfg["depth"], len(stamps) - 1)
n = len(stamps) - skip
span = stamps[-1] - (stamps[skip - 1] if skip else t_start)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2d49b4e: the excluded iterations are now depth + reader_depth (pipeline queue plus the reader's own queue), and --iters/--min_time apply to the window after them, so --quick keeps 3 timed iterations.

Comment thread internal_tools/video_reader_bench.py Outdated
res.update(
iters=len(stamps),
frames_per_s=n * frames_per_iter / span,
mpix_per_s=n * frames_per_iter * h * w / span / 1e6,

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2d49b4e: frame and pixel rates are computed from the actual output shapes of every timed batch (host-side shape metadata, no extra GPU work), so mixed-resolution sets are correct.

Comment thread internal_tools/video_reader_bench.py Outdated
Comment on lines +467 to +469
res["contaminated"] = bool(
pre.get("other_pids") or post.get("other_pids") or pre.get("dev_nvdec_util", 0) > 5
)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2d49b4e: the sampler records other GPU processes throughout the run (other_pids_seen), and any process seen before, during or after the run marks the result as contaminated.

@greptile-apps

greptile-apps Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Low impact] The changes appear safe to merge, with all three numbered findings addressed.

Summary

Adds a video-reader benchmark and a TL1 script that checks throughput and matching output across thread counts and reader queue depths.

  • The benchmark reports video-reader speed across several workloads.
  • A TL1 test measures video-reader speed and checks its output.

Reviews (16) · Last reviewed commit: "Address remaining review findings in the..." · Reviewed by Greptile

Comment thread internal_tools/video_reader_bench.py Outdated
Comment thread internal_tools/video_reader_bench.py Outdated
Comment thread internal_tools/video_reader_bench.py Outdated
Comment thread internal_tools/video_reader_bench.py Outdated
Comment thread internal_tools/video_reader_bench.py Outdated
Comment thread internal_tools/video_reader_bench.py Outdated
Comment thread internal_tools/video_reader_bench.py Outdated
Runs internal_tools/video_reader_bench.py the way TL1_decoder_perf runs
hw_decoder_bench.py: pinned to the same CPUs, re-run under nsys on failure with
the profile and results saved to core_artifacts.

The test fails if:
- any benchmark configuration fails,
- the output depends on num_threads (1 vs 16) or on the reader's
  prefetch_queue_depth (per-sample checksums, --hash/--compare),
- throughput drops by more than 10% when num_threads (lanes, cpu) or the
  reader's prefetch_queue_depth grow,
- an absolute minimum is set and not met. The minimums are empty for now, so
  the values are only printed for reference until they are calibrated on the
  CI machines.

The checks live in qa/TL1_video_reader_perf/check_results.py, which reads the
benchmark's JSON output.

Also make the benchmark tolerate NVML queries that older bindings
(nvidia-ml-py 11.450, as used in qa) do not support with newer drivers.

Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
@jantonguirao jantonguirao changed the title Add a benchmark for fn.experimental.readers.video Add a benchmark and a TL1 perf test for fn.experimental.readers.video Oct 21, 2026
Comment thread qa/TL1_video_reader_perf/check_results.py Fixed
Comment thread qa/TL1_video_reader_perf/check_results.py Outdated
@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [75570122]: BUILD STARTED

Results and checks:
- --compare fails (exit 1) when a configuration failed or is missing in either
  file, reporting each failed run once with the file it came from. A build that
  cannot decode a workload no longer passes as "all identical".
- A configuration counts as failed if any repetition failed: the benchmark exits
  with 1, and check_results.py rejects results with `errors`.

Measurement:
- Excluded iterations cover both the pipeline queue (`depth`) and the reader's own
  queue (`reader_depth`), and --iters/--min_time apply to the remaining window.
- Frame and pixel rates count the actual output shapes of the timed batches, so
  mixed-resolution sets are reported correctly.
- The GPU monitor records other GPU processes during the whole run, so a process
  that starts and exits mid-run marks the result as contaminated. A transient
  NVML failure no longer disables a sampler, and every NVML error is recorded.

Robustness and style:
- --dali_extra is made absolute, so symlinks in the temporary directory are valid
  with relative paths.
- Validation errors name the parameter and show the expected and actual values;
  malformed layouts get a descriptive error.
- Warnings use warnings.warn; files are always closed; no empty except.
- The cache_key docs describe alternate - same as the cost of opening another file
  relative to rewinding the same one, not as an isolated switch cost.

Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [75573348]: BUILD STARTED

nvidia-ml-py 11.450.51, which the TL1 tests install, returns bytes from
nvmlDeviceGetName and friends and expects bytes in nvmlDeviceGetHandleByUUID.

Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [75570122]: BUILD PASSED

@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [75573348]: BUILD PASSED

@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [75641666]: BUILD STARTED

@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [75641666]: BUILD FAILED

video_reader_bench.py gets --backends: besides the DALI reader, PyNvVideoCodec
(SimpleDecoder) and torchcodec (FFmpeg CUDA backend, its own NVDEC backend, and
CPU) decode the same samples through their own APIs, one decoder per thread of a
thread pool. They run the scenarios that mean the same for them. The default
stays dali only.

TL1_video_reader_perf installs torch, FFmpeg, PyNvVideoCodec and a torchcodec
matching the torch version on x86_64, and runs the comparison once there. The
checks still apply to the DALI results only.

Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
Comment thread internal_tools/video_reader_bench.py Fixed
A GPU backend can fall back to CPU decoding for a format it can't decode on
the GPU (e.g. torchcodec's FFmpeg CUDA backend on HEVC 10-bit). Also leave
build_s empty for the reference libraries, which open decoders lazily.

Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
- Reference libraries: min(threads, batch) decoder threads, cache_key at one
  thread, and no PyNvVideoCodec runs that seek mid-file (its SimpleDecoder
  raises IndexError there). Flag PyNvVideoCodec rows on clips whose lead-in
  frames are hidden by an MP4 edit list: it returns different frames.
- --hash checks at least 40 samples, so that the reader's prefetch queue
  wraps; hash seq_longgop and random_access (with the HEVC trailer) too.
- TL1: one hash run and compare, lanes at batch 16 only, --iters 8 (engines
  20), a short reference run without DALI rows, no startup scenario, and
  random_access in the performance run. Check seq_longgop and random_access
  over threads and engines scaling. Re-run under nsys only when the
  performance checks fail, save all results on failure and print them to the
  log on success.
- Remove dead code.

Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
Comment thread qa/TL1_video_reader_perf/test.sh Outdated
- Remove the switch and startup scenarios, the pixel rate, cpu at batch 1,
  --hash_batches (HASH_MIN_SAMPLES only), the device NVDEC sampler and
  --require_idle, which no check or result needs.
- torchcodec: select the CUDA backend explicitly (its default changed from
  FFmpeg to its own NVDEC decoder), rename torchcodec_beta to
  torchcodec_nvdec and report torchcodec's own cpu_fallback status.
- TL1: install torch 2.11.0 with torchcodec 0.15.0 from the PyTorch cu129 or
  cu130 index for the comparison, instead of the torch pinned for the
  framework tests, so both CUDA versions run the same torchcodec.

Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [75801308]: BUILD STARTED

Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [75801313]: BUILD STARTED

print_results ended with a failing [ -f ] when the last file was missing (no
reference run on GraceHopper, or a failed one), which exits test_body under
set -e. Also give the empty except in provenance() a reason.

Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [75802756]: BUILD STARTED

torchcodec 0.15 links against libpython3.X.so, which Ubuntu's python3
package does not include.

Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [75808789]: BUILD STARTED

@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [75808743]: BUILD STARTED

Per-process NVDEC utilization is averaged over the GPU's NVDEC engines, so on
GPUs with several engines a fully busy session reads well under 10% and the
flag misfired. Only torchcodec can fall back to CPU decoding, and it reports
that itself (CPU fallback flag).

Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
dali_legacy runs the legacy reader through the same pipeline as the DALI
reader, for reference: GPU only, no cache_key (both pairs include a variable
frame rate file, which the legacy reader rejects), and no seeking in the
open-GOP HEVC trailer, where it fails. The TL1 comparison step runs it too.

Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
The reference runner returned only the output shapes, so every sample's
frames were freed right after decoding and the process memory of the
libraries never included a batch, while DALI's includes its returned batch
and its queues. A batch's outputs are now kept until the next batch is
requested, as Pipeline.run() outputs are. PyNvVideoCodec's frames stay in
its decoder's buffers, which its next call reuses, so its memory still does
not include a batch; documented.

Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [75808789]: BUILD PASSED

@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [75808743]: BUILD PASSED

… video reader benchmark

The dali_decoder backend reads every sample's whole file with fn.readers.file and decodes its frames with fn.decoders.video (fn.experimental.decoders.video in older DALI), which decodes the samples of a batch in parallel on num_threads threads, with a new decoder per sample. It runs the same samples as the reader, in the reference scenarios plus engines and cpu, and the TL1 reference step. Like PyNvVideoCodec, it returns the lead-in frames that an MP4 edit list hides, so its rows on those clips are flagged.

Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
The checksum run of the video reader benchmark covered at least 40 samples
per configuration: three batches at batch 16. That is enough for the reader's
queues to wrap around, but in 'lanes' no decoder is ever reused in three
batches, seq_longgop and random_access reach the next epoch in none, and no
checksummed configuration runs several decode lanes with more than one batch
in the reader's queue.

--hash_epochs E checksums at least E epochs and one more batch of every
configuration, so that every file of 'lanes' is opened again and decoders
are reused across batches and epochs. A new scenario, lanes_depth, run only
with --hash, decodes batches of 16 on several lanes with the reader's
prefetch_queue_depth 1 and 2: whole files in order and shuffled, and many
distinct files.

TL1_video_reader_perf checks two epochs and lanes_depth.

Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
With decode lanes, one reader already spreads its decoding over the NVDEC engines, so 4 readers can
no longer reach 2.5x one reader (H100 PCIe: 2016 frames/s with 1 reader, 3824 with 4 at 98% NVDEC,
1.9x). The check now also passes when the run with the most readers keeps NVDEC at least 90% busy.

Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
Trim internal_tools/video_reader_bench.py from 1266 to 314 lines and the TL1 test.sh from 177
to 83, and drop check_results.py. The benchmark keeps the timing and checksum sweeps, a
--check gate (more threads or a deeper reader queue may be at most 10% slower) and the
torchcodec reference rows, which never fail the test.

Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
Select the CPU list by CPU architecture and GPU compute capability, as in the TL1 decoder
perf test, and fail on an unsupported platform instead of using the Grace Hopper settings.

Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
Comment thread internal_tools/video_reader_bench.py Outdated
Comment thread internal_tools/video_reader_bench.py Outdated
Comment thread internal_tools/video_reader_bench.py Outdated
Comment thread internal_tools/video_reader_bench.py Outdated
Comment thread internal_tools/video_reader_bench.py Outdated
Make --dali_extra absolute so symlink targets do not depend on the working directory, remove
the per-configuration temporary directory, set the pipeline's prefetch queue depth explicitly,
make --check compare every row with the fastest row that has fewer threads or a shallower
queue, and make --compare report a hash/throughput file mismatch instead of raising KeyError.

Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
Comment thread qa/TL1_video_reader_perf/test.sh
Raise FileNotFoundError instead of asserting on missing data, hash more than an
epoch so the reader's restart is covered, use a positive boolean for the
torchcodec option and add the copyright header to the TL1 test script.

Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants