Repository navigation
Add a benchmark and a TL1 perf test for fn.experimental.readers.video - #6541
jantonguirao wants to merge 21 commits into
Conversation
internal_tools/video_reader_bench.py measures the experimental video reader on DALI_extra data only, with one scenario per aspect of its performance: - lanes: many distinct files, one keyframe-aligned sequence each, so a batch holds `batch` different files: decode parallelism across samples vs. num_threads. - engines: K independent readers in one pipeline: scaling with the number of concurrent NVDEC sessions. - single_stream: per-stream throughput (H.264, HEVC 8/10-bit, 512x512). - reader_depth: the reader's own prefetch_queue_depth at batch 1. - cache_key: per-switch cost between two files of the same format, cropped (800x600 coded as 800x608) or not, against a no-switch control. - startup: build time and first batch latency with many files. - cpu: the CPU backend on many VP8/VP9 files vs. num_threads. - seq_longgop, random_access, switch: no-regression checks for sequential, shuffled and mixed-resolution access. Every configuration runs in a fresh process, --repeat times, round-robin; the median and spread are reported. NVDEC utilization and memory are sampled per process with NVML, and runs are flagged when other processes use the GPU. --hash records per-sample checksums and --compare checks that two builds, or two thread counts (--ignore threads), produce identical output. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
There was a problem hiding this comment.
🟡 Changes recommended
Comparison can falsely pass failed configurations, and path handling and measurement accuracy need correction.
5 open issues
Missing comparison outputs incorrectly reported as identical · New Relative dataset paths create broken temporary-directory symlinks · New Decoder prefetch buffering biases queue-depth benchmarks · New Heterogeneous video resolutions produce incorrect throughput metrics · New Transient competing GPU processes evade contamination detection · New
What changed in this PR
Adds a DALI_extra-based benchmark for fn.experimental.readers.video to measure planned scaling improvements and check output consistency, without changing the library.
Changes:
- Adds scenarios for decoding parallelism, buffering, startup, and decoder-cache behavior.
- Runs isolated benchmarks with GPU monitoring and JSON reporting.
- Adds per-sample hashes and cross-run comparison.
| File | Description |
|---|---|
internal_tools/video_reader_bench.py |
Implements benchmark scenarios, execution, monitoring, and checksum comparison. |
Review effort: Balanced
| if "hashes" not in r or "hashes" not in other: | ||
| print(f"skip (failed run): {label(r)}") | ||
| continue |
There was a problem hiding this comment.
Fixed in 2d49b4e: a configuration that failed or is missing in either file is counted as a failed comparison (FAILED in <file> / MISSING in <file>), and --compare exits with 1.
|
|
||
|
|
||
| def resolve_files(dali_extra, name): | ||
| root = os.path.join(dali_extra, VIDEO_DIR) |
There was a problem hiding this comment.
Fixed in 2d49b4e: --dali_extra / DALI_EXTRA_PATH is made absolute after parsing, and resolve_files() builds an absolute root, so symlink targets and file_list entries are absolute. Tested with --dali_extra ../DALI_extra.
| # the first `depth` batches may have been produced off the clock | ||
| skip = min(cfg["depth"], len(stamps) - 1) | ||
| n = len(stamps) - skip | ||
| span = stamps[-1] - (stamps[skip - 1] if skip else t_start) |
There was a problem hiding this comment.
Fixed in 2d49b4e: the excluded iterations are now depth + reader_depth (pipeline queue plus the reader's own queue), and --iters/--min_time apply to the window after them, so --quick keeps 3 timed iterations.
| res.update( | ||
| iters=len(stamps), | ||
| frames_per_s=n * frames_per_iter / span, | ||
| mpix_per_s=n * frames_per_iter * h * w / span / 1e6, |
There was a problem hiding this comment.
Fixed in 2d49b4e: frame and pixel rates are computed from the actual output shapes of every timed batch (host-side shape metadata, no extra GPU work), so mixed-resolution sets are correct.
| res["contaminated"] = bool( | ||
| pre.get("other_pids") or post.get("other_pids") or pre.get("dev_nvdec_util", 0) > 5 | ||
| ) |
There was a problem hiding this comment.
Fixed in 2d49b4e: the sampler records other GPU processes throughout the run (other_pids_seen), and any process seen before, during or after the run marks the result as contaminated.
[Low impact] The changes appear safe to merge, with all three numbered findings addressed.
|
Runs internal_tools/video_reader_bench.py the way TL1_decoder_perf runs hw_decoder_bench.py: pinned to the same CPUs, re-run under nsys on failure with the profile and results saved to core_artifacts. The test fails if: - any benchmark configuration fails, - the output depends on num_threads (1 vs 16) or on the reader's prefetch_queue_depth (per-sample checksums, --hash/--compare), - throughput drops by more than 10% when num_threads (lanes, cpu) or the reader's prefetch_queue_depth grow, - an absolute minimum is set and not met. The minimums are empty for now, so the values are only printed for reference until they are calibrated on the CI machines. The checks live in qa/TL1_video_reader_perf/check_results.py, which reads the benchmark's JSON output. Also make the benchmark tolerate NVML queries that older bindings (nvidia-ml-py 11.450, as used in qa) do not support with newer drivers. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
|
CI MESSAGE: [75570122]: BUILD STARTED |
Results and checks: - --compare fails (exit 1) when a configuration failed or is missing in either file, reporting each failed run once with the file it came from. A build that cannot decode a workload no longer passes as "all identical". - A configuration counts as failed if any repetition failed: the benchmark exits with 1, and check_results.py rejects results with `errors`. Measurement: - Excluded iterations cover both the pipeline queue (`depth`) and the reader's own queue (`reader_depth`), and --iters/--min_time apply to the remaining window. - Frame and pixel rates count the actual output shapes of the timed batches, so mixed-resolution sets are reported correctly. - The GPU monitor records other GPU processes during the whole run, so a process that starts and exits mid-run marks the result as contaminated. A transient NVML failure no longer disables a sampler, and every NVML error is recorded. Robustness and style: - --dali_extra is made absolute, so symlinks in the temporary directory are valid with relative paths. - Validation errors name the parameter and show the expected and actual values; malformed layouts get a descriptive error. - Warnings use warnings.warn; files are always closed; no empty except. - The cache_key docs describe alternate - same as the cost of opening another file relative to rewinding the same one, not as an isolated switch cost. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
|
CI MESSAGE: [75573348]: BUILD STARTED |
nvidia-ml-py 11.450.51, which the TL1 tests install, returns bytes from nvmlDeviceGetName and friends and expects bytes in nvmlDeviceGetHandleByUUID. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
|
CI MESSAGE: [75570122]: BUILD PASSED |
|
CI MESSAGE: [75573348]: BUILD PASSED |
|
CI MESSAGE: [75641666]: BUILD STARTED |
|
CI MESSAGE: [75641666]: BUILD FAILED |
video_reader_bench.py gets --backends: besides the DALI reader, PyNvVideoCodec (SimpleDecoder) and torchcodec (FFmpeg CUDA backend, its own NVDEC backend, and CPU) decode the same samples through their own APIs, one decoder per thread of a thread pool. They run the scenarios that mean the same for them. The default stays dali only. TL1_video_reader_perf installs torch, FFmpeg, PyNvVideoCodec and a torchcodec matching the torch version on x86_64, and runs the comparison once there. The checks still apply to the DALI results only. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
A GPU backend can fall back to CPU decoding for a format it can't decode on the GPU (e.g. torchcodec's FFmpeg CUDA backend on HEVC 10-bit). Also leave build_s empty for the reference libraries, which open decoders lazily. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
- Reference libraries: min(threads, batch) decoder threads, cache_key at one thread, and no PyNvVideoCodec runs that seek mid-file (its SimpleDecoder raises IndexError there). Flag PyNvVideoCodec rows on clips whose lead-in frames are hidden by an MP4 edit list: it returns different frames. - --hash checks at least 40 samples, so that the reader's prefetch queue wraps; hash seq_longgop and random_access (with the HEVC trailer) too. - TL1: one hash run and compare, lanes at batch 16 only, --iters 8 (engines 20), a short reference run without DALI rows, no startup scenario, and random_access in the performance run. Check seq_longgop and random_access over threads and engines scaling. Re-run under nsys only when the performance checks fail, save all results on failure and print them to the log on success. - Remove dead code. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
- Remove the switch and startup scenarios, the pixel rate, cpu at batch 1, --hash_batches (HASH_MIN_SAMPLES only), the device NVDEC sampler and --require_idle, which no check or result needs. - torchcodec: select the CUDA backend explicitly (its default changed from FFmpeg to its own NVDEC decoder), rename torchcodec_beta to torchcodec_nvdec and report torchcodec's own cpu_fallback status. - TL1: install torch 2.11.0 with torchcodec 0.15.0 from the PyTorch cu129 or cu130 index for the comparison, instead of the torch pinned for the framework tests, so both CUDA versions run the same torchcodec. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
|
CI MESSAGE: [75801308]: BUILD STARTED |
Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
|
CI MESSAGE: [75801313]: BUILD STARTED |
print_results ended with a failing [ -f ] when the last file was missing (no reference run on GraceHopper, or a failed one), which exits test_body under set -e. Also give the empty except in provenance() a reason. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
|
CI MESSAGE: [75802756]: BUILD STARTED |
torchcodec 0.15 links against libpython3.X.so, which Ubuntu's python3 package does not include. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
|
CI MESSAGE: [75808789]: BUILD STARTED |
|
CI MESSAGE: [75808743]: BUILD STARTED |
Per-process NVDEC utilization is averaged over the GPU's NVDEC engines, so on GPUs with several engines a fully busy session reads well under 10% and the flag misfired. Only torchcodec can fall back to CPU decoding, and it reports that itself (CPU fallback flag). Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
dali_legacy runs the legacy reader through the same pipeline as the DALI reader, for reference: GPU only, no cache_key (both pairs include a variable frame rate file, which the legacy reader rejects), and no seeking in the open-GOP HEVC trailer, where it fails. The TL1 comparison step runs it too. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
The reference runner returned only the output shapes, so every sample's frames were freed right after decoding and the process memory of the libraries never included a batch, while DALI's includes its returned batch and its queues. A batch's outputs are now kept until the next batch is requested, as Pipeline.run() outputs are. PyNvVideoCodec's frames stay in its decoder's buffers, which its next call reuses, so its memory still does not include a batch; documented. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
|
CI MESSAGE: [75808789]: BUILD PASSED |
|
CI MESSAGE: [75808743]: BUILD PASSED |
… video reader benchmark The dali_decoder backend reads every sample's whole file with fn.readers.file and decodes its frames with fn.decoders.video (fn.experimental.decoders.video in older DALI), which decodes the samples of a batch in parallel on num_threads threads, with a new decoder per sample. It runs the same samples as the reader, in the reference scenarios plus engines and cpu, and the TL1 reference step. Like PyNvVideoCodec, it returns the lead-in frames that an MP4 edit list hides, so its rows on those clips are flagged. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
The checksum run of the video reader benchmark covered at least 40 samples per configuration: three batches at batch 16. That is enough for the reader's queues to wrap around, but in 'lanes' no decoder is ever reused in three batches, seq_longgop and random_access reach the next epoch in none, and no checksummed configuration runs several decode lanes with more than one batch in the reader's queue. --hash_epochs E checksums at least E epochs and one more batch of every configuration, so that every file of 'lanes' is opened again and decoders are reused across batches and epochs. A new scenario, lanes_depth, run only with --hash, decodes batches of 16 on several lanes with the reader's prefetch_queue_depth 1 and 2: whole files in order and shuffled, and many distinct files. TL1_video_reader_perf checks two epochs and lanes_depth. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
With decode lanes, one reader already spreads its decoding over the NVDEC engines, so 4 readers can no longer reach 2.5x one reader (H100 PCIe: 2016 frames/s with 1 reader, 3824 with 4 at 98% NVDEC, 1.9x). The check now also passes when the run with the most readers keeps NVDEC at least 90% busy. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
Trim internal_tools/video_reader_bench.py from 1266 to 314 lines and the TL1 test.sh from 177 to 83, and drop check_results.py. The benchmark keeps the timing and checksum sweeps, a --check gate (more threads or a deeper reader queue may be at most 10% slower) and the torchcodec reference rows, which never fail the test. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
Select the CPU list by CPU architecture and GPU compute capability, as in the TL1 decoder perf test, and fail on an unsupported platform instead of using the Grace Hopper settings. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
Make --dali_extra absolute so symlink targets do not depend on the working directory, remove the per-configuration temporary directory, set the pipeline's prefetch queue depth explicitly, make --check compare every row with the fastest row that has fewer threads or a shallower queue, and make --compare report a hash/throughput file mismatch instead of raising KeyError. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
Raise FileNotFoundError instead of asserting on missing data, hash more than an epoch so the reader's restart is covered, use a positive boolean for the torchcodec option and add the copyright header to the TL1 test script. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>


Category:
Other (e.g. Documentation, Tests, Configuration)
Description:
Adds
internal_tools/video_reader_bench.py, a benchmark forfn.experimental.readers.videothat uses DALI_extra data only, and a TL1 test that runs it. It is meant to measure, before and after, the planned work on making the reader scale withnum_threads: decode lanes, NVDEC session pipelining, multiple batches in flight, the NVDEC decoder cache fix (#6540), and parallel indexing. It also checks that this work doesn't change the output, and compares the reader with PyNvVideoCodec and torchcodec on the same samples.Each scenario isolates one aspect of the reader:
lanesnum_threads: every sample is a different file, one keyframe-aligned sequence each, so nothing seeksenginessingle_streamreader_depthprefetch_queue_depthat batch 1cache_keyalternate(every sample switches files) vs.same(rewinds one file), one threadseq_longgop,random_accessrandom_accessalso the HEVC trailer (open GOPs)cpunum_threadsMethodology:
--repeattimes (default 3), round-robin, and the table shows the median and the spread.--itersand--min_timeare reached. The firstdepth+reader_depthiterations (the pipeline's queue and the reader's own queue) are excluded, since they may have been prefetched off the clock. Frame rates count the actual output shapes.Pipeline.run()withoutcuda_streamreturns host-synchronized outputs, so no extra copies are added to the timed loop.CONTAMINATEDif any other process uses the GPU before, during or after it, or NVDEC is busy before it starts. The provenance (DALI version and path, GPU, driver, library versions,DALI_*/CUDA_*env) goes into the JSON.--hashrecords per-sample checksums of frames, labels and frame numbers, for at least 40 samples per configuration so that the reader's prefetch queue wraps around.--compare A B [--ignore threads,reader_depth]checks that two builds, or two thread counts, produce identical output, and exits non-zero on a difference, or when a configuration failed or is missing on either side.--hash_epochs Echecksums at least E epochs and one more batch of every configuration, so that decoders are reused across batches and epochs. Thelanes_depthscenario, run only with--hash, decodes batches of 16 on several decode lanes with the reader'sprefetch_queue_depth1 and 2.--quickis a smoke mode that runs everything in a few minutes.Other readers and libraries (
-b/--backends, defaultdalionly), for reference:dali_legacy: DALI's legacyfn.readers.video, through the same pipeline as the reader. GPU only, and nocache_key(both pairs include a variable frame rate file) or seeking in the open-GOP HEVC trailer, which it does not support.dali_decoder: DALI's video decoder,fn.readers.file+fn.decoders.video(fn.experimental.decoders.videoin older DALI). Every sample is a whole file, decoded with a new decoder per sample; the samples of a batch are decoded in parallel onnum_threadsthreads. Same samples as the reader, shuffled by the file reader.pynvvideocodec: PyNvVideoCodecSimpleDecoder, RGB interleaved output.torchcodec: torchcodecVideoDecoderwith FFmpeg's NVDEC support (device="cuda"), or FFmpeg on the CPU (cpuscenario).torchcodec_nvdec: torchcodec's own NVDEC decoder. The CUDA backend is selected explicitly, as torchcodec's default changed between versions.The libraries are called through their own APIs and decode the same samples as the reader (same sequences of the same files, in the same order, or shuffled with a fixed seed). A batch is decoded by
min(threads, batch)threads, each keeping its own decoder while consecutive samples come from the same file; batches are not prefetched. A batch's outputs are kept until the next batch is requested, as DALI's, so the process memory includes a batch (DALI's also its queued batches); PyNvVideoCodec's frames stay in its decoder's buffers, reused by its next call. They run the scenarios that mean the same for them, with two limits:cache_keyruns at one thread (with more, each file of a pair stays on its own decoder), and PyNvVideoCodec runs only configurations whose decoders start every file at frame 0. Rows are flagged when torchcodec reports a CPU fallback, and PyNvVideoCodec anddali_decoderrows on clips with lead-in frames hidden by an MP4 edit list are flagged, as they return those frames.TL1 test (
qa/TL1_video_reader_perf): runs the benchmark the wayTL1_decoder_perfrunshw_decoder_bench.py, with the same CPU pinning. The checks are inqa/TL1_video_reader_perf/check_results.py, and the test fails if:num_threads(1 vs 16) or on the reader'sprefetch_queue_depth, compared by per-sample checksums over two epochs (lanes,lanes_depth,reader_depth,cache_key,cpu,seq_longgop,random_access);num_threads(lanes,seq_longgop,random_access,cpu) or the reader'sprefetch_queue_depthgrow;engineswith 4 readers reaches less than 2.5x the throughput of one reader while NVDEC is less than 90% busy (with decode lanes, one reader already spreads its decoding over the NVDEC engines, so 4 readers can saturate NVDEC below 2.5x);MIN_LANES,MIN_SINGLE_STREAM,MIN_CPU,MIN_ENGINESintest.sh) are empty until a few CI runs establish the baseline; the measured values are printed asREFERENCElines;On failure, it saves the JSON results to
core_artifacts; if the performance run or its checks failed, it also re-runs the performance scenarios under nsys and saves the profile. On success, the results are printed to the log.Usage:
Additional information:
Affected modules and functionalities:
New files
internal_tools/video_reader_bench.pyandqa/TL1_video_reader_perf/{test.sh,check_results.py}. No changes to the library. The CI job is added separately in the CI configuration.Key points relevant for the review:
lanesuses distinct symlinked files, andcache_keyuses a no-switch control.enginesscaling check assumes a GPU with several NVDEC engines, as on the CI machines.Tests:
Existing tests apply
New tests added
N/A
New TL1 test
qa/TL1_video_reader_perf, run in CI on H100 (CUDA 12 and 13).Manual runs of the TL1 flow (benchmark, hash run and comparison, checker, reference run) with
--quick.--comparechecked against a tampered checksum file (non-zero exit code).Checklist
Documentation
DALI team only
Requirements
REQ IDs: N/A
JIRA TASK: DALI-4939