Repository navigation
[CI test] Video reader: combine #6540, #6541 and #6545 - #6549
Draft
jantonguirao wants to merge 49 commits into
Draft
jantonguirao wants to merge 49 commits into
jantonguirao wants to merge 49 commits into
Conversation
The NVDEC decoder cache looked up decoders by the coded size of the stream (CUVIDEOFORMAT::coded_height/width), but stored the display area size (ulTargetHeight/Width) of the decoder it created. The two differ whenever the stream is cropped, e.g. 1080p H.264 is coded as 1920x1088 and displayed as 1920x1080, so for such videos no cached decoder ever matched. As the cuvidReconfigureDecoder fallback is compiled out (ENABLE_NVDEC_RECONFIGURE is never defined), every video opened by FramesDecoderGpu got a freshly created decoder, and unused ones piled up in the cache. In an nsys profile of experimental.readers.video on 1080p H.264 clips, each file switch stalled the reader for ~70 ms before the first kernel, ~57 ms of which were ioctls inside libnvcuvid, consistent with creating a new decoder. Store both the coded size and the display area in DecInstance and require both to match, so a decoder is reused only for streams it is really suited for. Add NVDECCacheSize() and a test that decodes the same 800x600 (coded 800x608) H.264 file with two consecutive decoders and checks that the second one reuses the cached decoder. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
Comparing the cache size before and after the second decode does not detect a cache miss when the cache is full: the miss inserts a new decoder and evicts an idle one, so the size stays the same. Replace NVDECCacheSize() with NVDECDecodersCreated(), a cumulative count of decoders created by the cache, and check that the second decoder does not increase it. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
internal_tools/video_reader_bench.py measures the experimental video reader on DALI_extra data only, with one scenario per aspect of its performance: - lanes: many distinct files, one keyframe-aligned sequence each, so a batch holds `batch` different files: decode parallelism across samples vs. num_threads. - engines: K independent readers in one pipeline: scaling with the number of concurrent NVDEC sessions. - single_stream: per-stream throughput (H.264, HEVC 8/10-bit, 512x512). - reader_depth: the reader's own prefetch_queue_depth at batch 1. - cache_key: per-switch cost between two files of the same format, cropped (800x600 coded as 800x608) or not, against a no-switch control. - startup: build time and first batch latency with many files. - cpu: the CPU backend on many VP8/VP9 files vs. num_threads. - seq_longgop, random_access, switch: no-regression checks for sequential, shuffled and mixed-resolution access. Every configuration runs in a fresh process, --repeat times, round-robin; the median and spread are reported. NVDEC utilization and memory are sampled per process with NVML, and runs are flagged when other processes use the GPU. --hash records per-sample checksums and --compare checks that two builds, or two thread counts (--ignore threads), produce identical output. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
Runs internal_tools/video_reader_bench.py the way TL1_decoder_perf runs hw_decoder_bench.py: pinned to the same CPUs, re-run under nsys on failure with the profile and results saved to core_artifacts. The test fails if: - any benchmark configuration fails, - the output depends on num_threads (1 vs 16) or on the reader's prefetch_queue_depth (per-sample checksums, --hash/--compare), - throughput drops by more than 10% when num_threads (lanes, cpu) or the reader's prefetch_queue_depth grow, - an absolute minimum is set and not met. The minimums are empty for now, so the values are only printed for reference until they are calibrated on the CI machines. The checks live in qa/TL1_video_reader_perf/check_results.py, which reads the benchmark's JSON output. Also make the benchmark tolerate NVML queries that older bindings (nvidia-ml-py 11.450, as used in qa) do not support with newer drivers. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
Results and checks: - --compare fails (exit 1) when a configuration failed or is missing in either file, reporting each failed run once with the file it came from. A build that cannot decode a workload no longer passes as "all identical". - A configuration counts as failed if any repetition failed: the benchmark exits with 1, and check_results.py rejects results with `errors`. Measurement: - Excluded iterations cover both the pipeline queue (`depth`) and the reader's own queue (`reader_depth`), and --iters/--min_time apply to the remaining window. - Frame and pixel rates count the actual output shapes of the timed batches, so mixed-resolution sets are reported correctly. - The GPU monitor records other GPU processes during the whole run, so a process that starts and exits mid-run marks the result as contaminated. A transient NVML failure no longer disables a sampler, and every NVML error is recorded. Robustness and style: - --dali_extra is made absolute, so symlinks in the temporary directory are valid with relative paths. - Validation errors name the parameter and show the expected and actual values; malformed layouts get a descriptive error. - Warnings use warnings.warn; files are always closed; no empty except. - The cache_key docs describe alternate - same as the cost of opening another file relative to rewinding the same one, not as an isolated switch cost. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
Move the check whether a cached decoder can be reused into NVDECCache::IsReusable and document each criterion, in particular why both the coded size and the display area have to match. No functional change. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
nvidia-ml-py 11.450.51, which the TL1 tests install, returns bytes from nvmlDeviceGetName and friends and expects bytes in nvmlDeviceGetHandleByUUID. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
video_reader_bench.py gets --backends: besides the DALI reader, PyNvVideoCodec (SimpleDecoder) and torchcodec (FFmpeg CUDA backend, its own NVDEC backend, and CPU) decode the same samples through their own APIs, one decoder per thread of a thread pool. They run the scenarios that mean the same for them. The default stays dali only. TL1_video_reader_perf installs torch, FFmpeg, PyNvVideoCodec and a torchcodec matching the torch version on x86_64, and runs the comparison once there. The checks still apply to the DALI results only. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
A GPU backend can fall back to CPU decoding for a format it can't decode on the GPU (e.g. torchcodec's FFmpeg CUDA backend on HEVC 10-bit). Also leave build_s empty for the reference libraries, which open decoders lazily. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
- Reference libraries: min(threads, batch) decoder threads, cache_key at one thread, and no PyNvVideoCodec runs that seek mid-file (its SimpleDecoder raises IndexError there). Flag PyNvVideoCodec rows on clips whose lead-in frames are hidden by an MP4 edit list: it returns different frames. - --hash checks at least 40 samples, so that the reader's prefetch queue wraps; hash seq_longgop and random_access (with the HEVC trailer) too. - TL1: one hash run and compare, lanes at batch 16 only, --iters 8 (engines 20), a short reference run without DALI rows, no startup scenario, and random_access in the performance run. Check seq_longgop and random_access over threads and engines scaling. Re-run under nsys only when the performance checks fail, save all results on failure and print them to the log on success. - Remove dead code. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
- Remove the switch and startup scenarios, the pixel rate, cpu at batch 1, --hash_batches (HASH_MIN_SAMPLES only), the device NVDEC sampler and --require_idle, which no check or result needs. - torchcodec: select the CUDA backend explicitly (its default changed from FFmpeg to its own NVDEC decoder), rename torchcodec_beta to torchcodec_nvdec and report torchcodec's own cpu_fallback status. - TL1: install torch 2.11.0 with torchcodec 0.15.0 from the PyTorch cu129 or cu130 index for the comparison, instead of the torch pinned for the framework tests, so both CUDA versions run the same torchcodec. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
print_results ended with a failing [ -f ] when the last file was missing (no reference run on GraceHopper, or a failed one), which exits test_body under set -e. Also give the empty except in provenance() a reason. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
torchcodec 0.15 links against libpython3.X.so, which Ubuntu's python3 package does not include. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
Per-process NVDEC utilization is averaged over the GPU's NVDEC engines, so on GPUs with several engines a fully busy session reads well under 10% and the flag misfired. Only torchcodec can fall back to CPU decoding, and it reports that itself (CPU fallback flag). Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
dali_legacy runs the legacy reader through the same pipeline as the DALI reader, for reference: GPU only, no cache_key (both pairs include a variable frame rate file, which the legacy reader rejects), and no seeking in the open-GOP HEVC trailer, where it fails. The TL1 comparison step runs it too. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
The reference runner returned only the output shapes, so every sample's frames were freed right after decoding and the process memory of the libraries never included a batch, while DALI's includes its returned batch and its queues. A batch's outputs are now kept until the next batch is requested, as Pipeline.run() outputs are. PyNvVideoCodec's frames stay in its decoder's buffers, which its next call reuses, so its memory still does not include a batch; documented. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
… video reader benchmark The dali_decoder backend reads every sample's whole file with fn.readers.file and decodes its frames with fn.decoders.video (fn.experimental.decoders.video in older DALI), which decodes the samples of a batch in parallel on num_threads threads, with a new decoder per sample. It runs the same samples as the reader, in the reference scenarios plus engines and cpu, and the TL1 reference step. Like PyNvVideoCodec, it returns the lead-in frames that an MP4 edit list hides, so its rows on those clips are flagged. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
FrameIndexCache::find returned an iterator after releasing its lock, and callers copied the index through it without the lock, while insert() could overwrite the same entry (two readers indexing the same file) or rehash the map. The cache moves to its own header with GetOrBuild, which builds every index once per file, outside the lock, and returns an immutable shared_ptr, and Find, which never builds or waits. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
…ffers FramesDecoderGpu's frame_buffer_ is returned to the memory pool without stream ordering when the decoder is destroyed (the reader replaces its decoder on every file switch), while an asynchronous copy out of it may still be pending on the decoder's stream. Another stream could then be given that memory. The destructor now synchronizes the stream first. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
…eads av_error_string formatted into a function-static buffer, and every decoder constructor set FFmpeg's process-wide log level. Use a local buffer, and set the log level once. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
VideoReaderDecoder::Prefetch decoded the batch in one loop with one decoder, one stream and shared scratch. It now plans the batch first, as the NemoAsr reader prepares its batch before decoding it: samples of one file that a decoder can decode in a row without seeking form a run, runs are assigned to decode lanes (longest first, preferring a lane whose decoder already holds the file), and every lane decodes its runs with its own decoder, stream, constant frame and scratch. With one lane, as in this change, the samples of a batch are grouped by file, so files alternating within a batch are reopened less often. A sample repeated in the batch is decoded once, and a sample the loader repeats in the next batch (pad_last_batch) is not decoded again while the previous batch may still be read. An error is reported for the first failing sample in the batch. FramesDecoderBase::HandleBoundary becomes static and the seek threshold a named constant, so that the planner can use them without a decoder. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
Every decode lane now runs on its own thread of an operator-owned pool, as the NemoAsr reader decodes its batch: up to num_threads lanes, no more than the batch size, and on the GPU at most two per NVDEC engine (cuvidGetDecoderCaps reports the engine count). With one engine, two lanes already keep it busy while a lane does its host work between pictures; more lanes add memory, one decoder each. The planning from the previous change makes the output independent of the number of lanes: which samples a lane decodes, and so its decoder's history, depends only on the samples. Tests check that the output is the same with 1 and 4 threads for consecutive, sparse, overlapping, shuffled, uniformly sampled and padded samples, and that a sample repeated by pad_last_batch into the next batch is the same sample. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
Compare the reader's frames to PyAV-decoded reference frames in the num_threads and pad_last_batch tests, not only to its own other runs. Use make_string in the frame index cache test, rename the public decoded_ member to decoded and ReachesWithoutSeek to CanReachWithoutSeek. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
Collaborator
|
CI MESSAGE: [76080463]: BUILD STARTED |
Collaborator
|
CI MESSAGE: [76080463]: BUILD PASSED |
Collaborator
|
CI MESSAGE: [76814524]: BUILD STARTED |
Collaborator
|
CI MESSAGE: [76849499]: BUILD STARTED |
Collaborator
|
CI MESSAGE: [76814524]: BUILD FAILED |
The reader plans its decode lanes by an estimate of the work of each sample. To estimate it in decoded pictures, the frame index needs two things it does not have: where each frame's packet is in decoding order, counting the packets hidden by an edit list (negative timestamps), which are decoded but not indexed, and where a seek to a keyframe really starts decoding. IndexEntry gets decode_idx and seek_decode_idx, so it grows from 16 to 24 bytes: the cached index of a dataset takes 8 more bytes per frame. BuildIndex counts the stream's packets as it reads them. A seek to a frame that SeekFrame seeks to (the last keyframe of a frame) lands on the keyframe itself, unless the stream has hidden packets. Then the demuxer is asked where av_seek_frame lands, as AvSeekFrame does, since its rule is not exposed: mov searches its sync samples by decoding timestamp offset by the edit list, so in some Sintel clips a seek to the shown IDR lands on the hidden one, 118 packets earlier. Asking costs a seek and a packet read per keyframe, which only these streams pay; in all-intra streams it would read the whole file a second time. Nothing that affects the output reads the new fields; the index order, keyframes and last_keyframe_id are unchanged. An index without decoding order (e.g. hand-built) is taken to be in decoding order. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
The planner of the reader's decode lanes estimates the work of a sample as the frames from the first to the last plus a preroll. What a decoder really does differs a lot: a frame costs every picture up to its own in decoding order (B-frames), clips with an edit list decode a hidden lead-in of up to ~190 pictures before their first frame, a seek can land a GOP before the keyframe, and a duplicate of the last frame (edge or reflect padding) reopens the file. FramesDecoderBase::AddDecodeWork estimates it: it mirrors SeekFrame (Reset, kMinimumSeekLeap, same keyframe, backward seek) on a DecodePosition and counts the pictures decoded, from the decoding order and seek landings recorded in the index, and the reopenings. PositionAt gives the position of a decoder from its next frame. Both decoders count the pictures they decode (DecodedPictures), so the tests check the estimate against real decoding on the Sintel clips, VP9, H.264 and HEVC files, in every path of SeekFrame. On the CPU it is exact. The GPU parser decodes a picture when the next packet arrives, and now and then only with the one after, so a GPU decoder can be one picture ahead of the estimate. By default the GPU tests run one Sintel clip of each kind (no lead-in, a single GOP, a seek landing on the hidden IDR, seeks landing on the shown IDRs, a long lead-in), which takes ~9 s; disabled tests sweep all 15 clips (~30 s). Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
The video reader assigns the runs of a batch to its decode lanes in AssignRuns, a private member of the operator. Nothing tested it: by design the output does not depend on the plan, so every test passes whatever the planner does. Move the planning, unchanged, into PlanLanes (decode_lane_planner.h), which takes the runs, where the lanes' decoders are and the open cost, and returns each lane's runs in decoding order and its load. CanReachWithoutSeek moves with it. AssignRuns now only fills the inputs from the batch and the decoders, copies the plans back and releases the decoders of empty lanes, as before. The new unit tests cover the longest-first, earliest-finish assignment, the continuation of a held file, the missing index and, against a copy of the previous code, the order in which a lane decodes its runs. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
The planner charged a run its frame range plus, after a seek, the frames back to the keyframe, and simulated a lane's runs in the order it assigned them (longest first), not in the order the lane decodes them. Both are far off for real files: a frame costs every picture up to its own in decoding order, clips with an edit list decode up to ~190 hidden pictures before their first frame, a duplicate of the last frame (edge or reflect padding) reopens the file, and a lane holding file X that is given a run of Y and then a run of X decodes X first on its decoder, while it was charged for reopening X after Y. Each sample now carries the frames its decoder seeks to (SampleFrames), and the lanes' starting positions come from PositionAt (DecoderPosition). PlanLanes charges a run on a lane by re-simulating, with AddDecodeWork, the runs of that file on that lane in the order the lane will decode them (the file the decoder holds first, with the continuing run in front, the others from a newly opened decoder), so the work of a lane is the sum over its files. The assignment stays longest first to the lane that finishes first, and the final order of a lane is the same as before, with ties within a file broken by run number. Without an index, a run still costs its frame range. kOpenCost (16) is now in pictures, also charged per reset. A lane's plan also gives its pictures, the load without kOpenCost per open and reset. The reader logs them next to the pictures the lane decoded (LOG_LINE, with DALI_VERBOSE_LOGS), and a test checks that they match on CPU decoders driven as the reader drives them, from SampleFrames and DecoderPosition. Checked against the decoders in the TL1 lanes, random_access, seq_longgop and cpu scenarios, with 1 to 16 threads: per lane and batch, they match exactly on the CPU and within 2 on the GPU (the parser holds one picture back at some seeks). The exception is +3 to +4 pictures for an HEVC decoder held from the previous batch whose last frames precede a keyframe in display order but follow it in decoding order: PositionAt takes it to have decoded only from that keyframe. Only the plan changes; a decoder's output does not depend on its history, which the reader's tests check across thread counts, now also on clips with hidden lead-ins and with lanes that decode several runs. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
FramesDecoderGpu waited for the color space conversion of every picture (cudaStreamSynchronize on its stream) before it unmapped the output surface, so the host was idle while each conversion ran. The surface now stays mapped after the conversion is launched, and an event is recorded behind it. The wait and the unmap move to ReleaseMappedFrame, which runs right before the next cuvidDecodePicture, in Flush, and in the destructor (after its stream sync), before the decoder goes back to the NVDEC cache. So NVDEC is still used only after the host has seen every earlier conversion finish, as before, and at most one surface is mapped at a time; only host work (reading, filtering and parsing the next packet) now overlaps the conversion. The output does not change. The destructor unmaps the surface on the decoder's device, as it may run on a thread where another device is current. A surface counts as unmapped only once cuvidUnmapVideoFrame succeeds: one that failed to unmap stays tracked, so the destructor tries again and the tests see it. A conversion that fails after the map (e.g. for an unsupported output type) no longer leaks the mapping: the surface is tracked before anything can throw, and released after a sync of the whole stream. NVDECFramesMapped counts the mapped surfaces, for tests. The move constructor is deleted: the parser calls back into the object, and a moved-from copy of a mapping would be unmapped twice. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
The GPU frames decoder no longer waits for each picture, so an asynchronous error in a sample's conversions or copies would surface at a later sample of the same decoder, or at the end of a reader lane, where it is not attributed to any sample. The video reader lanes and decoders.video now wait for the decoder's stream after each sample, inside that sample's error handling. In the reader this is a wait for the last conversion and copies of the sample; the lane-end sync stays as a safety net. In decoders.video the decoder's destructor waited for that stream anyway, but swallowed its errors. inputs.video keeps its decoder across Run calls, so it waits for its stream at the end of each Run: the errors of the last pictures of a Run are reported by this operator, not by the next Run or by whatever touches the context next. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
The decoder no longer waits for each picture's conversion. Hold up the decoder's stream before every frame, so that the conversions are still pending when the next picture is decoded, when the decoder seeks and flushes, and when it is destroyed, and check that the frames are the same as those decoded on the default stream without the delay, with and without an index. Also check that the output surface is unmapped before the NVDEC decoder goes back to the cache, where the next decoder takes it on another stream; that decoders can be destroyed in another thread than the one that decoded; and that a conversion failing after the map does not leave the surface mapped. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
The GPU frames decoder mapped and color-converted every picture that it did not return as it came, also those that are never read. With an index, these are e.g. the pictures that an edit list hides at the start of a clip (up to 188 in the Sintel clips), which are decoded before the first frames, and again after each seek that lands on them. Each of them also took a frame buffer of its own, kept until the decoder is destroyed: up to ~190 buffers (~500 MB at 720p) per decoder. A buffered picture is now mapped and converted only if it may be read with its data later: if its timestamp is that of a frame at or after the next one in the index. Until the decoder is flushed, the buffered frames are only looked up for those frames (the next frame only moves forward between flushes), so the other pictures are never read. Such a picture still takes its slot in the frame buffer, with the same timestamp as before, so which frames are found, in which order, and what is decoded do not change; only the frame buffer is not needed. The slots now share the frame buffers: a slot that needs one takes the buffer of a free slot, or of one that keeps only a timestamp, and allocates one only if there is none. Without an index, every picture is mapped, as before. NVDEC decodes the same pictures, and the output does not change. Reading a frame of which only the timestamp was kept is an internal error, so a wrong decision gives an error, not a wrong frame. MappedPictures, NumFrameBuffers and SetBufferedFrameDataForTesting are meant for tests: the last one lets a test map every buffered picture, to compare the frames read with those of the decoder as it was, or none. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
To read a frame, the decoder decodes the pictures from its keyframe on, and to read the next frames of a sample with gaps (a stride, a random or uniform sampling) it decodes through the frames in between. Those skipped frames that it decodes before it reaches them (e.g. P-frames decoded ahead of B-frames) are buffered, mapped and converted, although they are only skipped later. DecodeFrames now tells the decoder which frames it reads next, and SeekFrame, when called on its own, the frame it seeks to. A buffered frame up to the last of those frames that is not one of them is only skipped, so the GPU decoder keeps only its timestamp, as it does for the frames it never reads. Frames after the last one are still mapped: a later DecodeFrames on the same decoder (the next sample of a decode lane) may read them. A frame left without its data is below the next frame once DecodeFrames or SeekFrame returns, and reading a frame below the next one flushes the decoder first, so it is never read with its data afterwards either: not by a later sample, a duplicate or padded frame, nor after a checkpoint is restored. As before, which frames are buffered and decoded does not change, and neither does the output. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
Check that a GPU frames decoder returns the same frames as one that maps every picture (as the decoder did before), after decoding the same pictures, when it decodes samples with skipped frames, duplicates and padding, when it seeks, also with AvSeekFrame to a frame that is not a keyframe, and when it decodes one sample after another, also one that starts before the end of the previous one. The files: H.264 clips with long edit-list lead-ins, a VFR H.264 file and, if supported, HEVC ones. For a clip with a lead-in of 188 hidden pictures, check that the decoder maps no more than the frames it reads and the few decoded ahead of the last one, also when it skips frames, and that it keeps at most 10 frame buffers (before, one for each hidden picture). Reading a frame that was not mapped fails with an internal error. The reader tests also run on such a clip now, checked against PyAV around each frame (the whole file is slow to compare against). Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
DataReader::Prefetch reads a batch into the producer's slot of the prefetch queue and stores the loader's state after it. A reader that reads its next batch ahead of the slot it is published in needs the two steps on their own: ReadBatch reads a batch into any batch vector, and SaveLoaderSnapshot takes the snapshot to store, taken when that batch was read. No behaviour change: Prefetch does the same through the new functions. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
The video reader plans which lane decodes which samples of a batch from where the lanes' decoders are. It read that from the decoders themselves, so a batch could be planned only once the lanes were done with the batch before it. Where decoding a sample leaves its decoder is known before decoding: after the largest frame it decodes (DecodeFrames maps the frames into the file once more and decodes them in order), or where it was if it decodes none, or at frame 0 of a newly opened decoder. So the reader now keeps, for each lane, where its decoder will be once it is done with the batches planned so far, and plans the next batch from that. The plans are the same as before: the positions are normalized as the planner sees them (any frame past the end is the end). A decoder may be elsewhere only if it does not decode every frame it is asked for, which already leaves that sample's frames undefined. Decoding does not depend on it (it seeks from where the decoder is), only the plan can be worse. Each lane checks its decoder against the plan before decoding; DALI_VIDEO_READER_STRICT_PLANS, for tests, turns a mismatch into an error, and the tests of the lanes' output set it. Without it, a mismatch is only logged (in debug builds). A sample is marked when it is planned, not when it is decoded, which also covers samples repeated within a batch. DecoderPosition, now used only by the planner test, moves there. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
The video reader decoded a batch's lanes on a thread pool and waited for all of them (RunAll) before reading the next batch. A lane that finished early had nothing to do until the slowest lane finished too. Each lane now runs on its own thread, started with the first batch, and does its parts of the batches from its own queue, in the order the batches are read. A batch keeps its samples, its plan, the loader's snapshot and its errors until it is published, and is published once every lane is done with it. An idle lane releases its decoder on its thread, in its turn. With one lane, no thread is started: the batch is decoded on the prefetch thread, as before. The lanes' threads use the operator's device, as the thread pool's did. This commit still reads, decodes and publishes one batch at a time, so the output and the plans are as before. An error of a batch is still the first one by position in the batch; once a batch fails, or while the reader is destroyed, the lanes skip the samples they have not started and nothing is published any more. Tests: the output over two epochs with the reader's prefetch_queue_depth 1 and 2, pad_last_batch repeating the last sample in its own batch as well as in the next one, the first error of a batch reported with errors in several lanes and in the next batch, destroying the reader while it decodes or before it starts, and checkpoints with prefetch_queue_depth 1 and 2 and a shuffle buffer. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
With more than one decode lane, every batch ended with all lanes waiting for the slowest one: a lane that was done with its part had nothing to do until the batch was published and the next one read and planned. The reader now reads, plans and queues one batch ahead of the one it publishes. A lane that is done with its part of a batch goes on to its part of the next one, while the others finish theirs. Batches are still published in reading order, each once all lanes are done with it, with the loader's snapshot taken when it was read, and its first error by position in the batch; once a batch fails, the lanes skip the batches after it. The samples read and the plans are the same as before, so is the output. The lookahead is bounded by the reader's queue: the batch after the next one is read only when the queue has room. A consumed batch's samples go back to the loader before the next batch is read, so the loader's pool (2 * prefetch_queue_depth batches) suffices. With more than one lane, one more batch of decoded samples is held in memory; the documentation says so. With one lane nothing changes. The tests added with the lanes' threads now also cover the lookahead: the output with the next batch decoded ahead, an error in the batch after the failing one that the lanes may already be decoding, and destroying the reader while lanes decode ahead. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
The planner gives each run of a batch to the lane that would finish it first, and a tie to the lowest lane. Lanes without a decoder for the run's file often tie, e.g. at the start of every batch of distinct files, so lane 0 kept getting the longest runs. While the lanes had to wait for each other at the end of every batch, that did not matter. Now that a lane goes on to the next batch as soon as it is done with its part, the work a lane has more than another carries over. The reader keeps each lane's planned work beyond the least loaded lane's (its backlog), and the planner gives a tie to the lane with the least backlog. The backlog comes from the plans only, so the plans still depend only on the samples, not on timing; it starts at 0, also after a checkpoint is restored. This only picks among lanes the planner rates as finishing a run at the same time, but it changes which lane decodes which run, and so where the decoders are for later batches. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
A non-reference picture that is not read with its data (hidden by an edit list, or only skipped on the way to the frames to read) is no longer decoded: no other picture refers to it, and only its timestamp is kept. ParsedPictures (decoded + left out) is what AddDecodeWork estimates; the tests and the lanes' decode accounting compare it. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
- The cache keeps an index only while a reader holds it (weak_ptr), so indices are freed with their readers instead of living for the whole process, and an index is built again when the file's size or modification time changed since it was built. - Each reader holds the indices of its files and plans and decodes with those (no cache lookups after setup), so an index cannot change under a reader. - shared_mutex (lookups in parallel, double-checked insertion) and string_view lookups. - instance() is defined in a .cc file, so that there is one cache per process. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
zip(..., strict=True) in _assert_frames_match_file, so that no decoded frame goes unchecked, and the lanes reader with one sample per batch. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
The checksum run of the video reader benchmark covered at least 40 samples per configuration: three batches at batch 16. That is enough for the reader's queues to wrap around, but in 'lanes' no decoder is ever reused in three batches, seq_longgop and random_access reach the next epoch in none, and no checksummed configuration runs several decode lanes with more than one batch in the reader's queue. --hash_epochs E checksums at least E epochs and one more batch of every configuration, so that every file of 'lanes' is opened again and decoders are reused across batches and epochs. A new scenario, lanes_depth, run only with --hash, decodes batches of 16 on several lanes with the reader's prefetch_queue_depth 1 and 2: whole files in order and shuffled, and many distinct files. TL1_video_reader_perf checks two epochs and lanes_depth. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
With decode lanes, one reader already spreads its decoding over the NVDEC engines, so 4 readers can no longer reach 2.5x one reader (H100 PCIe: 2016 frames/s with 1 reader, 3824 with 4 at 98% NVDEC, 1.9x). The check now also passes when the run with the most readers keeps NVDEC at least 90% busy. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
jantonguirao
force-pushed
the
janton/video-effort-ci-test
branch
from
October 27, 2026 12:10
831e39d to
74cff1c
Compare
Collaborator
|
CI MESSAGE: [77005033]: BUILD STARTED |
Collaborator
|
CI MESSAGE: [76905662]: BUILD PASSED |
A single script (video_reader_bench.py) that runs each configuration in a fresh process, reports the median frames/s and checks that the output does not depend on num_threads or the reader's prefetch_queue_depth. The GPU configurations also run through torchcodec, for reference only. - Configurations: lanes, cpu, seq, random, random_hevc, reader_depth and two single-stream ones; reader_depth 2 with batches of 16 is checked for output only. - --check fails when more threads (or a deeper reader queue) are more than 10% slower than fewer; --compare compares two result files. - The TL1 test installs torch and torchcodec on a best-effort basis: without them the torchcodec rows are skipped and the test still passes. - check_results.py is gone: its checks are part of the script. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>
Collaborator
|
CI MESSAGE: [77071160]: BUILD STARTED |
Collaborator
|
CI MESSAGE: [77005033]: BUILD PASSED |
Collaborator
|
CI MESSAGE: [77071160]: BUILD PASSED |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Category:
Other (CI test branch, not for merging)
Description:
Draft PR that combines #6540, #6541 and #6545 on top of
main, to exercise them together on CI. It is not meant to be merged: each change is reviewed and merged in its own PR, and this branch will be closed afterwards.Additional information:
Affected modules and functionalities:
frames_decoder_gpu).fn.experimental.readers.videobenchmark and theTL1_video_reader_perfqa test.video_reader_decoder_op, frame index cache, FFmpeg helpers).Key points relevant for the review:
Review the individual PRs, not this one. The branch is the three PR heads merged into
mainwithout conflicts.Tests:
Tests are those of the three PRs (
frames_decoder_test.cc,frame_index_cache_test.cc,test_video.py,TL1_video_reader_perf).Checklist
Documentation
DALI team only
Requirements
REQ IDs: N/A
JIRA TASK: N/A