[jantonguirao]

Category:

New feature (non-breaking change which adds functionality)

Description:

fn.experimental.readers.video (CPU and GPU) decodes the samples of a batch in parallel, with up to num_threads decoders ("decode lanes"), and waits for all of them at the end of the batch.

Problem. Before this PR, the reader decoded every sample of a batch one after another, with one decoder, one CUDA stream and one thread. num_threads had no effect on decoding, and only one NVDEC session per reader was ever active. Video decoding is the bulk of the reader's work, so with large batches, or with samples spread over many files, the reader is limited by one decoder's throughput, while the CPU cores (CPU backend) or the NVDEC engines (GPU backend) that are available stay idle.

Solution. The loader still only describes the samples. The reader now plans each batch and decodes it on its lanes. Opening a video decoder is costly and a decoder decodes consecutive samples of a file without seeking, so the unit of work is a run, not a sample:

1. Plan (PlanBatch). Samples of one file that a decoder reaches without seeking form a run: as SeekFrame decides, at most kMinimumSeekLeap frames ahead or in the same keyframe segment (CanReachWithoutSeek). A sample that pad_last_batch repeats in the batch is decoded once, and a sample that it repeats into the next batch is not decoded again (a planned flag on the sample). Otherwise two lanes could decode the same sample at once, or a repeat could overwrite the data of a batch that may still be in use.
2. Assign (PlanLanes, new decode_lane_planner.{h,cc}).
   • Runs are costed in the pictures their decoder decodes, simulated with FramesDecoderBase::AddDecodeWork in the order the lane decodes them. That counts B-frames, pictures hidden by an edit list, where a seek really lands and decoder resets, plus a fixed cost (kOpenCost) per opening or reset of a decoder. Without an index, a run costs its frame range.
   • Runs go to lanes longest first, each to the lane that would finish it first, counting what a lane saves when its decoder already holds the file.
   • A lane decodes the runs of one file together, by first frame, starting with the file its decoder holds and the run that continues where the decoder is; the other files follow in batch order, each on a newly opened decoder. With more than one lane, a lane without runs releases its decoder, so that its frame buffers and NVDEC decoder are not kept while idle.
3. Decode (DecodeBatch, DecodeLanePart). Every lane has its own decoder, CUDA stream (GPU), constant frame and scratch, and decodes its runs on a thread pool with one thread per lane. When only one lane has work in a batch (always with num_threads=1), it decodes on the calling thread, as before. The batch is done when every lane is. A batch's error is the one of its first failing sample in batch order, whatever the lanes' timing.

The assignment depends only on the samples and on where the lanes' decoders are, never on thread timing, so neither does the output. The new tests check that the output is the same with one and with several threads.

Number of lanes: min(num_threads, max_batch_size) (at least one) and, on the GPU, at most two per NVDEC engine of the device (new FramesDecoderGpu::NumEngines, from CUVIDDECODECAPS::nNumNVDECs). With num_threads=1 there is one lane: one decoder, decoding on the calling thread, as before. The one difference is the order within a batch: the samples are decoded grouped by file and by first frame, not in batch order, so the samples of a file share one decoder even when they are not consecutive in the batch.

Every batch waits for its slowest lane. Running the lanes on their own threads and decoding a batch ahead is done in #6564.

This PR is split out of #6545.

Depends on #6558, #6560 and #6562, and on what they build on. Their commits are also on this branch until they merge:

• Sync the stream before a GPU video decoder frees its frame buffers #6556: "Sync the stream before a GPU video decoder frees its frame buffers"
• Estimate a video decoder's work from the decoding order in its index #6559: "Estimate a video decoder's work from the decoding order in its index" (AddDecodeWork, PositionAt, used by the planner)
• Fix NVDEC decoder cache never matching cropped videos #6540: the NVDEC decoder cache fix (three commits and a merge commit)
• Do not wait on the host for every picture the GPU video decoder converts #6561: "Do not wait on the host for every picture the GPU video decoder converts"
• Do not map, convert nor decode video pictures that are never read #6562: "Do not map, convert nor decode video pictures that are never read" (ParsedPictures, used for the lanes' diagnostics)
• Make the video reader's frame index cache thread-safe #6558: "Make the video reader's frame index cache thread-safe" (each reader holds its files' indices, which the planner and the lanes use from several threads)
• Cache cuvidGetDecoderCaps per process #6560: "Cache cuvidGetDecoderCaps per process" (decoders that open files at once in several lanes would otherwise serialize on the driver query)

This PR's own change is the last commit only: "Decode the video reader's batches with up to num_threads decode lanes".

Additional information:

Affected modules and functionalities:

• dali/operators/video/reader/video_reader_decoder_op.cc:
  • PlanBatch, AssignRuns, DecodeBatch, DecodeLanePart; DecodeSample is the former per-sample body of Prefetch, now on a lane's decoder, stream and constant frame;
  • the reader's single decoder, stream and constant frame are replaced by a vector of DecodeLanes and a thread pool (only with more than one lane);
  • VideoSample::planned, for samples repeated by pad_last_batch;
  • the first error of a batch is reported, after all lanes are done;
  • docstring: decoding in parallel by up to num_threads decoders, and memory growing with the number of decoders.
• dali/operators/video/reader/decode_lane_planner.{h,cc} (new): SampleFrames, DecoderPosition, CanReachWithoutSeek, PlanLanes. They are separate from the operator so that they can be tested on their own.
• dali/operators/video/frames_decoder_gpu.{h,cc}: FramesDecoderGpu::NumEngines(device_id).

fn.experimental.decoders.video is not affected.

Key points relevant for the review:

• Plan prediction. The planner must predict where SeekFrame seeks (CanReachWithoutSeek) and where each sample leaves its decoder (DecoderPosition, AddDecodeWork). A wrong prediction only makes the plan worse, never the output wrong: each sample is still decoded by DecodeSample with the decoder's own seeking logic, as before.
• Determinism. Which samples a lane decodes, and in which order, must not depend on thread timing (ties in PlanLanes go to the lowest lane). Otherwise a decoder's history, and with it the output, could change from run to run.
• Errors. A lane that fails a sample resets its decoder (its destructor waits for its stream) and goes on with its other samples. The batch reports the error of its first failing sample in batch order. On the GPU, a lane synchronizes its stream at the end of its part, and an asynchronous error there is reported only if no sample failed.
• GPU memory. Every lane has its own decoder and frame buffers, up to two per NVDEC engine. Lanes without work in a batch release their decoders.

Tests:

• Existing tests apply
  • decoder/test_video.py, checkpointing/test_dali_checkpointing.py: test_experimental_video_reader*, test_video_pipeline.py
• New tests added
  • Python tests
    • decoder/test_video.py:
      • test_reader_output_independent_of_num_threads, test_reader_batch_size_1_output_independent_of_num_threads, test_reader_multi_run_lanes_output_independent_of_num_threads (fewer lanes than samples), CPU and GPU, and test_reader_lead_in_output_independent_of_num_threads (GPU only: H.264 clip with a long edit-list lead-in). Each compares the output with 1 and with N threads, and checks the frames against PyAV.
      • test_reader_pad_last_batch_repeats_last_sample: a shard's last sample repeated in its batch and as the whole next batch.
      • test_reader_reports_first_error_of_batch: samples fail in two lanes and in the next batch; the error is the batch's first one.
  • GTests
    • decode_lane_planner_test.cc (new): DecodeLanePlannerTest.SampleFrames, CanReachWithoutSeek, LongestFirstToEarliestFinish, HeldFileContinues, NoIndex, HeldFileGoesFirst, RunsOfAFileInAscendingOrder, LeadIn, ResetAfterTheLastFrame, LaneOrder, PlannedPicturesAreDecoded.
    • frame_index_cache_test.cc: FrameIndexCacheTest.HandBuiltIndexIsInDecodingOrder, FrameIndexCacheTest.KeepsDecodingOrder (an index without decoding order falls back to frame order; a cached index, and a decoder's copy of it, keep the decoding order that the planner uses).
  • Benchmark
  • Other
• N/A

Checklist

Documentation

• Existing documentation applies
• Documentation updated
  • Docstring
  • Doxygen
  • RST
  • Jupyter
  • Other
• N/A

DALI team only

Requirements

• Implements new requirements
• Affects existing requirements
• N/A

REQ IDs: N/A

JIRA TASK: DALI-4935

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.