Convert video to RGB on the GPU with the BT.709 matrix when the stream uses it by jantonguirao · Pull Request #6568 · NVIDIA/DALI

jantonguirao · 2026-10-28T08:33:58Z

Category:

Bug fix (non-breaking change which fixes an issue)

Description:

The GPU video decoders (fn.experimental.readers.video, fn.experimental.decoders.video and fn.experimental.inputs.video on the GPU) always convert YCbCr to RGB with the ITU-R BT.601 matrix, whatever the stream says. HD video is normally BT.709, so its colors are wrong, noticeably for saturated colors: for example, pure red (255, 0, 0) in a BT.709 stream is decoded as about (234, 0, 3), and pure green (0, 255, 0) as about (21, 255, 10).

This PR makes the decoders convert with the BT.709 matrix, in limited or in full range, when the stream is BT.709, and keep BT.601 for every other stream (untagged streams are not affected). It uses the new itu_r_bt_709 and itu_r_bt_709_full_range conversions from #6567: the conversion kernel in color_space.cu gets two new conversion types.

Depends on #6567; its commit is also on this branch until it merges. The new test uses the video clips and the ffmpeg references added in NVIDIA/DALI_extra#137 (db/video/color_matrix), which is merged: DALI_EXTRA_VERSION is updated to the commit that merged it.

On the test clip (1080p, tagged BT.709, limited range) the error of the mean color of the saturated bars against the original pattern, and the mean difference of the whole frame against ffmpeg's decode (the reference), are:

	worst error in the bars	mean difference to ffmpeg
before	22.0	6.59
after	2.0	1.43

(ffmpeg itself is within 2.0 of the original pattern.)

Not changed:

the legacy fn.readers.video has the same BT.601 conversion (legacy/reader/nvdecoder.cc), and is left as it is;
only BT.709 is added; other matrices (e.g. BT.2020) are still converted as BT.601.

Additional information:

Affected modules and functionalities:

dali/operators/video/color_space.{h,cu}: new conversion types VIDEO_COLOR_SPACE_CONVERSION_TYPE_YUV_TO_RGB_BT709 and ..._BT709_FULL_RANGE.
dali/operators/video/frames_decoder_gpu.{h,cc}: FramesDecoderGpu::IsBT709 and the choice of the conversion type in InitGpuDecoder.
dali/test/python/decoder/test_video.py: new test.

Key points relevant for the review:

The matrix is read from FFmpeg (codec_params_->color_space), not from NVDEC's video_signal_description.matrix_coefficients. NVDEC leaves the whole color description zeroed for VP9 (the matrix is 0 even for a BT.709 stream), and for MPEG-4 it reports BT.709, the codec's default, for streams that have no color description, which would change the output of untagged MPEG-4 videos. This is also what the CPU decoder uses.
Streams for which FFmpeg doesn't report BT.709 use exactly the same conversion as before, bit for bit. Of the video files in DALI_extra, only the new fixtures are tagged as BT.709 (the others are untagged or BT.470BG).
The colors of BT.709 streams do change (they become correct), so a user comparing against old outputs of such videos will see a difference.

Tests:

Existing tests apply
- decoder/test_video.py: all of it (the GPU and CPU decoders, test_full_range_video*, the codec support tests).
New tests added
- Python tests
- GTests
- Benchmark
- Other

New test: decoder/test_video.py::test_color_matrix_bt709_gpu, on the H.264 and the VP9 clips. It checks the mean color of the interior of each of the 10 bars of the pattern (the 8 colors, a black and a white bar) against ffmpeg's decode (at most 4 apart; BT.601 gives 22) and the mean difference of the whole frame (below 2). It also checks that the black bar is at most 2 and the white bar at least 253, i.e. that the luma is mapped to the whole range and clamped there (Y=16 and Y=235 in the limited range); those two bars do not depend on the matrix, so that check passes without this change too, and the saturated bars are the ones that fail without it.

Run locally on a TITAN V (sm_70) with the three PRs' changes: test_video.py passes (136 tests, 10 skipped). After the black and white bars were added to the clips, I ran test_color_matrix_bt709_gpu again with the new clips: the 4 cases (H.264 limited and full range, VP9, and the stream with the matrix only in the H.264 stream) pass.

Checklist

Documentation

DALI team only

Requirements

Implements new requirements
Affects existing requirements
N/A

REQ IDs: N/A

JIRA TASK: DALI-4954

Add itu_r_bt_709 (16-235 range, the same API as itu_r_bt_601; the gray conversions do not depend on the matrix and are inherited) and itu_r_bt_709_full_range (YCbCr to RGB over the whole range of the type, the counterpart of jpeg with the BT.709 matrix). HD video is normally BT.709, while only BT.601 was available. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>

copy-pr-bot · 2026-10-28T08:34:03Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Copilot

🟡 Changes recommended

BT.709 metadata present only in H.264 sequence headers can be missed when dimensions are known, causing incorrect BT.601 conversion, while full-range decoder selection and CUDA dispatch remain untested.

2 open issues

What changed in this PR

This PR adds BT.709-aware RGB conversion to the experimental GPU video decoders, using FFmpeg color metadata while preserving BT.601 conversion for other streams.

Changes:

Adds limited- and full-range BT.709 conversion helpers and numerical reference tests.
Connects BT.709 conversion types to decoder selection and CUDA dispatch.
Adds limited-range H.264 and VP9 video regressions against FFmpeg references.

File	Description
`dali/test/python/decoder/test_video.py`	Adds BT.709 GPU decoding regressions and color-bar comparisons.
`dali/operators/video/frames_decoder_gpu.h`	Declares the BT.709 metadata check.
`dali/operators/video/frames_decoder_gpu.cc`	Selects RGB conversion by stream matrix and range.
`dali/operators/video/color_space.h`	Adds limited- and full-range BT.709 conversion types.
`dali/operators/video/color_space.cu`	Dispatches the new BT.709 CUDA conversions.
`dali/kernels/imgproc/color_manipulation/color_space_conversion_impl.h`	Implements reusable BT.709 conversion formulas.
`dali/kernels/imgproc/color_manipulation/color_space_conversion_impl_test.cc`	Tests reference accuracy, primary colors, and round trips.

_{Review effort: Balanced}

+  // Use the matrix that FFmpeg got from the stream (as the CPU decoder does), not the one that
+  // NVDEC reports: NVDEC doesn't report the color description for some codecs (e.g. VP9), and for
+  // others it reports the codec's default (BT.709 for MPEG-4) when the stream doesn't have one.
+  return codec_params_->color_space == AVCOL_SPC_BT709;


+        conversion_type_ = is_full_range
+                               ? VIDEO_COLOR_SPACE_CONVERSION_TYPE_YUV_TO_RGB_BT709_FULL_RANGE
+                               : VIDEO_COLOR_SPACE_CONVERSION_TYPE_YUV_TO_RGB_BT709;


greptile-apps · 2026-10-28T08:39:36Z

Confidence Score: 4/5

[Medium impact] The change affects RGB output in GPU video decoding workflows.

The PR still needs the earlier fix for BT.709 metadata lost when a stream is reopened.

Findings

Reopening loses the matrix ▶

Summary

Adds BT.709 conversion for experimental GPU video readers, decoders, and inputs.

GPU video uses BT.709 when FFmpeg marks the stream BT.709.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Open video] --> B[Probe color metadata]
  B --> C[Initialize GPU parser]
  C --> D[Receive video sequence]
  D --> E{RGB output?}
  E -->|No| F[Upsample YCbCr]
  E -->|Yes| G{BT.709?}
  G -->|Yes| H[BT.709 conversion]
  G -->|No| I[BT.601 conversion]
  H --> J[Use stream range]
  I --> J

_{Reviews (3) · Last reviewed commit: "Probe the stream for its color matrix an..." · Reviewed by Greptile}

…all types The BT.709 conversions inherited from BT.601 only to reuse the gray conversions. Move those to a common base of both, which says what is shared. Run the BT.709 conversions through the typed tests (all input/output type combinations, signed and out-of-range inputs, clamping rules of floating point outputs) with a reference parametrized by the luma weights. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>

greptile-apps · 2026-10-28T10:06:27Z

+  if (image_type_ == DALI_RGB && codec_params_->color_space == AVCOL_SPC_UNSPECIFIED)
+    avformat_find_stream_info(ctx_, nullptr);  // best effort, the stream is usable without it


Reopening loses the matrix

[Bug] Probing updates only the current codec_params_. BuildIndex() ends with Reset(), and NumFrames() also resets files whose frames must be counted. Reopening replaces those parameters with fresh, unprobed values.

For BT.709 streams whose matrix is only in the SPS, IsBT709() then sees AVCOL_SPC_UNSPECIFIED when decoding starts. The decoder selects BT.601 and produces wrong colors. Save the probed matrix separately, or probe again after reopening.

Implement rgb_to_y/cb/cr, rgb_to_ycbcr and ycbcr_to_rgb once, in detail::ycbcr_method, from a matrix (the scaled coefficients) and a range (limited or full, with its gray conversions). itu_r_bt_601, itu_r_bt_709, jpeg and itu_r_bt_709_full_range are now one-line instantiations. The results of the existing methods are unchanged. itu_r_bt_709_full_range gets the forward conversion and the gray conversions too, and the tests cover it with the other methods; the separate YCbCr -> RGB reference test is dropped, as the typed tests already compare against the reference with the Kr, Kb of every method. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>

State the RGB <-> YCbCr formulas once, in a comment, and compute the coefficients of every method from the luma weights Kr, Kb of the standard (BT.601, BT.709) and the scales of the range (limited or full), instead of listing the scaled constants of four matrices. The results are bit-identical to the previous constants. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>

Move the coefficients into detail::ycbcr_coefficients so the conversion structs only expose the conversion functions, share the clamp/dot/offset/round code of rgb_to_y, rgb_to_cb and rgb_to_cr, use the range's y_bias in the gray conversions, note the floating point offsets in the formulas comment and add pinned values for itu_r_bt_709_full_range. The results are unchanged. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>

Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>

…m uses it The GPU video decoders always converted YCbCr to RGB with the ITU-R BT.601 matrix, while HD video is normally BT.709, which gives visibly wrong colors for saturated colors (e.g. pure red 255,0,0 came out as ~234,0,3). Use the BT.709 matrix, in limited or full range, when the matrix that FFmpeg reads from the stream is BT.709. Streams that don't specify it keep using BT.601. FFmpeg's value is used instead of NVDEC's: NVDEC doesn't report the color description of some codecs (e.g. VP9) and reports the codec's default (BT.709 for MPEG-4) for streams that don't have one. The legacy fn.readers.video is not changed. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>

…y BT.709 FFmpeg doesn't read the color description that is only in the H.264 stream (SPS) when opening an MP4 without a 'colr' atom, so such BT.709 streams were still converted with BT.601. Probe the stream when converting to RGB and the matrix is not known. Test the full range clip and the clip with the description only in the stream. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>

The test pattern in DALI_extra has 10 bars now (black and white were added): use their width in the helper that compares the bars, and check that black and white are mapped to the ends of the range (the luma is not left at 16 and 235). Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>

The color matrix tests read db/video/color_matrix, which DALI_extra#137 adds. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>

Copilot AI balanced review requested due to automatic review settings October 28, 2026 08:34

jantonguirao mentioned this pull request Oct 28, 2026

Convert video to RGB on the CPU with the matrix of the stream #6569

Open

5 of 17 tasks

Copilot started reviewing on behalf of jantonguirao October 28, 2026 08:34

View session

Copilot AI reviewed Oct 28, 2026

View reviewed changes

greptile-apps Bot reviewed Oct 28, 2026

View reviewed changes

Comment thread dali/operators/video/frames_decoder_gpu.cc Outdated

Comment thread dali/test/python/decoder/test_video.py Outdated

Comment thread dali/test/python/decoder/test_video.py Outdated

greptile-apps Bot reviewed Oct 28, 2026

View reviewed changes

jantonguirao added 4 commits October 28, 2026 11:15

Spell out the reference parameters of the BT.709 methods in the test

896ebd4

Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>

dali-automaton assigned rostan-t and mdabek-nvidia Oct 28, 2026

jantonguirao added 2 commits October 28, 2026 12:12

jantonguirao force-pushed the janton/dali-4954-gpu-bt709 branch from 2e6bf4c to ffd4411 Compare October 28, 2026 11:12

jantonguirao marked this pull request as draft October 28, 2026 11:12

jantonguirao added 2 commits October 28, 2026 15:27

Update DALI_extra version for the BT.709 video clips

37e1d3d

The color matrix tests read db/video/color_matrix, which DALI_extra#137 adds. Signed-off-by: Joaquin Anton Guirao <janton@nvidia.com>

Navigation Menu

Repository navigation

Convert video to RGB on the GPU with the BT.709 matrix when the stream uses it - #6568

Convert video to RGB on the GPU with the BT.709 matrix when the stream uses it#6568
jantonguirao wants to merge 10 commits into
NVIDIA:mainfrom
jantonguirao:janton/dali-4954-gpu-bt709

jantonguirao commented Oct 28, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Oct 28, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

greptile-apps Bot commented Oct 28, 2026 •

edited

Loading

Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot Oct 28, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		if (image_type_ == DALI_RGB && codec_params_->color_space == AVCOL_SPC_UNSPECIFIED)
		avformat_find_stream_info(ctx_, nullptr); // best effort, the stream is usable without it

Conversation

jantonguirao commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Category:

Description:

Additional information:

Affected modules and functionalities:

Key points relevant for the review:

Tests:

Checklist

Documentation

DALI team only

Requirements

Uh oh!

copy-pr-bot Bot commented Oct 28, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

greptile-apps Bot commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 4/5

Findings

Summary

Diagram

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jantonguirao commented Oct 28, 2026 •

edited

Loading

greptile-apps Bot commented Oct 28, 2026 •

edited

Loading