feature: Capture video from a USB grabber on the ESP32-P4

[Gohnnyman]

What's new

• USB capture - read USB video input as UVC webcam, MJPEG decoded by the P4's JPEG hardware. hdr declares the source curve (PQ / HLG), since MJPEG carries no metadata. Gated on the chip and both SoC capabilities; every other target links a failing stub.
• White balance — balanceRed/Green/Blue trim a die down to correct a cast; whiteLevel trims the white die, which the RGB trims cannot reach. Folded into the same table fill as the curve, so the hot path stays one lookup.

Built for esp32p4rev3-eth-wifi, which links the UVC driver, and esp32s3-n16r8, which links stubs and no component. Verified on a P4: capture, decode and the trims all live.

Note

This PR is a part of #138 and #140
This PR Stacked on #138 , so that PR should be merged first

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: Repository: MoonModules/MoonLight/.coderabbit.yaml
• Review profile: ASSERTIVE
• Plan: Advanced
• Run ID: 951193f3-7331-44bf-81ab-f9235c5c0a23

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[ewowi]

This fits the plan (linked in #138) as a picture source: a USB transport plus the P4's MJPEG decoder. Like #138, it stays open while I research the plan further; comments on the plan are very welcome.

(As in my comment on #138: one topic per PR, roughly 400 lines and ten files, so each can be reviewed properly.)

The color trim ("Trim each LED die…") is a separate idea from capture: output calibration, touching every board's LED output. I'm considering it on its own as well, and haven't decided on it yet. Two notes for when we get there: the brightness table is rebuilt 1280 times instead of 256 every 20 ms during the 5 s soft start, which filling the neutral row once and copying it would avoid; and the textbook form is a 3×3 correction matrix rather than three gains.

Please hold off on updating this PR until I've done the groundwork; then we'll look together at where the capture fits, so no effort goes into a shape that changes. For reference only, the review points so far on the capture part:

• the Teensy 4.1 stubs (hasUsbVideo = false and the six videoCapture* functions, like the desktop's), since the Teensy merged meanwhile;
• no 3 s freeze when no grabber is attached (skip the stream open until a device has enumerated);
• the three USB tasks pinned to core 1;
• the UVC payload header length derived from its flags (UVC 1.5 §2.4.3.3), so a grabber without timestamps works;
• the flash cost measured, and capture linked only into the P4 images with room, as the SD card was done;
• mm::formatTo instead of snprintf under MM_NONBLOCKING.

Good work on the capture seam itself: the opaque handle, the generation counter and formats learned at enumeration are exactly the shape we want.

[ewowi]

One more note for future PRs: please branch each one from current main rather than from your previous PR's branch. #139 and #140 each carry the commits of the PRs before them (#140 shows 34 files for its own ~160 lines), so a review can't see one topic on its own, and a change in an earlier PR ripples into every later one. Only when one topic needs several steps should a step build on the one below it; it then gets reviewed once that one merges.

[ewowi]

The groundwork for USB capture is now on main (#141), so here is what to build on.

What is set up

• PictureService is the picture source, and each source is an option of its source setting. A source that runs in its own task, as a capture does, writes like this:
  • beginWrite() hands it the back picture, and endWrite() swaps it to the front, so effects never see half a picture.
  • Short of memory, the service falls back to one picture locked while written, and says so in its status.
• PictureEffect samples whatever the service publishes onto any layout.

What to do
A new PR on current main that adds USB capture as a source:

• the option in Source, marked as running in its own task in kRunsInTask;
• the capture task decoding each MJPEG frame into beginWrite(), then endWrite();
• the UVC transport and the MJPEG decoder behind a platform seam in src/platform, with stubs on the desktop and the Teensy.

The review points from my earlier comment still apply: no 3 s freeze without a grabber, the USB tasks on core 1, the UVC payload header length from its flags, the flash cost measured, and mm::formatTo under MM_NONBLOCKING.

This PR can then close. The color trim is a separate idea and a separate PR whenever you want to take it up.

Where to read

• How to add a picture or channel-data source: the steps, the write contract and what a PR brings.
• The Streams plan: G3 and the "USB capture" row under How contributions arrive.

For the PR: one topic, based on current main with only its own commits, max 400 changed lines and ten files (the rules).

Your capture seam was the right shape from the start, and it now has a place to land.

[ewowi]

Thanks @Gohnnyman, closing this one: its capture half landed as the groundwork in #151, and the USB transport follows in the UVC PR you announced there. The color trims (balanceRed/Green/Blue, whiteLevel) are a separate topic, output calibration for every board, so they'd come back as a PR of their own if we take them up; I've noted them in our backlog.