Repository navigation
Support 3-D convolution in the portable convolution kernel - #24385
shoumikhin wants to merge 25 commits into
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24385
Note: Links to docs will display an error until the docs builds have been completed. ⏳ No Failures, 189 PendingAs of commit 33241ab with merge base e68a824 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
@claude /pr-review |
|
Claude finished @shoumikhin's task in 32m 34s —— View job PR Review: #24385Needs Discussion: the PR removes two helpers from a public util target, multiplies by Detailed reviewSummaryThe PR sends 1-D, 2-D and the new 3-D convolutions through a single 3-D implementation in the portable Code Quality
Testing
Backward Compatibility
Performance
RecommendationNeeds Discussion The reviewers found no blocking bug in the kernel. Three decisions are needed before merge:
The
|
|
Thanks for the review. Pushed:
|
|
@claude /pr-review |
|
Claude finished @shoumikhin's task in 35m 37s —— View job PR Review: #24385Approve once #24380 lands: I found no bugs in the new 3-D loops, and the remaining items don't block: an overflow that gets past the new kernel-size check, an unchecked bias dtype, and two tests that check less than they appear to. Detailed reviewSummaryThis PR sends 1-D, 2-D and new 3-D convolutions in portable
Code Quality
Testing
RecommendationApprove. Merge only after #24380 lands on main, or together with it. This branch is based on Review steps
Not done:
|
There was a problem hiding this comment.
🟡 Changes recommended
Negative transposed-convolution output sizes can narrow to valid depths and silently produce incorrect results, while tensors without stored strides can trigger invalid memory access or crashes.
2 open issues
Review effort: Balanced
| for (const auto d : c10::irange(t.dim())) { | ||
| const ssize_t view_d = d < 2 ? d : d + added_dims; | ||
| sizes[view_d] = t.size(d); | ||
| strides[view_d] = t.strides()[d]; |
There was a problem hiding this comment.
Thanks. Tensors are required to have strides: the TensorImpl constructor documents that strides must contain one entry per dim. The program loader and make_tensor_ptr both fill them in. Many portable kernels already read strides directly, including the shared broadcast and reduction helpers, so a tensor without strides would fail in those too. So I kept this kernel reading strides directly, like those kernels do.
There was a problem hiding this comment.
🔵 Needs a closer look
Maintainers must validate the internal jobs that caused the prior Conv3d revert and coordinate landing with still-open #24380 to prevent incorrect quantized XNNPACK outputs.
1 resolved since last review
Review effort: Balanced
There was a problem hiding this comment.
🔵 Needs a closer look
Internal-job validation remains unavailable after the prior Conv3d regression-driven revert, and merge ordering with #24380 must be coordinated to prevent silent quantized XNNPACK output corruption.
Review effort: Balanced
The conv3d case ran a 1x1x1 kernel with constant weights on one channel, so its output was just twice the input. It could not catch an indexing mistake, and with one channel its channels-last input had the same layout as a contiguous one. It now has two input and four output channels, a 2x3x3 kernel, stride (1, 2, 2), padding 1 and the default random weights, on a channels-last input.
PyTorch's CPU conv3d returns a contiguous output even for a channels_last_3d input, so the exported program gave the convolution a channels-last input and a contiguous output. The portable kernel needs both in the same layout, so the run on the Arm simulator failed. With one channel the two layouts were the same, which is why the old case passed. The C++ tests cover the kernel's channels-last 3-D path.
The 32-bit bound on stride, padding and dilation left two gaps: - An output size above 2^31 - 1 still fit int64_t, but was then cut to a 32-bit tensor size. Padding 2^31 - 1 on an input of 3 with a kernel of 1 gives 2^32 + 1, which became 1, so a call that should fail ran. - An empty spatial dim made the transposed output size math compute 0 - 1 in size_t. On 32-bit targets that overflowed int64_t. PyTorch rejects empty spatial dims, and now check_convolution_args does too. It also computes each output size and rejects one that does not fit in the 32-bit tensor size. The tests cover both.
A transposed convolution with a large padding can have a negative output size. For example, an input of 3, a kernel of 1 and padding 2^31 - 1 give 5 - 2^32. The output size check only had an upper bound, and the size became a positive 5 when cut to 32 bits, so the call ran. PyTorch rejects it as too small. Now each output size must be at least 1. The test now covers that case instead of a stride above the 32-bit bound, which the output size check catches too, so it is renamed to HugePaddingAndDilation.
With a kernel of 3, dilation 2^62 gave a negative output size once the bound was removed, so the new check on output sizes rejected the call too, and the test passed without the bound. With a kernel of 5 the dilated kernel size wraps to exactly 1 and the output size to a valid 3, so only the bound rejects the call.
PyTorch's CPU conv3d returns a contiguous output even for a channels_last_3d input, so an exported Conv3d with a channels-last input gives the kernel a channels-last input and a contiguous output. The kernel rejected that pair, so such a model failed at run time. The kernel reads the input, weight and output through their own strides, and check_convolution_args already limits the input, the output and a non-transposed weight to the contiguous or the channels-last dim order, so the input and output do not need to match. Drop that check. Both 3-D tests now also run a channels-last input and weight with a contiguous output, and the Cortex-M conv3d test uses a channels_last_3d input again.
check_convolution_args read the kernel size from the weight but never checked it. With a kernel dim of 0, the dilated kernel size became 1 - dilation, so even the kernel size check passed, and an input of 3 gave an output of 4 that no kernel tap ever wrote. PyTorch rejects a kernel size of 0. Check it with kernel_size_is_valid, like the pooling checks do.
The new check for empty spatial dims also rejected inputs whose batch or channel dim is empty, which main and PyTorch accept and turn into an empty output. Allow those again, like PyTorch. The transposed output size math now subtracts 1 from the input size in int64_t instead of size_t, so an empty spatial dim cannot make it wrap on 32-bit targets.
With a channels_last_3d input the case now ran on the Arm simulator but gave wrong values. The Arm test runner writes an input in its logical order unless its dim order is NHWC or the runner's own 5-D order, so a channels_last_3d input reached the program in the wrong byte order. The kernel handles a channels-last input with a contiguous output, as the C++ 3-D tests show, so give the Cortex-M case a contiguous input, with a comment that says why.
The kernel size check fixes 1-D and 2-D convolutions as well, but its test used only a 5-D input, which main rejected at the rank check anyway. Add a 1-D case: an input of 2 with a kernel of 3 and stride 2, which main ran from two of the three kernel taps.
Say why the bias length check comes after the groups checks: they keep groups * weight.size(1) from overflowing. Before this, only a test comment said so. Also fix three comments that were hard to read or not quite right: the output size math can round a too-large kernel up to 1, but only when the kernel is too large by less than the stride; the range helper comment is reworded; and the depth loop comment now says the loop runs once for 1-D and 2-D inputs.
Three cases had no test, so the code that handles them could break without any test failing: - An empty spatial dim with an empty channel dim, which PyTorch accepts. Only the empty batch case was tested. - A transposed convolution with a huge stride or padding, where the 64-bit output size math would wrap around to a valid size. Only the dilation part of the 32-bit bound was tested. - A channels-last 5-D input with a contiguous weight and a contiguous output, which is what export gives when only the input is channels-last. The mixed layout calls used a channels-last weight.
The backend test suite skipped every model with a 5-D convolution, because the portable kernel could not run one. Now it can, so remove the skip, and the suite's Conv3d and ConvTranspose3d cases run in every flow.
caf610b to
33241ab
Compare
|
@claude /pr-review |
|
Claude finished @shoumikhin's task in 25m 40s —— View job Approve: the three reviewers found no bugs in the 3-D kernel, the argument checks or the tests, so what remains is to merge after #24380 and to check the Conv3d results of this head's backend test jobs, which are still running and pass even when cases fail. Detailed reviewSummaryThis PR runs 1-D, 2-D and 3-D convolutions in the portable RecommendationApprove
Review steps
Moved to the Recommendation
Dropped
Not done
|

What is wrong today
A model with
torch.nn.Conv3dlowers without an error but fails at run time on CPU, for example with XNNPACK:XNNPACK does not take 3-D or bfloat16 convolutions, so they run on the portable convolution kernel, which only supported 1-D and 2-D. This blocks vision encoders with a
Conv3dpatch embedding, such as Qwen2-VL and Qwen2.5-VL. It is the failure in #11704.What this change does
The portable convolution kernel now runs 3-D convolutions, regular and transposed. 1-D and 2-D convolutions run as 3-D ones with extra dims of size 1 and add up their terms in the same order as before, so their results stay the same (except that, with fused multiply-add, a result of -0 can now be +0, as in PyTorch). The input and output may now have different dim orders, because
torch.exportgives aConv3dwith a channels-last input a contiguous output. The backend test suite no longer skips 3-D convolutions.It also fixes older bugs in the argument checks, which affect 1-D and 2-D too:
groups = 0divided by zero, and a hugegroupscould overflow in the bias check.Testing
conv3dtest, now with several channels, a 2x3x3 kernel, stride and padding, no longer expects a failure and passes in CI on the Arm simulator.Conv3dpatch embedding lowered with XNNPACK, and random convolutions on the portable kernel, match eager PyTorch in float32 and bfloat16.-Os, and 18 percent faster to 11 percent slower with Clang at-O2and-O3. Transposed ones are 5 percent faster to 17 percent slower.op_convolution.ogrows from 32,484 to 33,263 bytes at-Osand from 48,477 to 61,457 at-O3.kernel_ops_util.o, which holds the argument checks, grows from 15,746 to 17,743 bytes at-Osand from 25,932 to 28,979 at-O3.Notes
Conv3dlowered with XNNPACK returns wrong values with no error, or fails to load, and the backend test suite's static int8 XNNPACKConv3dcases fail. Please merge Keep ops above 6 dims out of XNNPACK and fix quantized ops it leaves out #24380 first or together with this change.