Repository navigation
Fix 0-d tensors, dynamic output sizes and integer reductions, and default to float32 on CPU, in the OpenVINO backend - #24377
Open
pytorchbot wants to merge 34 commits into
Open
pytorchbot wants to merge 34 commits into
pytorchbot wants to merge 34 commits into
Conversation
…and in exir (#24222) Fixes found while exporting models to ExecuTorch backends. ## Fixes - **exir:** `constant_prop_pass` keeps tensors that in-place ops write, also through views or in `no_grad`/`autocast` blocks. `to_backend` keeps the output node last. `to_edge` lowers `torch._check_tensor_all` and `torch.sym_ite`. `GroupBasedPartitioner` no longer makes partitions that depend on each other, unless caller node groups already do. Tensor constants get static shapes; deduplicated constants keep their data. - **AOTI (AOTInductor) and CUDA:** the missing-kernel check catches scatter and index_put fallbacks. `detach` lowers, also inside `torch.cond`. Strided outputs are copied. Transposed convolutions (1-D, 2-D, no dilation) and `torch._check_tensor_all` lower. - **XNNPACK:** the quantizer skips bfloat16 tensors, and `cat` respects scoped configs. ELU is delegated when XNNPACK can run it. - **Qualcomm:** autocast regions are inlined correctly. Unquantized bfloat16 nodes, data-dependent shapes and HTP partitions without runtime inputs stay on the CPU. Fixes for stacks, 0-d selects, RMSNorm and int64 constants. - **OpenVINO:** model data stays alive while OpenVINO 2026.4 reads it. Partitions are converted with their input types. - **Core ML:** bfloat16 ops, and ops above rank 5 outside static layout chains, stay on the CPU. - **MLX, Arm, Cortex-M, Vulkan, NXP, portable:** dtype checks for MLX tensor lists, one constant slot per weight, slice updates on a constant base; TOSA keeps unused delegate inputs for VGF; `MatmulToBmmPass` takes a constant operand; Vulkan constants above 4 dims; non-strict programs in the NXP quantizer; `cummin.out`. ## Behavior changes - A CUDA `.pte` loads only the weights it was exported with. A `.ptd` with other weights, for example after fine-tuning, needs a newly exported `.pte`. - CUDA graphs turn off when outputs do not match the ETensor layout. If only the capture call sees such a layout, that call returns an error. - `GroupBasedPartitioner` is about 2x slower on large graphs, and some graphs get a different number of partitions. `to_backend` keeps the original order of top-level nodes. - XNNPACK: weight-only quantized linears, and 8da4w or 8da8w linears (8-bit dynamic activations, 4-bit or 8-bit weights) lowered with `to_backend`, fail at export instead of giving wrong results. Delegated ELU flushes subnormal values to zero. Cats outside a scoped quantizer config are no longer quantized. - Ethos-U raises an error for a delegate with unused inputs instead of returning wrong values. We would like to cherry-pick this PR to release/1.6 after it merges. It needs #23915 there first. ## Testing Each fix has a test that fails without it. Run on an A100 (CUDA, Vulkan), an x86 CPU (OpenVINO, QNN SDK), macOS (Core ML, MLX), the TOSA reference model and VGF emulation (Arm), and the eIQ Neutron SDK (NXP). Not run: Metal, ROCm, Windows CUDA. (cherry picked from commit 054de1f)
This branch was picked before #24263 and #24295 reached release/1.6, so three files carried their content only in part. Take release/1.6 and use the versions a clean pick of 054de1f gives for backends/qualcomm/partition/qnn_partitioner.py, backends/qualcomm/tests/test_passes.py and exir/tests/test_pass_infra.py.
release/1.6 regenerated aoti_cuda_shims.lib to add aoti_torch_dtype_element_size, and this branch regenerated it to add aoti_torch_as_strided. Regenerate it per aoti_cuda_shims.lib.md with both names, so it lists exactly the shims this tree exports.
…ault to float32 on CPU, in the OpenVINO backend (#24294) ## What is wrong today The OpenVINO backend has these problems: 1. A 0-d (scalar) input or output fails on every run, because OpenVINO's `ov_shape_create` refuses rank 0. 2. A float32 program does not compute in float32 on many CPUs. OpenVINO picks bfloat16 (two to three significant digits) on x86 CPUs with bfloat16 hardware and float16 on ARM CPUs, and no compile spec can change it. 3. An output whose size depends on an int input keeps its largest size. `x[:n].view(n, -1)` with x of shape (4, 8) and n = 3 returns (4, 8). 4. `cumsum` and `prod` of a bool or integer tensor are not computed in int64, as PyTorch does. A cumsum over a bool mask wraps after 255, so pi0 and pi0.5 get wrong position ids and wrong actions. ## What this change does - **0-d tensors:** a rank-0 tensor keeps its empty shape, which OpenVINO accepts. - **Precision:** when the device is `CPU`, a program computes in float32 by default, both when it is lowered and when it is loaded. A program can ask for a faster precision: ```python OpenvinoPartitioner( [CompileSpec("device", b"CPU"), CompileSpec("inference_precision", b"bf16")] ) ``` The value is `f32`, `f16` or `bf16`, checked before conversion starts. Other devices keep OpenVINO's default. On x86 CPUs OpenVINO applies the precision when the runtime loads the program, so the runtime passes it on too. - **Dynamic output sizes:** an output whose size can change goes to a buffer that OpenVINO owns, reused by every call. It starts empty and OpenVINO grows it to the largest size a call writes, so a large upper bound costs nothing until a call needs it. After inference the backend resizes the ExecuTorch output to OpenVINO's shape and copies the data in, so one delegate can also serve several calls, as programs from ExecuTorch 0.7 and earlier do. Static outputs still use the ExecuTorch buffer directly. - **Integer reductions:** the input of `cumsum` and `prod` is cast to int64 before conversion. An integer cumsum that reads only constants is computed during lowering, because OpenVINO's CPU plugin gives it wrong results from the second inference on. This includes a cumsum over `ones_like` or `zeros_like` of an input with a static shape, which OpenVINO folds into a constant too. Each one is folded on its own, and one that cannot be folded is named in a warning. Trade-offs: - float32 is slower. On a Xeon with bfloat16 hardware, a 4-layer MLP takes 2.4 ms in float32 and 1.5 ms in bfloat16. On x86 this also applies to programs exported before this change, because OpenVINO applies the precision when a program loads. To get the old speed, export again with `inference_precision` set to `bf16`. - OpenVINO's CPU plugin computes int64 in 32 bits, so an integer result past the int32 range wraps without an error. That was already true for int64 programs, and now also applies to `cumsum` and `prod` over smaller integer types. ## Tests Each new test fails without its fix: 0-d inputs and outputs; a 256 x 256 addmm checked to 1e-4, and the same addmm lowered with `bf16` checked not to reach 1e-4, so the runtime must apply an explicit precision (both skipped where OpenVINO already computes in float32); an unknown precision refused before conversion; dynamic output sizes, on one program and on one delegate shared by two calls; bool and int32 cumsum and prod; pi0-style masks with int64 and int32 cumsums over two calls, built from a constant and from `ones_like`; and folding when one cumsum cannot be evaluated. Two guards pass either way: integer `sum`, which OpenVINO already computes in int64, and a static output after a dynamic view. The full OpenVINO op suite passes on Linux x86_64 (a CPU with bfloat16 units) with OpenVINO 2026.4.1, through both pytest and the CI test runner. With OpenVINO 2025.1.0 it also passes, except that the prod test is skipped because that release does not delegate prod. (cherry picked from commit 7b14148)
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24377
Note: Links to docs will display an error until the docs builds have been completed. ❌ 2 New Failures, 1 Cancelled Job, 1 Unclassified FailureAs of commit f2cf062 with merge base 4db778e ( NEW FAILURES - The following jobs have failed:
UNCLASSIFIED FAILURE - DrCI could not classify the following job because the workflow did not run on the merge base. The failure may be pre-existing on trunk or introduced by this PR:
CANCELLED JOB - The following job was cancelled. Please retry:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
…ault to float32 on CPU, in the OpenVINO backend (#24294) ## What is wrong today The OpenVINO backend has these problems: 1. A 0-d (scalar) input or output fails on every run, because OpenVINO's `ov_shape_create` refuses rank 0. 2. A float32 program does not compute in float32 on many CPUs. OpenVINO picks bfloat16 (two to three significant digits) on x86 CPUs with bfloat16 hardware and float16 on ARM CPUs, and no compile spec can change it. 3. An output whose size depends on an int input keeps its largest size. `x[:n].view(n, -1)` with x of shape (4, 8) and n = 3 returns (4, 8). 4. `cumsum` and `prod` of a bool or integer tensor are not computed in int64, as PyTorch does. A cumsum over a bool mask wraps after 255, so pi0 and pi0.5 get wrong position ids and wrong actions. ## What this change does - **0-d tensors:** a rank-0 tensor keeps its empty shape, which OpenVINO accepts. - **Precision:** when the device is `CPU`, a program computes in float32 by default, both when it is lowered and when it is loaded. A program can ask for a faster precision: ```python OpenvinoPartitioner( [CompileSpec("device", b"CPU"), CompileSpec("inference_precision", b"bf16")] ) ``` The value is `f32`, `f16` or `bf16`, checked before conversion starts. Other devices keep OpenVINO's default. On x86 CPUs OpenVINO applies the precision when the runtime loads the program, so the runtime passes it on too. - **Dynamic output sizes:** an output whose size can change goes to a buffer that OpenVINO owns, reused by every call. It starts empty and OpenVINO grows it to the largest size a call writes, so a large upper bound costs nothing until a call needs it. After inference the backend resizes the ExecuTorch output to OpenVINO's shape and copies the data in, so one delegate can also serve several calls, as programs from ExecuTorch 0.7 and earlier do. Static outputs still use the ExecuTorch buffer directly. - **Integer reductions:** the input of `cumsum` and `prod` is cast to int64 before conversion. An integer cumsum that reads only constants is computed during lowering, because OpenVINO's CPU plugin gives it wrong results from the second inference on. This includes a cumsum over `ones_like` or `zeros_like` of an input with a static shape, which OpenVINO folds into a constant too. Each one is folded on its own, and one that cannot be folded is named in a warning. Trade-offs: - float32 is slower. On a Xeon with bfloat16 hardware, a 4-layer MLP takes 2.4 ms in float32 and 1.5 ms in bfloat16. On x86 this also applies to programs exported before this change, because OpenVINO applies the precision when a program loads. To get the old speed, export again with `inference_precision` set to `bf16`. - OpenVINO's CPU plugin computes int64 in 32 bits, so an integer result past the int32 range wraps without an error. That was already true for int64 programs, and now also applies to `cumsum` and `prod` over smaller integer types. ## Tests Each new test fails without its fix: 0-d inputs and outputs; a 256 x 256 addmm checked to 1e-4, and the same addmm lowered with `bf16` checked not to reach 1e-4, so the runtime must apply an explicit precision (both skipped where OpenVINO already computes in float32); an unknown precision refused before conversion; dynamic output sizes, on one program and on one delegate shared by two calls; bool and int32 cumsum and prod; pi0-style masks with int64 and int32 cumsums over two calls, built from a constant and from `ones_like`; and folding when one cumsum cannot be evaluated. Two guards pass either way: integer `sum`, which OpenVINO already computes in int64, and a static output after a dynamic view. The full OpenVINO op suite passes on Linux x86_64 (a CPU with bfloat16 units) with OpenVINO 2026.4.1, through both pytest and the CI test runner. With OpenVINO 2025.1.0 it also passes, except that the prod test is skipped because that release does not delegate prod. (cherry picked from commit 7b14148)
The pick was made on release/1.6, which does not have #24298 (the #24222 pick) yet. #24294 builds on #24222, so the pick depends on code that release/1.6 lacks, such as _nodes_written_in_place in exir's constant_prop_pass, and its conflict resolution brought in some of #24222's OpenVINO changes but not others. This merges a clean pick of 7b14148 onto #24298's branch and takes the clean pick's side in the OpenVINO files, so that, on top of #24298, the change is exactly #24294 (7b14148). A plain merge of the two would keep two copies of _PRECISIONS in preprocess.py.
The OpenVINO CI job runs backends/openvino/tests/test_runner.py. When a test failed, the runner printed "completed with failures" but still exited with code 0, so the CI job stayed green and the failure went unnoticed. Now the runner exits with code 1 when any test fails, so CI turns red. This is the test runner fix from #24305. Co-authored-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
…_precision The check that the runtime takes the last inference_precision entry ran only on x86 CPUs with bfloat16 units, so CI could skip it. A new test gives an invalid value and then an empty one. Lowering keeps only the empty one, and the runtime has to as well, or it fails to load the program. On the runtime before that fix it fails with "unsupported inference_precision 'f7'".
…ioner
OpenVINO converts min, max, amin, amax, minimum, maximum, clamp and sign
to its ReduceMin, ReduceMax, Minimum, Maximum and Sign operations, and
none of those accept bool tensors. The partitioner still gave these ops
to OpenVINO when they read bool tensors, so the whole program failed to
lower:
ReduceMin ... Element type of data input must be numeric. Got: boolean
With PyTorch 2.15, MultivariateNormal exports two such mins when it
checks scale_tril, so a model that builds one could not use OpenVINO.
The partitioner now leaves these ops to the CPU when every tensor they
read is bool, and the portable kernels compute them as PyTorch does. An
op with a bool and a float input, such as maximum(mask, x), is computed
in float, which OpenVINO converts, so it stays with OpenVINO. Other
dtypes are not affected.
Lowering passes the precision too, so the comment was wrong to say only the runtime does. What makes the check depend on the runtime is that on x86 OpenVINO applies only the precision it gets when it loads the model.
The cumsum fold lines had landed between the comment and the decoder call it explains, so the comment read as if it were about the fold.
Since each cumsum is folded on its own, _evaluate is only ever asked for one node. Taking and returning that one value drops the list and dict it no longer needs, and a check that could never be false.
Only the attention mask test covered an int32 cumsum, and only through an OpenVINO CPU bug that newer OpenVINO releases fix. A fold limited to int64 now fails a pass-level test too.
Only x86 CPUs with bfloat16 units computed in bfloat16 before, so only they get slower, and only there does asking for bf16 bring the earlier speed back. On other x86 CPUs OpenVINO already computed in float32.
The precision docs said int64 results only go wrong outside the int32 range. Measured through the delegate on a CPU, an int64 sum is computed in float32: 16777216 + 1 gives 16777216, and 123456789 + 987654321 gives 1111111168. cumsum and prod near 2^24 stay exact.
Only an output compiled with a fully fixed shape refuses a buffer of another shape. A partly dynamic output accepts an empty one. Comment only.
OpenVINO 2025.1 has the same CPU plugin bug with integer cumsums over constants as the later releases, so the fold's docstring now starts the range there. Docstring only.
The OpenVINO CI job runs the op tests and then the model tests under set -e. Now that the test runner exits with code 1 when a test fails, a failing op test stopped the job before the model tests started, so a red run said nothing about the models. The script now runs both suites and exits with code 1 if either failed.
Preprocess put the first partition input back whenever the converted model had one parameter fewer than the partition has inputs. Now it does so only when that first input is really unused, and lowering fails when the counts still differ. The runtime binds inputs by position, so another mismatch would have made every call fail, or swapped two inputs of the same type and shape. A partition can also have no inputs at all, such as a comparison that reads only a buffer next to a bool min on the CPU, and a test covers that case.
TestBoolMinMax compared only the values of min(dim) and max(dim). Their indices are where the order of ties matters, and the all-true and all-false rows are all ties, so the test now compares the indices with eager too. Its docstring also says more exactly why these ops leave OpenVINO.
x was int64 in both subtests, so the result was int64 whatever dtype the folded cumsum had, and a fold that dropped dtype=torch.int32 still passed. x now has the subtest's dtype, so such a fold fails the int32 subtest.
The docs said only sum reduces integers in float32. Measured through the delegate on a CPU with OpenVINO 2026.4.1 and 2025.1.0, amax, amin, max and min of int32 and int64 tensors do too: amax over a tensor that holds 16,777,217 gives 16,777,216. Inputs and running totals round as well, so the sum of 16,777,217 and -16,777,216 gives 0. cumsum and maximum stay exact, and so does prod where OpenVINO computes it (2026.4.1).
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is wrong today
The OpenVINO backend has these problems:
ov_shape_createrefuses rank 0.x[:n].view(n, -1)with x of shape (4, 8) and n = 3 returns (4, 8).cumsumandprodof a bool or integer tensor are not computed in int64, as PyTorch does. A cumsum over a bool mask wraps after 255, so pi0 and pi0.5 get wrong position ids and wrong actions.What this change does
0-d tensors: a rank-0 tensor keeps its empty shape, which OpenVINO accepts.
Precision: when the device is
CPU, a program computes in float32 by default, both when it is lowered and when it is loaded. A program can ask for a faster precision:The value is
f32,f16orbf16, checked before conversion starts. Other devices keep OpenVINO's default. On x86 CPUs OpenVINO applies the precision when the runtime loads the program, so the runtime passes it on too.Dynamic output sizes: an output whose size can change goes to a buffer that OpenVINO owns, reused by every call. It starts empty along each dynamic dimension, and OpenVINO grows it to the largest size a call writes, so a large upper bound costs nothing until a call needs it. After inference the backend resizes the ExecuTorch output to OpenVINO's shape and copies the data in, so one delegate can also serve several calls, as programs from ExecuTorch 0.7 and earlier do. Static outputs still use the ExecuTorch buffer directly.
Integer reductions: the input of
cumsumandprodis cast to int64 before conversion. An integer cumsum that reads only constants is computed during lowering, because OpenVINO's CPU plugin gives it wrong results from the second inference on. This includes a cumsum overones_likeorzeros_likeof an input with a static shape, which OpenVINO folds into a constant too. Each one is folded on its own, and one that cannot be folded is named in a warning.Trade-offs:
inference_precisionset tobf16.cumsumandprodover smaller integer types.Tests
Each new test fails without its fix: 0-d inputs and outputs; a 256 x 256 addmm checked to 1e-4, and the same addmm lowered with
bf16checked not to reach 1e-4, so the runtime must apply an explicit precision (both skipped where OpenVINO already computes in float32); an unknown precision refused before conversion; dynamic output sizes, on one program and on one delegate shared by two calls; bool and int32 cumsum, and int32 prod; pi0-style masks with int64 and int32 cumsums over two calls, built from a constant and fromones_like; and folding when one cumsum cannot be evaluated. The tests for integersum, which OpenVINO already counts past 255, and for a static output after a dynamic view are guards that pass either way.The full OpenVINO op suite passes on Linux x86_64 (a CPU with bfloat16 units) with OpenVINO 2026.4.1, through both pytest and the CI test runner. With OpenVINO 2025.1.0 it also passes, except that the prod test is skipped because that release does not delegate prod.
Follow-up fix from #24305
This pick also carries the test runner fix from #24305. When a test failed, the OpenVINO test runner printed "completed with failures" but still exited with code 0, so the CI job stayed green. It now exits with code 1 when a test fails, so the job fails.
Follow-up fixes from #24396
This pick also carries these fixes from #24396 for code that #24294 added:
cumsumwas the only reader of the first partition input, OpenVINO's converter dropped that input, and every call failed. Lowering now puts it back. If the converted model still takes a different number of inputs, lowering now fails with a clear error.with torch.device("cuda"):, thecumsumfold put a temporary tensor on the default device, so lowering failed on a CPU-only PyTorch. It now makes that tensor on the CPU.inference_precisionwas given twice and the last value was empty, the runtime used the earlier value while lowering used the default. The runtime now takes the last value too.It also carries a fix for an older bug.
min,max,amin,amax,minimum,maximum,clampandsignthat read only bool tensors made the whole program fail to lower, because the OpenVINO operations they become do not take bool tensors. The partitioner now leaves them to the CPU. With PyTorch 2.15,MultivariateNormalexports such aminwhen it checksscale_tril.The OpenVINO CI script now runs the model tests even when an op test fails, and still fails the job.
The matching tests, comment and doc fixes, and small cleanups of the
cumsumfold and the new output buffer code come from the same PR. With all of them, the OpenVINO op and model tests pass on Linux x86_64 with OpenVINO 2026.4.1, and with 2025.1.0 apart from the skipped prod test.Merge order
This branch is stacked on #24298, the pick of #24222, and contains all of its commits. Merge #24298 first: merging this PR first would land #24298's changes too. #24396 is still open on main, and it should land there before this PR merges.