Repository navigation
Tmp - #24226
Closed
Erik-Lundell wants to merge 25 commits into
Closed
Tmp#24226Erik-Lundell wants to merge 25 commits into
Erik-Lundell wants to merge 25 commits into
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24226
Note: Links to docs will display an error until the docs builds have been completed. ❌ 7 Pending, 2 Unclassified FailuresAs of commit a492daf with merge base a519331 ( UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
Erik-Lundell
requested review from
JacobSzwejbka,
larryliu0820 and
mergennachin
as code owners
October 28, 2026 08:06
meta-cla
Bot
added
the
CLA Signed
Erik-Lundell
requested review from
SS-JIA,
digantdesai and
psiddh
as code owners
October 28, 2026 08:06
Erik-Lundell
force-pushed
the
codex/arm-docgen-github-check
branch
from
October 28, 2026 08:19
87d3ad4 to
37e757d
Compare
…ule (pytorch#24127) ## What is wrong today A method can call the same lowered module several times, for example a denoising step that is lowered once and called 10 times in a loop. The emitter then writes one `BackendDelegate` per call. So at load time the runtime calls the backend's `init()` once per call and keeps every handle alive, even though all of them come from the same compiled blob. This wastes load time and memory. ## What this change does A backend can now return `PreprocessResult(..., share_handle_across_calls=True)`. Then every call of that lowered module in a method uses one `BackendDelegate`, so `init()` runs once. Each call still passes its own arguments. - It is off by default, so existing backends behave as before. Some backends keep argument pointers on the handle, so sharing is not safe for all of them. - A delegate that declares scratch memory is never shared, because scratch is planned per call and the runtime keeps it on the delegate. `Method::init` now rejects a program where such a delegate has more than one call site. It also checks every delegate index at load, not only at run time. - Only calls of one lowered module share a handle. Separate partitions keep separate delegates, even when their blobs are identical. - The flag is saved and loaded like `scratch_specs`, including through `exir.save` and `exir.load`, so the serde schema moves to version 5.6. The `.pte` file format does not change. - The demo test backend opts in. Before pytorch#14564, the emitter shared one delegate between calls with the same blob for every backend. Now it shares by lowered module, and only for backends that ask for it. ## What was tested - New tests cover one delegate for repeated calls with the same output as separate calls, the partitioner path, separate modules keeping their own delegates, save and load, the default, and scratch, including a hand-made program that the runtime rejects. The runtime test for scratch fails without the new runtime check. The load-time index check, calls inside control flow, the multi-method lowering path and the number of `init()` calls have no dedicated test. - On an NVIDIA Jetson, a TensorRT step that a method calls 10 times went from 10 delegates to 1 shared delegate: load time 435 ms to 74 ms, memory 1060 MiB to 284 MiB, with the same outputs and speed.
Pull Request resolved: pytorch#23411 Enable Gemma 4 E2B quantized embeddings to lower to ET-VK without losing per-row quantization metadata. **What**: Teach `embedding_q4gsw` to interpret one-dimensional scales as one scale per vocabulary row and validate supported scale layouts. **Why**: Gemma's token and per-layer embeddings use `[vocab]` scales. The reference implementation previously reshaped that tensor as one global scalar, which failed during Vulkan preprocessing. - Per-row scales - **Before**: Every one-dimensional scale tensor was reshaped to `[1,1,1]`. - **After**: `[vocab]` scales reshape to `[vocab,1,1]`. - Groupwise scales - **Before**: Shape compatibility was implicit. - **After**: `[vocab,groups]`, row counts, and group tiling are validated. - Runtime coverage now includes Gemma embedding widths 1536 and 8960 for both buffer and texture storage. Authored with Codex. ghstack-source-id: 457100300 @exported-using-ghexport ___ @overriding_review_checks_triggers_an_audit_and_retroactive_review Oncall Short Name: ai_infra_mobile_platform @landed-with-radar-review Differential Revision: [D122866874](https://our.internmc.facebook.com/intern/diff/D122866874/)
Pull Request resolved: pytorch#23412 ## Ulterior Motive Support Gemma 4 attention on ET-VK without changing its normalized-attention math or breaking YOCO cache sharing. ## Rationale **What**: Honor explicit SDPA scales and restrict cache-update fusion to cache updates owned exclusively by one donor attention operation. **Why**: Gemma uses scale `1.0`, while ET-VK previously forced `1/sqrt(head_dim)`. In addition, fusing an SDPA donor with later YOCO consumers created a private runtime cache and left the shared cache stale. ## Details - Attention scale - **Before**: Direct and fused LLM SDPA rejected or ignored an explicit scale. - **After**: Explicit scale values are forwarded to the QK shader; `None` retains the existing `1/sqrt(head_dim)` default. - Cache fusion - **Before**: A donor could fuse when another attention read its cache or when a cache update had incompatible operands, position, ordering, or live users. - **After**: Fusion requires distinct K/V caches, matching update operands and positions, updates before attention, dead update results, and exact cache user sets. Shared or ambiguous caches retain explicit update nodes. - Cache initialization - **Before**: Fused SDPA allocated empty private caches, discarding serialized or runtime prefix contents. - **After**: Runtime cache tensors pass through, while serialized cache data is prepacked before the fused update. - Runtime coverage includes Gemma D256/D512, real prefill geometry, prefix decode, GQA selectors, and explicit/default scales. Authored with Codex. ghstack-source-id: 457100302 @exported-using-ghexport ___ @overriding_review_checks_triggers_an_audit_and_retroactive_review Oncall Short Name: ai_infra_mobile_platform @landed-with-radar-review Differential Revision: [D122866875](https://our.internmc.facebook.com/intern/diff/D122866875/)
Pull Request resolved: pytorch#23414 ## Ulterior Motive Make Gemma 4 prefill and decode bottlenecks measurable with ETDump and Vulkan GPU timestamps. ## Rationale **What**: Add optional event-tracer plumbing to the Gemma runner and expand the Vulkan timestamp query pool for dispatch-heavy models. **Why**: The runner could not emit ETDump data. After adding a tracer, Gemma prefill exceeded the previous 4,096-query limit before completing. ## Details - `--etdump_path` creates an `ETDumpGen`, transfers ownership through `Gemma4Runner` into `Module`, and writes the dump after generation. - Tracer-disabled and speculative runners reject `--etdump_path` instead of silently producing no artifact. - iOS resolves ETDump output inside the app sandbox for absolute and relative flag paths. - Empty, partial, failed, or unclosed writes fail explicitly; dump buffers are released on every path. - Timestamp-query capacity increases from 4,096 to 65,536. Query pools remain disabled and unallocated in non-profiled builds. - The Gemma runner links ETDump support explicitly. Authored with Codex. ghstack-source-id: 457100314 @exported-using-ghexport ___ @overriding_review_checks_triggers_an_audit_and_retroactive_review Oncall Short Name: ai_infra_mobile_platform @landed-with-radar-review Differential Revision: [D122866878](https://our.internmc.facebook.com/intern/diff/D122866878/)
…pytorch#24088) ## What is wrong today When a delegate runs on a GPU, `PropagateDevicePass` copies its inputs to the device (`_h2d_copy`) and its outputs back (`_d2h_copy`). `skip_h2d_for_method_inputs` and `skip_d2h_for_method_outputs` remove those copies at the method's edges. 1. A transposed parameter is stored contiguous, but a delegate reading it is compiled for its original strides. `to_edge` does this silently. 2. When a host op writes a delegate output in place, the next delegate on that device reads the old value. Such a `relu_` makes memory planning fail. 3. A lowered module returning one tensor, called from another module, gets no `_d2h_copy`. 4. With the output skip flag, a returned output that something else reads comes back on the host. 5. A constant that feeds a device delegate makes `to_executorch` fail. With the input skip flag, it arrives as host memory marked as device memory. 6. With the input skip flag, an input with more than one reader is refused. 7. A delegate targeting the CPU gets copies that fail at run time. ## What this change does Numbered as above: 1. `to_edge` makes such a parameter contiguous before partitioning, and warns for each tensor whose layout it changes. 2. The host op writes the host copy of the output. The copies stay when it writes between two delegates, also through a view. 3. The output gets its `_d2h_copy`. 4. Returned outputs stay on the device. One that a host op writes in place is returned on the host, with a warning. 5. The constant gets a device copy, with a warning. One neither contiguous nor channels-last is refused. 6. The input stays on the device of its first delegate. Shape-only ops read it there. Other readers share one `_d2h_copy` of it, with a warning. An input written in place on the host is refused. 7. The delegate gets no copies. Also: - A tensor that several delegates on one device read is copied there once, unless written in between. - New option `PropagateDeviceConfig(error_on_per_call_copies=True)` turns the warnings in 4 to 6 into errors when the matching skip flag is set. ## What was tested - Device pass tests and new `exir/tests/test_passes.py` tests pass. All new tests but one fail without their fix. - `backends/cuda/tests/test_device_placement.py` matches eager PyTorch with real CUDA delegates on an H100. The `unittest-cuda` job now runs when these passes change, and fails if that test skips.
This is a draft because the page describes the intended end state of Torch-TensorRT 2.15. Some links go to sections of the Torch-TensorRT guide that exist only after pytorch/TensorRT#4927 merges and its docs site is rebuilt. And the examples rely on 2.15 changes from pytorch/TensorRT#4909, such as `torch_tensorrt.save` keeping the compiled graph by default. ## What is wrong today The TensorRT backend page has drifted away from Torch-TensorRT, the project that builds and ships the TensorRT delegate. - It points readers at a nightly build from early October. - It says CUDA graph replay never runs for engines on the shared activation scratch. Newer builds run it. - It repeats most of the Torch-TensorRT guide, so every delegate change is made twice. - The backends table lists TensorRT as Linux only. The CUDA backend page and the getting started page also disagree with the TensorRT page and with the code: on outputs kept on the GPU, on what happens when a planned input gets GPU memory, on CUDA versions, GPUs and Python versions, and on Windows. ## What this change does The TensorRT page keeps the usual backend sections, says less in each, and shows more code: - Features, target requirements, and development requirements with the one-line install command. - Export, including sending the operators TensorRT cannot compile to the CUDA backend, and keeping inputs and outputs on the GPU. - Runtime integration from Python, and a complete C++ project with the commands to build and run it. - A short list of performance options that link to the Torch-TensorRT guide. It describes Torch-TensorRT 2.15 and newer, and marks the two features that need a newer build. A second commit fixes the CUDA backend page, the getting started page and one docstring on the points above. To keep data on the GPU, all pages now leave inputs and outputs unplanned, which copies the least. ## What was tested On Linux x86-64 with an H100 GPU, an earlier version of the TensorRT page was run end to end: every Python block in order with nightly Torch-TensorRT, and the C++ example built and run against the installed packages. With nightly packages, the GPU input and output recipe was run from Python for both backends, and the CUDA page's C++ lines were built and run. A profiler showed one GPU copy fewer per Python call than with planned outputs. The other fixes were checked only by reading them against the sources. Windows was not tested.
Summary: Add separate Linux jobs for `runtime`, `ggml`, and `vulkan`, running each native test suite once and building `run_classification` in the Vulkan job. Scope PR triggers to relevant build inputs. Authored with AI assistance from Codex. Reviewed By: digantdesai Differential Revision: D125695225 Pull Request resolved: pytorch#23930
Summary: CPU partitioning and delegate I/O required every tensor to be FP32, leaving indices, masks and integer outputs outside the delegate. Accept static uint8, int8, int16, int32, int64 and bool tensors, and validate runtime I/O against the serialized dtype. Existing allocation and ET kernel routes preserve the values without conversion through FP32. Integer delegation also exposes `index.Tensor`/`index_put`, whose indices are `Tensor?[]`. Native serialization emits a plain tensor list when every index is present, but ET's kernels unbox a list of optional tensors. Mark those Reference routes with `optional_tensor_lists` so the CPU runtime binds optional lists for them; the shared serializer and the other Native engines are unchanged. Add exact runtime regressions for integer precision, indices, mixed outputs, constants and optional index lists. Dynamic shapes/scalars and quantized execution remain separate. The Native schema and storage contract do not change. Authored with AI assistance using Codex and Claude Code. Reviewed By: digantdesai Differential Revision: D123332329 Pull Request resolved: pytorch#24191
Summary: Adds `FuseBatchNormWithLinearPass` to `backends/transforms`, the linear counterpart of `FuseBatchNormWithConvPass`. It folds an inference batch norm into the preceding `aten.linear` weight and bias using `torch.nn.utils.fusion.fuse_linear_bn_weights`, so backends can drop the BN without serializing extra weights. Fusion only happens when it is numerically equivalent: the linear output must be rank 2 (the only case where BN's channel dim is linear's output dim), every weight/bias/running-stat input must be a constant parameter, BN must only be consumed through `getitem(0)`, and training-mode `native_batch_norm` is skipped. Authored with Claude Code. Reviewed By: digantdesai Differential Revision: D125532471 Pull Request resolved: pytorch#24192
…orch#24181) ## What is wrong today A constant tensor can be a view into a larger tensor. For example: ```python self.register_buffer("weight", torch.randn(3, 8, 8)[1]) # starts 64 floats into its storage ... return F.linear(x, self.weight) ``` Constant folding can also leave a weight as a view, like one chunk of the packed `in_proj_weight` of `nn.MultiheadAttention` in cross-attention. The XNNPACK and Vulkan serializers copy the bytes of the whole underlying storage, starting at its first byte. For a view that starts later in the storage, the delegate gets the wrong weights, and the .pte gives wrong results with no error. In the model above, the largest XNNPACK error was about 6 to 8. For a view that starts at the beginning, the values are right, but the file carries the whole parent. XNNPACK per-channel scales are read the same way. They also skip the `contiguous()` step that weights get, so a transposed groupwise scale is written in the wrong order. ## What this change does - Before reading a weight's bytes, both serializers now clone it if its storage is larger than the tensor. The ExecuTorch emitter already uses this rule for weights that are views (`exir/tensor.py`, `exir/emit/_emitter.py`). - XNNPACK per-channel scales are small, so they are now always copied into contiguous storage of their own. - The Vulkan path that serializes raw tensors (only tests use it today) now does the same. ## What was tested On macOS arm64. Each of these fails before this change and passes after: - `test_fp32_linear_view_weight` (XNNPACK): a linear whose weight is a view. - `test_qd8_linear_transposed_scale` (XNNPACK): a groupwise int4 linear with a transposed scale. - `test_view_constant_serializes_only_its_own_bytes` (Vulkan graph builder): checks the bytes of a view constant, no GPU needed. - `test_serialize_vulkan_binary` (existing): now also serializes a view and a transposed view. The default `backends/xnnpack/test` run passes. It skips `test_linear.py`, so that file was run directly: all tests pass except `test_linear_qd8_{f32,f16}_per_channel_int4`, which fail the same way without this change. A per-channel scale replaced by an equal view was also checked by hand: off by about 4e4 before, exact after. ## Not in this change `_tensor_to_bytes` in `exir/_serialize/_named_data_store.py` has a similar problem: for a non-contiguous tensor, such as a channels_last slice of a bigger tensor, it writes the whole storage. XNNPACK and Vulkan pass it plain bytes, so this change does not depend on it. It is left for a separate fix. cc @SS-JIA @manuelcandales @digantdesai @cbilgin
Summary: Extend the shared `FuseBatchNormWithConvPass` to fold frozen evaluation BatchNorm during Native AOT export, using a thin `ExportedProgram` adapter in Native's default pass pipeline. Preserve existing `GraphModule` callers and grouped/transposed convolution support. Functionalize all methods before collecting mutable buffers. This protects running statistics updated by another `.ptn` method while allowing folding through `ReLU(inplace=True)`. Authored with assistance from Codex. Reviewed By: digantdesai Differential Revision: D126696057 Pull Request resolved: pytorch#24068
Summary: Add `export_torchvision`, which exports torchvision classifiers through the registered CPU and XNNPACK FP32 recipes. Alongside `model.pte`, it saves the `.pt2` capture, a deterministic `input.bin`, the eager logits in `eager.bin`, the edge-program logits in `golden.bin`, and an `export.json` manifest with artifact hashes and preprocessing provenance. Runners and benchmarks can use the saved input and eager logits to validate a PTE before measuring it. Authored with OpenAI Codex. Reviewed By: digantdesai Differential Revision: D125714169 Pull Request resolved: pytorch#23934
…torch#24193) Summary: Run the shared `FuseBatchNormWithConvPass` and `FuseBatchNormWithLinearPass` in `CpuBackend.preprocess`. Unfused `_native_batch_norm_legit_no_training` already executes through the reference provider, but as a separate kernel with its own serialized weight and stats; folding it into the preceding conv/linear removes that kernel from the delegate. The passes are listed in `CPU_PREPROCESS_PASSES` (mirroring `XNNPACKPassManager.passes`) so every lowering flow gets them from one extension point. Fusion introduces `get_attr` constants, so preprocess follows with `lift_constant_tensor_pass` (the native serializer rejects unlifted constants), then drops parameter/buffer/constant inputs left without users. Without that step the delegate serialized the original weights and batch norm stats next to the fused ones. Authored with Claude Code. Reviewed By: digantdesai Differential Revision: D126784606
Summary: `aten::_neg_view_copy` (negation with view-copy autograd metadata, identical numerics to neg for inference) had no provider route, crashing at backend init (0x10). Add it to the ET spec table mapped to `aten::neg.out`. Reviewed By: digantdesai Differential Revision: D126929421
…int16 (pytorch#22924) ### Summary Follow-up to: pytorch#20805 Remove the FP32 tagging workaround for multimodal encoder quantized I/O and tag it as uint16 instead. ### Test plan ``` bash python examples/qualcomm/oss_scripts/llama/llama.py --build_folder build-android --device ${SERIAL_NUM} --soc_model ${SOC_MODEL} --decoder_model granite_speech_3_3-2b --model_mode hybrid --prefill_ar_len 128 --max_seq_len 1024 --prompt "can you transcribe the speech into a written format?" --audio_path "https://huggingface.co/ibm-granite/granite-speech-3.3-2b/resolve/main/10226_10111_000000.wav?download=true" --calib_samples examples/qualcomm/oss_scripts/llama/assets/samples/audio.json ``` cc: @psiddh @shewu-quic
`backends/transforms/targets.bzl` defines the `fuse_batch_norm_with_linear` `python_library` twice (lines 138 and 152). The first copy comes from pytorch#24192. The second came in with the squash of pytorch#24193, whose branch picked up the block again when it was merged with `main`. Buck refuses to register the same target twice, so any command that loads `//backends/transforms/...` fails: ``` error: Attempted to register target root//backends/transforms:fuse_batch_norm_with_linear twice ``` This breaks `unittest-buck` on `main` and on open PRs. This change deletes the second, identical definition. No other targets in the file are duplicated. ### Test plan - Checked that `targets.bzl` now defines each target name once. - CI: `unittest-buck` should get past the `buck2 query` step.
…ze on CPU (pytorch#23915) ### Summary FoldQDQ marks a quantized parameter's dequantize as a bypass node, so the partitioner force-passes it. When its consumer isn't delegated (rejected by op validation, or in `skip_node_id_set`), the dequantize is lowered alone as a QNN Dequantize, which rejects per-channel encodings, and context binary generation fails. The partitioner now leaves it on CPU with its consumer. Fixes pytorch#19908. ### Test plan `test_passes.py::TestPasses::test_skipped_conv_keeps_its_weight_dequantize_on_cpu` fails on main ("Op Dequantize does not support per-channel quant tensor") and passes here; the rest of `test_passes.py` passes on both. cc @cccclai @winskuo-quic @shewu-quic @haowhsu-quic @DannyYuyang-quic @cbilgin @abhinaykukkadapu @psiddh @nil-is-all
Summary: Use the standard type directly in MTIA, ExecuTorch, TorchComms, FBLearner, text, and inference consumers before removing the C10 alias. Replace obsolete string-view includes where no C10 helpers are used and keep Vulkan's type matching consistent with fully qualified torchgen output. Authored with Metamate AI assistance. Differential Revision: D127278310 Pull Request resolved: pytorch#24139
Summary: The CPU delegate rejected any binding whose strides/dim-order were not canonical-contiguous, including empty (0-byte) tensors whose layout is meaningless: PyTorch does not canonicalize strides for 0-size dims, so e.g. a (0,0,0,0,0) input arrives with dim-order != identity and fails with 0x12 (InvalidArgument). Skip the contiguity check for 0-byte bindings; the input/output memcpys already skip empty buffers, and providers handle 0-size execution. Reviewed By: digantdesai Differential Revision: D126930703 Pull Request resolved: pytorch#24195
Summary: Add `EXECUTORCH_BUILD_NATIVE_LLM_RUNNER` to build and install `run_hf_llm_vulkan` and `run_hf_llm_ggml` for the enabled engines. Share gflags setup with the vision runner, resolve tokenizers with regex lookahead support, and add `native_llm_runner_utils_test`. Document building and running the GGML CPU LLM runner. Authored with assistance from Codex. Reviewed By: digantdesai Differential Revision: D125479665 Pull Request resolved: pytorch#23899
Summary: Exported methods can reuse a constant name with different immutable tensors, causing `to_native` to reject the combined program. Namespace every immutable constant as `<method>/<key>`, including single-method programs, and update its graph references, including quantization metadata and nested graphs. Identical tensors still deduplicate when saving the `.ptn`. Keys mutated by any method use shared `state/<key>` names throughout the program, preserving state shared by readers and writers. Update non-persistent buffer bindings and mutation outputs consistently, and reserve the `state` method namespace. Reject differing values for shared mutable keys and collisions between final names. Authored with assistance from OpenAI Codex. Reviewed By: digantdesai Differential Revision: D125294428 Pull Request resolved: pytorch#23891
… pybindings (pytorch#23183) ## Summary Adds `ExecuTorchMethod.set_output(tensor, index)`, exposed as `executorch.runtime.Method.set_output`. It binds an output that is not memory planned to caller storage: the method writes that output straight into `tensor` on every later call, with no copy, and returns it as that tensor (viewed at the written shape when a dynamic output is smaller), uncloned, since binding is the opt-in. Later calls overwrite it. `set_output` rejects, with `ValueError`, a tensor that doesn't match the output's dtype and device, isn't contiguous or is too small for the output's largest shape, or shares memory with another bound output or an input; and an output that is memory planned, returned at more than one index, not contiguous, or also an input. Storage the output used before binding is kept alive, so tensors already returned with `clone_outputs=False` stay valid. The ATen bindings don't support it: they swap the data pointer of the output's storage in place, so earlier outputs would read the bound tensor, or freed memory after a rebind. An output that is also an input is found by checking that both are the same value in the method (`&get_output(i) == &get_input(j)`), not by comparing data pointers, which could match the wrong tensor. The check runs once, after the first successful `set_inputs()`, not in the constructor: `get_input()` marks an input as set, which would disable `execute()`'s "Input N has not been set" check. So binding such an output before the first call can't be rejected up front: the first call removes that binding and raises, and later calls work normally. Unplanned inputs are handled as on main: a mutation is written into the caller's tensor, and an output that is that input is a clone by default and main's detached alias of the caller's storage with `clone_outputs=False` (pytorch#23089). Only `PyMethod` changes. The MLX `index_copy` changes from the first version of this PR are in pytorch#23508. ## Test plan New tests in `runtime/test/test_runtime.py` cover binding, rebinding, and dynamic outputs smaller than the bound tensor; rejection of bad dtype, size, layout, planned, duplicated, input-aliasing and memory-sharing bindings; recovery from a binding made before the first call; `execute()` without inputs after a binding; mutated unplanned inputs (updated in place, returned as a copy by default and sharing the caller's storage with `clone_outputs=False`) and planned ones (not updated); an uncloned output fed back in as the next input; and a failed `set_inputs()`. pytest runtime/test/test_runtime.py extension/pybindings/test This PR was authored with Claude Code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @rascani @nil-is-all @metascroy --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…rch#24214) The TensorRT backend page marked two features as needing a Torch-TensorRT build newer than 2.15: keeping the engines in a separate `.ptd` data file, and optimization profiles. Both are already in Torch-TensorRT 2.15, so the marks sent readers to a nightly build they do not need. This change drops the marks and lists both as plain features, the same way the Torch-TensorRT guide describes them. The links to the guide sections are unchanged. Two intro lines now say that the engines are stored in the `.pte` by default, since the `.ptd` option is no longer marked as newer. Checked on the Torch-TensorRT `release/2.15` branch: - `.ptd` engines: `external_engine_data` in `torch_tensorrt.save`, and the delegate reads the engine from the data file passed at load time (`data_path=` in Python, `Module(pte, ptd)` in C++). - Optimization profiles: `torch_tensorrt.Input(profiles=...)` at export, and `OptimizationProfileGuard` in the delegate header. - The linked export scripts and reference runner programs, including the multi-profile ones, are the same files on `release/2.15` as on `main`, so the links work for 2.15. Docs only; no code changes.
## What is wrong today Programs that go through `constant_prop_pass` can crash in two ways: one inside the pass, and one later when the program is lowered. **1. A folded value that is not a tensor.** The pass also folds values such as the int from `.item()` on a buffer, and the bools that `torch._check` turns into runtime checks. When a node that stays in the graph reads such a value, the pass tries to turn it into a constant tensor input and fails: ```python n = self.n.item() # self.n is a buffer torch._check(n >= 1) torch._check(n <= x.shape[0]) return x[:n] * self.w[:n] ``` ``` TypeError: cannot create weak reference to 'bool' object ``` **2. A folded parameter that still requires grad.** Folding `self.w.t()`, or one chunk of a packed weight, gives a view that still has `requires_grad=True`, because `torch.no_grad()` does not clear that flag on a view. For some views, such as a transposed one, `to_edge` then copies the constant into a non-leaf tensor, and lowering to a backend can fail when it deep-copies the program: ``` RuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment. ``` An op can also return the parameter itself, like `self.w.to(torch.float32)` on a float32 parameter. Then the pass stores the `nn.Parameter` as a constant, and `to_edge` fails with `AttributeError: ... is not a buffer`. ## What this change does Two commits, one per crash: 1. Only tensors can become constant inputs. So a non-tensor value that a node outside the fold reads now stays in the graph, together with the non-tensor values it is computed from. Tensors computed from it still fold: `self.w[:n]` still becomes one constant. Programs that `constant_prop_pass` handled before are unchanged. 2. A folded constant must be a plain tensor that does not require grad, so the pass now detaches every folded tensor. ## What was tested On macOS arm64: - Two new tests in `exir/tests/test_passes.py`, one per commit. Each fails before its commit and passes after: the first with `TypeError: cannot create weak reference to 'int' object`, the second with the `AttributeError` above. - The first covers a scalar read at runtime, one returned as an output, one whose first reader is folded, and one read only by folded nodes (it folds away). The program also lowers with `to_edge(...).to_executorch()`. - The second covers the transposed view and the `to` case, and checks that the output matches eager. - The rest of `exir/tests/test_passes.py`, `exir/emit/test/test_emit.py` and `exir/tests/test_quant_fusion_pass.py`. Five tests in those files fail the same way with and without this change, because they need the quantized and custom op libraries.
This is a test commit to test docgen check added in pytorch#24225 Signed-off-by: Erik Lundell <erik.lundell@arm.com> Change-Id: I5eb03f4d14f666f062a8c30ae712248e18b44e46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
[PLEASE REMOVE] See CONTRIBUTING.md's Pull Requests for ExecuTorch PR guidelines.
[PLEASE REMOVE] If this PR closes an issue, please add a
Fixes #<issue-id>line.[PLEASE REMOVE] If this PR introduces a fix or feature that should be the upcoming release notes, please add a "Release notes: " label. For a list of available release notes labels, check out CONTRIBUTING.md's Pull Requests.
Test plan
[PLEASE REMOVE] How did you test this PR? Please write down any manual commands you used and note down tests that you have written if applicable.
cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @rascani