Repository navigation
docs(executorch): rewrite the ExecuTorch deployment docs and fix the archive-only delegate build - #4927
Open
shoumikhin wants to merge 10 commits into
Open
docs(executorch): rewrite the ExecuTorch deployment docs and fix the archive-only delegate build#4927shoumikhin wants to merge 10 commits into
shoumikhin wants to merge 10 commits into
Conversation
github-actions
Bot
added
documentation
shoumikhin
force-pushed
the
trt-executorch-docs
branch
4 times, most recently
from
October 26, 2026 02:03
bc98f3a to
2e58331
Compare
shoumikhin
changed the title
shoumikhin
force-pushed
the
trt-executorch-docs
branch
5 times, most recently
from
October 26, 2026 16:42
5dc4975 to
dbdc9a5
Compare
shoumikhin
force-pushed
the
trt-executorch-docs
branch
from
October 26, 2026 22:09
dbdc9a5 to
93f0107
Compare
shoumikhin
added a commit
to pytorch/executorch
that referenced
this pull request
Oct 27, 2026
This is a draft because the page describes the intended end state of Torch-TensorRT 2.15. Some links go to sections of the Torch-TensorRT guide that exist only after pytorch/TensorRT#4927 merges and its docs site is rebuilt. And the examples rely on 2.15 changes from pytorch/TensorRT#4909, such as `torch_tensorrt.save` keeping the compiled graph by default. ## What is wrong today The TensorRT backend page has drifted away from Torch-TensorRT, the project that builds and ships the TensorRT delegate. - It points readers at a nightly build from early October. - It says CUDA graph replay never runs for engines on the shared activation scratch. Newer builds run it. - It repeats most of the Torch-TensorRT guide, so every delegate change is made twice. - The backends table lists TensorRT as Linux only. The CUDA backend page and the getting started page also disagree with the TensorRT page and with the code: on outputs kept on the GPU, on what happens when a planned input gets GPU memory, on CUDA versions, GPUs and Python versions, and on Windows. ## What this change does The TensorRT page keeps the usual backend sections, says less in each, and shows more code: - Features, target requirements, and development requirements with the one-line install command. - Export, including sending the operators TensorRT cannot compile to the CUDA backend, and keeping inputs and outputs on the GPU. - Runtime integration from Python, and a complete C++ project with the commands to build and run it. - A short list of performance options that link to the Torch-TensorRT guide. It describes Torch-TensorRT 2.15 and newer, and marks the two features that need a newer build. A second commit fixes the CUDA backend page, the getting started page and one docstring on the points above. To keep data on the GPU, all pages now leave inputs and outputs unplanned, which copies the least. ## What was tested On Linux x86-64 with an H100 GPU, an earlier version of the TensorRT page was run end to end: every Python block in order with nightly Torch-TensorRT, and the C++ example built and run against the installed packages. With nightly packages, the GPU input and output recipe was run from Python for both backends, and the CUDA page's C++ lines were built and run. A profiler showed one GPU copy fewer per Python call than with planned outputs. The other fixes were checked only by reading them against the sources. Windows was not tested.
shoumikhin
force-pushed
the
trt-executorch-docs
branch
3 times, most recently
from
October 27, 2026 21:59
bc00035 to
78cf5d8
Compare
…0 headers and core The delegate can build ExecuTorch's shared extension_cuda library from a checkout when nothing else provides it. Built standalone, as the delegate README's archive-only recipe does, that build failed twice. First it stopped compiling on c10/util/safe_numerics.h, because the include list it was given held only the checkout's parent folder, not ExecuTorch's own copy of c10. With that fixed, it stopped linking with "cannot find -lexecutorch_core": extension_cuda links executorch_core, and only an ExecuTorch build in the same project defines that target. The source-build branch now adds the c10 folder to the include list, as ExecuTorch's own build does, and, when no executorch_core target exists, defines one from the prebuilt core library. It is the same libexecutorch_core.a the delegate already links further down, and EXECUTORCH_CORE_LIBRARY, given on the command line or by a parent project, still overrides it. A project that adds ExecuTorch with add_subdirectory already has the target, so nothing changes there. Tested on Linux aarch64 with CUDA 13.2: the archive-only recipe without a prebuilt extension_cuda fails before this change and builds after it. A project that adds ExecuTorch without its CUDA backend builds before and after. A new unit test checks the branch keeps both pieces, and fails without this change.
…ction The ExecuTorch pages of the user guide were hard to follow and partly out of date: - The saving section carried its own install command, which disagreed with the guide's. - The install section opened with a command that gives no published wheel its delegate: only a nightly extra brings it today. No command on the page installed the delegate for Torch-TensorRT 2.15. The section also listed Windows, which 2.15 has no delegate for. - It said each delegate of a coalesced program runs on its own CUDA stream by default. Both now run on the calling thread's default stream. - The guide said engines on the shared activation scratch never replay a CUDA graph. In builds newer than 2.15 they do. - The guide said Python cannot run a program whose outputs are not planned. It can now, with ExecuTorch builds that have pytorch/executorch#23547, which both 2.15 and nightly install. - Features that 2.15 lacks were not marked, and two were missing: engines in a separate .ptd file, and optimization profiles. Weight streaming was not described at all. - The zero-copy KV cache section was a wall of text, and the list of save() options for this format named only six of them. - The comparison table said a .pt2 runs on Windows. save() refuses that format outside Linux. - The save() docstring said a target_device compile spec overrides the default target device. It does not move the engines: an engine runs on the GPU it was built on. - The saving section showed how to write an ETRecord with generate_etrecord=True. For a program with TensorRT engines that fails today with a SerializeError, on nightly and on 2.15. - Nothing said that TensorRT warns, on a green context stream, that the engine was built without green context constraints and may crash. - The guide's note said ordinary Torch-TensorRT builds keep their CUDA 12 support. The 2.15 release candidate and current nightly builds are built for CUDA 13 only. The guide is rewritten in shorter sentences. It keeps one install section, which the other pages now link to. It starts with the commands that work today. The 2.15 command pins the release and names its 0.2 delegate, from the test channel, so it keeps working when a newer 0.2 delegate ships with the next release candidate. It also lists the nightly channel for the ExecuTorch build that delegate pins, which that channel keeps only for a limited time, and the section says so. The nightly command follows, and then the plain form, which is enough once a release's extra brings the delegate. Each feature that 2.15 lacks is marked "Not in 2.15", which stays true after the next release branches, and a new section shows how to set a runtime option from C++ and from Python. The saving section now covers only the saving step: save() versus export(), every option of the executorch format, several methods in one file, and a condensed zero-copy KV cache section. The API page no longer says the module is Linux only, and the save() docstring points to the deployment guide, where the CUDA backend recipe moved. That docstring also stops saying that target_device moves the engines, and says that generate_etrecord does not work yet for programs with TensorRT engines. The export() docstring says the same. Windows is marked "not in 2.15", because the 2.15 release candidate publishes no Windows delegate wheel. The test that checks every printed install command requires --pre on any command that installs the extra from the nightly channel. The 2.15 command lists that channel only for the ExecuTorch build, and must not take --pre, or pip takes nightly builds of everything else. The test now skips the --pre requirement, and only that, for a user docs command that also names the delegate with == or ~=. Every other check, including the supported channel check, still runs on it. New cases pin the 2.15 command, the same command with an unsupported channel, and a nightly command without --pre. Tested on Linux x86-64 with an H100 and nightly Torch-TensorRT. Ran every Python block of the guide in order: they export the plain, coalesced, GPU-resident and separate engine file programs, run the first three, and set the runtime options from Python. A separate run loaded the separate engine file program, which fails without its .ptd and works with it. Built and ran the C++ snippets against the installed wheels, all but the optimization profile one, and ran one save() example and one export() example from the saving section. Checked these 2.15 differences against the Torch-TensorRT 2.15.0 release candidate: the re-export default, the Python runtime options, the separate engine file, and a CPU input to a GPU-resident program. On both versions, ran a GPU-resident program whose last part runs on the CUDA backend, with planned and with unplanned outputs. Windows was not tested. The pages build with Sphinx with no new warnings, and the tests that read these files pass. The new cases fail without the test change: the 2.15 command is rejected for lacking --pre. On Linux aarch64, the 2.15 command as written resolves in a fresh environment with pip --dry-run: torch-tensorrt 2.15.0, torch-tensorrt-executorch-runtime 0.2.0 and torch 2.15.0 from the test channel, ExecuTorch 1.6.0.dev20261009 from the nightly channel, TensorRT 11.3 from PyPI.
…unner READMEs The three READMEs next to the ExecuTorch delegate had grown long and hard to read. The delegate README was over 900 lines, with broken line wraps and long measurement notes. The runtime wheel README repeated its Python example three times. It and the reference runner README each carried their own install command, and those disagreed with the guide. Each README now says what its directory is, links to the deployment guide for installing, and keeps to its own topic: - The delegate README: how to build the delegate from source, the rules a C++ caller has to keep (threads, buffers, streams), load errors, known issues, and a reference for each runtime option. Implementation notes that already live in the code comments are gone. So is a known issue about Python outputs in host memory, which pytorch/executorch#23547 fixed. The capture rules and the notes on how the shared scratch pool grows stay, shorter, because the delegate's error message and header point there. The build recipe now sets EXECUTORCH_SOURCE_DIR, which the delegate needs to find the ExecuTorch checkout when the CUDA backend is off. The recipe for building the archive alone now passes the prebuilt extension_cuda library from libtorchtrt.tar.gz, and says how to build that library from the checkout instead. It also builds and links ExecuTorch's portable kernels, which carry the copy operators at the edges of a default-exported program; without them the program fails to load with Error::OperatorMissing. The recipes list PyYAML next to PyTorch, because ExecuTorch's kernel code generator needs both, so a missing PyTorch is no longer given as a reason to build without the CUDA backend. The CUDA graph section says that the second call records and launches the graph, and that with the shared scratch on, recording can move to a later call. - Features that 2.15 lacks are marked "Not in 2.15", in the delegate and reference runner READMEs. - The runtime wheel README: version matching, use from Python and C++, runtime library paths, and how to build the wheel. It no longer says that building the wheel needs NVIDIA's package index: TensorRT also installs from PyPI alone, more slowly. The wheel build renders MODULE.bazel with 13.4 as the CUDA upper bound, the value CI uses, because TensorRT has no download for 14.0. It says what register() is for (registering after an import made with TORCH_TENSORRT_SKIP_DELEGATE_REGISTRATION), and that the import order does not matter. - The advice to load programs from one Python thread is gone from both READMEs. Its reason was an output redirection in ExecuTorch's Python bindings, which ExecuTorch 1.6 no longer does. - The reference runner README: what each example shows, one build recipe for all of them (with PyTorch and PyYAML in the Python environment), and how to run each one. It also says that the prebuilt runner has no CUDA backend, so it cannot run a program split with the CUDA backend, and that the JetPack and TensorRT-RTX builds do not ship it. Tested on Linux x86-64 with an H100 and nightly Torch-TensorRT: checked the wheel's dependency metadata, runtime paths and exported symbols against the README, built and ran the C++ link recipe, ran the device-resident export and the Python loader from the reference runner README, and ran the prebuilt runner. Built the delegate archive alone with a prebuilt extension_cuda library. On Linux aarch64 with CUDA 13.2, built the archive alone without the prebuilt library, and the delegate inside a project that adds ExecuTorch without its CUDA backend. Both work. Also on Linux aarch64, built ExecuTorch's core and portable kernels as the standalone recipe says, and linked a test program with them as listed: it finds both copy operators, and without the portable kernels it finds neither. The other source build recipes were not run. The tests that read these READMEs (the ExecuTorch pin checks and the pin updater) pass.
Each ExecuTorch export example told readers to install Torch-TensorRT with its own command, an editable install from a source checkout, which is not how most readers get it and disagreed with the guide. Two docstrings were also out of date. One said that Python cannot run a program whose outputs are not planned, which it can with ExecuTorch builds that have pytorch/executorch#23547. The other said a CPU input to a program that expects GPU inputs still works, with a copy, but ExecuTorch Python builds newer than the one 2.15 installs refuse it. A code comment in export_device_resident.py said both memory planning flags are needed, when only the input one is. The examples now link to the guide's install section, the multi-profile example says that it needs a build newer than 2.15 and links to the nightly install, and the docstrings and the comment say what happens today, including when a CPU input is copied and when it is refused. The device-resident example names the ExecuTorch change and the first build that has it. Tested on Linux x86-64 with an H100 and nightly Torch-TensorRT: ran export_device_resident.py and load_model_device_resident.py, which pass, and checked that the Python runtime refuses a CPU input for that program, and that a planned input fails while planned outputs work. The test files that read these examples pass.
The install section showed a long command for 2.15 that named the delegate package and four
indexes, and a nightly command with two indexes. Each channel now has one command:
pip install "torch_tensorrt[executorch]"
pip install --pre "torch_tensorrt[executorch]" --index-url https://download.pytorch.org/whl/nightly/cu132
A release from PyPI is built for CUDA 13.2, the CUDA version of PyTorch's own PyPI wheel, and its
executorch extra brings ExecuTorch's CUDA 13.2 build and the delegate. A nightly's extra brings
the matching nightlies from the nightly index.
On Windows, PyPI has only the CPU builds of PyTorch and ExecuTorch, so the release section says to
add the PyTorch CUDA index there.
The pages also said that 2.15 re-exports in save() by default, never replays an engine on the
shared activation scratch, has no Windows delegate, and cannot pass runtime options from Python.
The 2.15 backport (pytorch#4909) brings the save() and replay fixes and the Windows delegate, and the
ExecuTorch release it pins has the Python option API, so those notes go.
The install command test let a docs command that names the delegate skip --pre. Only the old
2.15 command needed that, so the exception and its test cases go. The case that refuses a
nightly command without --pre now uses the guide's --index-url form.
… and docstrings Review of the rewritten pages found a few things a newcomer would trip on: - The C++ section showed a CMake fragment with no project, no executable and a literal "...", and no build or run step. It now shows a whole CMakeLists.txt and the commands that build and run the program, and the runtime wheel README's command no longer ends in "...". - "Not in 2.15." reads as if the feature was removed in 2.15. The marks now say "Newer than 2.15" and that the feature needs a nightly build or a later release. - The guide said only builds newer than 2.15 refuse a CPU input to a GPU-resident program from Python. The refusal comes from ExecuTorch's Python runtime, not from a Torch-TensorRT version; from C++ the input is copied or refused depending on the first delegate. The guide and the loader example now say that. - The JetPack build joins the builds without a delegate, and the prebuilt runner is said to be missing from the JetPack and TensorRT-RTX builds, as the build files and the error message say. - The use_cuda_graphs docstrings of save() and export() now say a process option set to False also overrides the saved choice, and the TensorRTPartitioner docstring says each partition's target_device comes from its own engine unless compile_specs sets one. The two save() import errors no longer say the integration supports only Linux; it also supports Windows x86-64. Tested: on Linux x86-64 with an H100 and nightly Torch-TensorRT, the guide's export block, the new CMakeLists.txt, main.cpp and build commands, used exactly as written, built and ran the program.
Several pages said "yet", "until it is", "later", "Needs a nightly build or a later release", "Workaround", or named the ExecuTorch nightly where a behaviour started. They now state the behaviour. A feature that 2.15 lacks says "Needs Torch-TensorRT newer than 2.15, such as a nightly build", which stays true after the next release. The reference runner README no longer sends readers to a source checkout of main for the two multi-profile programs, since the tarball of a newer release carries them. The install section's subheading now says "these commands", and two sentences without a verb are split. The bullet on unplanned outputs no longer calls their buffers memory you own, since the Python runtime allocates them.
The guide's recipe for keeping inputs and outputs on the GPU left the outputs planned. Then the outputs live in the program's own memory, the next call overwrites them, and ExecuTorch's Python execute() copies each one on every call unless it gets clone_outputs=False. With alloc_graph_output=False the runtime does not copy them: TensorRT writes straight into the output memory, the Python runtime allocates that memory and does not overwrite an output the caller still holds, and a C++ caller sets it once with Module::set_output. The guide now passes alloc_graph_output=False, like export_device_resident.py already does, and explains the planned alternative in one bullet. The example's docstring and comment say the same. The bullet on planned inputs now says that every call fails, not only one given a GPU tensor.
…e details The guide and the backend README lost two notes when they were rewritten. A .ptd file is not named after its .pte, and save() replaces a file with the same name, so two programs saved into one directory with the same external_engine_data leave the first one unable to load. The guide's example now names the file after its program and says so in a warning. The backend README again says that a stripped engine saved again as it is stays refused, and when the check cannot catch it.
The 2.15 branch now has the .ptd engine file, the refusal of engines without weights and optimization profiles, so the docs, the reference runner README and the multi-profile example no longer mark them as newer than 2.15.
shoumikhin
force-pushed
the
trt-executorch-docs
branch
from
October 27, 2026 22:01
78cf5d8 to
1ad5dcf
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is wrong today
The ExecuTorch docs had drifted from the code:
generate_etrecord=Truefailing with TensorRT engines.extension_cuda, failed to compile and then to link.What this change does
extension_cudagets ExecuTorch'sc10headers and, when nothing defines it, the prebuilt core library.pip install "torch_tensorrt[executorch]". On Windows, add the PyTorch CUDA index.pip install --pre "torch_tensorrt[executorch]" --index-url https://download.pytorch.org/whl/nightly/cu132..ptdfile and optimization profiles. All of them are on the 2.15 branch, so nothing is marked as newer.save(),export()andTensorRTPartitionerdocstrings and twosave()import errors, and trims each README to its topic..ptdfile after its program.The docs describe the target
dllist, TensorRT and about a dozen ExecuTorch dependencies, and, on Windows, torchao. Today it needs--extra-index-url https://pypi.org/simpleadded, and on Windows also the CPU nightly index.What was tested