Skip to content

[2.15] Bump the ExecuTorch delegate to 0.2.1 and fix the ExecuTorch guide - #4909

Open
shoumikhin wants to merge 5 commits into
pytorch:release/2.15from
shoumikhin:backport-2.15-4833
Open

[2.15] Bump the ExecuTorch delegate to 0.2.1 and fix the ExecuTorch guide#4909
shoumikhin wants to merge 5 commits into
pytorch:release/2.15from
shoumikhin:backport-2.15-4833

Conversation

@shoumikhin

@shoumikhin shoumikhin commented Oct 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What is wrong today

The 2.15 branch now has #4919 and the other ExecuTorch changes from main. Two problems are left:

What this change does

  1. Bump the delegate to 0.2.1, so pip sees it as newer than 0.2.0 and replaces the old delegate wherever it installs one.
  2. Add a test that the guide's release install stays one command, pip install "torch_tensorrt[executorch]", and a test case that refuses a command naming the nightly index without --pre.
  3. Point the .pte saving section at the guide's install commands.
  4. Fix contradictions between the guide, the saving section and the READMEs, and two save() errors that said only Linux works.
  5. Update the guide for feat(executorch): do not retrace by default when saving a .pte #4881 and fix(executorch): allow CUDA graph replay on shared activation scratch #4833: its examples drop retrace=False and arg_inputs, which save() no longer needs, and it now says engines on the shared activation scratch replay CUDA graphs.

What the release command still needs

The guide's one-command release install brings the delegate only after these steps:

  1. ExecuTorch 1.6.0 CUDA wheels on the test index for CUDA 13.2 and 13.4, Linux and Windows.
  2. This branch moves its ExecuTorch pin to 1.6.0, and the next release candidate is built from it.
  3. ExecuTorch 1.6.0's Linux PyPI wheels are its CUDA 13.2 build.
  4. ExecuTorch, the delegate and torch-tensorrt are published in that order.

Do not promote the first 2.15.0 release candidate: it predates this change.

The nightly index does not carry dllist or TensorRT, so the nightly command also needs --extra-index-url https://pypi.org/simple for now.

What was tested

  • The ExecuTorch Python tests on CPU-only PyTorch 2.15 give the same results as the branch without this change. The pin test passes at every commit. Tests that need a GPU, C++ or a built torch-tensorrt are left to CI.
  • The changed guide pages build with Sphinx, with no new warnings.

@meta-cla meta-cla Bot added the cla signed label Oct 21, 2026
@github-actions github-actions Bot added component: tests Issues re: Tests component: api [C++] Issues re: C++ API labels Oct 21, 2026
@github-actions
github-actions Bot requested a review from narendasan October 21, 2026 17:59
@shoumikhin shoumikhin changed the title [2.15] Backport #4833: allow CUDA graph replay on shared activation scratch [2.15] Backport ExecuTorch fixes: #4833, #4839, #4821, #4881, #4882 Oct 22, 2026
@github-actions github-actions Bot added documentation Improvements or additions to documentation component: build system Issues re: Build system component: api [Python] Issues re: Python API labels Oct 22, 2026
@shoumikhin
shoumikhin force-pushed the backport-2.15-4833 branch 2 times, most recently from 74dcb90 to fa1d8a3 Compare October 23, 2026 13:34
@shoumikhin shoumikhin changed the title [2.15] Backport ExecuTorch fixes: #4833, #4839, #4821, #4881, #4882 [2.15] Backport ExecuTorch fixes: #4833, #4839, #4821, #4881, #4882, #4919 Oct 23, 2026
@shoumikhin shoumikhin mentioned this pull request Oct 24, 2026
Merged
@shoumikhin
shoumikhin force-pushed the backport-2.15-4833 branch 2 times, most recently from 566d13c to 9bac35f Compare October 25, 2026 05:30
@shoumikhin shoumikhin mentioned this pull request Oct 25, 2026
Open
shoumikhin added a commit to shoumikhin/TensorRT that referenced this pull request Oct 26, 2026
@shoumikhin
docs(executorch): show one install command per channel
a9116e1 
The install section showed a long command for 2.15 that named the delegate package and four
indexes, and a nightly command with two indexes. Each channel now has one command:

    pip install "torch_tensorrt[executorch]"
    pip install --pre "torch_tensorrt[executorch]" --index-url https://download.pytorch.org/whl/nightly/cu132

A release from PyPI is built for CUDA 13.2, the CUDA version of PyTorch's own PyPI wheel, and its
executorch extra brings ExecuTorch's CUDA 13.2 build and the delegate. A nightly's extra brings
the matching nightlies from the nightly index.

On Windows, PyPI has only the CPU builds of PyTorch and ExecuTorch, so the release section says to
add the PyTorch CUDA index there.

The pages also said that 2.15 re-exports in save() by default, never replays an engine on the
shared activation scratch, has no Windows delegate, and cannot pass runtime options from Python.
The 2.15 backport (pytorch#4909) brings the save() and replay fixes and the Windows delegate, and the
ExecuTorch release it pins has the Python option API, so those notes go.

The install command test let a docs command that names the delegate skip --pre. Only the old
2.15 command needed that, so the exception and its test cases go. The case that refuses a
nightly command without --pre now uses the guide's --index-url form.
shoumikhin added a commit to shoumikhin/TensorRT that referenced this pull request Oct 26, 2026
@shoumikhin
docs(executorch): show one install command per channel
0b4ad34 
The install section showed a long command for 2.15 that named the delegate package and four
indexes, and a nightly command with two indexes. Each channel now has one command:

    pip install "torch_tensorrt[executorch]"
    pip install --pre "torch_tensorrt[executorch]" --index-url https://download.pytorch.org/whl/nightly/cu132

A release from PyPI is built for CUDA 13.2, the CUDA version of PyTorch's own PyPI wheel, and its
executorch extra brings ExecuTorch's CUDA 13.2 build and the delegate. A nightly's extra brings
the matching nightlies from the nightly index.

On Windows, PyPI has only the CPU builds of PyTorch and ExecuTorch, so the release section says to
add the PyTorch CUDA index there.

The pages also said that 2.15 re-exports in save() by default, never replays an engine on the
shared activation scratch, has no Windows delegate, and cannot pass runtime options from Python.
The 2.15 backport (pytorch#4909) brings the save() and replay fixes and the Windows delegate, and the
ExecuTorch release it pins has the Python option API, so those notes go.

The install command test let a docs command that names the delegate skip --pre. Only the old
2.15 command needed that, so the exception and its test cases go. The case that refuses a
nightly command without --pre now uses the guide's --index-url form.
@shoumikhin shoumikhin mentioned this pull request Oct 26, 2026
Merged
@shoumikhin shoumikhin changed the title [2.15] Backport ExecuTorch fixes: #4833, #4839, #4821, #4881, #4882, #4919 [2.15] Backport #4919 and bring the ExecuTorch delegate version and guide up to date Oct 26, 2026
@lanluo-nvidia
lanluo-nvidia self-requested a review October 26, 2026 20:24
shoumikhin added a commit to shoumikhin/TensorRT that referenced this pull request Oct 26, 2026
@shoumikhin
docs(executorch): show one install command per channel
04e42ae 
The install section showed a long command for 2.15 that named the delegate package and four
indexes, and a nightly command with two indexes. Each channel now has one command:

    pip install "torch_tensorrt[executorch]"
    pip install --pre "torch_tensorrt[executorch]" --index-url https://download.pytorch.org/whl/nightly/cu132

A release from PyPI is built for CUDA 13.2, the CUDA version of PyTorch's own PyPI wheel, and its
executorch extra brings ExecuTorch's CUDA 13.2 build and the delegate. A nightly's extra brings
the matching nightlies from the nightly index.

On Windows, PyPI has only the CPU builds of PyTorch and ExecuTorch, so the release section says to
add the PyTorch CUDA index there.

The pages also said that 2.15 re-exports in save() by default, never replays an engine on the
shared activation scratch, has no Windows delegate, and cannot pass runtime options from Python.
The 2.15 backport (pytorch#4909) brings the save() and replay fixes and the Windows delegate, and the
ExecuTorch release it pins has the Python option API, so those notes go.

The install command test let a docs command that names the delegate skip --pre. Only the old
2.15 command needed that, so the exception and its test cases go. The case that refuses a
nightly command without --pre now uses the guide's --index-url form.
shoumikhin added a commit to pytorch/executorch that referenced this pull request Oct 27, 2026
@shoumikhin
Update the TensorRT backend docs page (#24136)
d6abe23 
This is a draft because the page describes the intended end state of
Torch-TensorRT 2.15. Some links go to sections of the Torch-TensorRT
guide that exist only after pytorch/TensorRT#4927 merges and its docs
site is rebuilt. And the examples rely on 2.15 changes from
pytorch/TensorRT#4909, such as `torch_tensorrt.save` keeping the
compiled graph by default.

## What is wrong today

The TensorRT backend page has drifted away from Torch-TensorRT, the
project that builds and ships the TensorRT delegate.

- It points readers at a nightly build from early October.
- It says CUDA graph replay never runs for engines on the shared
activation scratch. Newer builds run it.
- It repeats most of the Torch-TensorRT guide, so every delegate change
is made twice.
- The backends table lists TensorRT as Linux only.

The CUDA backend page and the getting started page also disagree with
the TensorRT page and with the code: on outputs kept on the GPU, on what
happens when a planned input gets GPU memory, on CUDA versions, GPUs and
Python versions, and on Windows.

## What this change does

The TensorRT page keeps the usual backend sections, says less in each,
and shows more code:

- Features, target requirements, and development requirements with the
one-line install command.
- Export, including sending the operators TensorRT cannot compile to the
CUDA backend, and keeping inputs and outputs on the GPU.
- Runtime integration from Python, and a complete C++ project with the
commands to build and run it.
- A short list of performance options that link to the Torch-TensorRT
guide.

It describes Torch-TensorRT 2.15 and newer, and marks the two features
that need a newer build.

A second commit fixes the CUDA backend page, the getting started page
and one docstring on the points above. To keep data on the GPU, all
pages now leave inputs and outputs unplanned, which copies the least.

## What was tested

On Linux x86-64 with an H100 GPU, an earlier version of the TensorRT
page was run end to end: every Python block in order with nightly
Torch-TensorRT, and the C++ example built and run against the installed
packages.

With nightly packages, the GPU input and output recipe was run from
Python for both backends, and the CUDA page's C++ lines were built and
run. A profiler showed one GPU copy fewer per Python call than with
planned outputs.

The other fixes were checked only by reading them against the sources.
Windows was not tested.
@pytorchbot pytorchbot mentioned this pull request Oct 27, 2026
Open
shoumikhin added a commit to shoumikhin/TensorRT that referenced this pull request Oct 27, 2026
@shoumikhin
docs(executorch): show one install command per channel
8d107d5 
The install section showed a long command for 2.15 that named the delegate package and four
indexes, and a nightly command with two indexes. Each channel now has one command:

    pip install "torch_tensorrt[executorch]"
    pip install --pre "torch_tensorrt[executorch]" --index-url https://download.pytorch.org/whl/nightly/cu132

A release from PyPI is built for CUDA 13.2, the CUDA version of PyTorch's own PyPI wheel, and its
executorch extra brings ExecuTorch's CUDA 13.2 build and the delegate. A nightly's extra brings
the matching nightlies from the nightly index.

On Windows, PyPI has only the CPU builds of PyTorch and ExecuTorch, so the release section says to
add the PyTorch CUDA index there.

The pages also said that 2.15 re-exports in save() by default, never replays an engine on the
shared activation scratch, has no Windows delegate, and cannot pass runtime options from Python.
The 2.15 backport (pytorch#4909) brings the save() and replay fixes and the Windows delegate, and the
ExecuTorch release it pins has the Python option API, so those notes go.

The install command test let a docs command that names the delegate skip --pre. Only the old
2.15 command needed that, so the exception and its test cases go. The case that refuses a
nightly command without --pre now uses the guide's --index-url form.
@shoumikhin
[2.15] bump the ExecuTorch delegate to 0.2.1
9b70eae 
Since the pytorch#4821 backport, every .pte saved from this branch stores its
engines as named data (blob tag TR03). The 0.2.0 delegate published with
the first 2.15.0 release candidate predates that and refuses these files
with only "failed to parse TensorRT blob". A delegate rebuilt from this
branch at the same version would look identical to pip, so an
environment that already has the old one would keep it.

Bump the delegate to 0.2.1, so pip sees it as newer than 0.2.0 and
replaces the old delegate wherever it installs one. The README's build
recipe uses the new version.
@shoumikhin
[2.15] test that the ExecuTorch guide's release install is one command
1cccbd7 
The guide on this branch installs a release with one command:

    pip install "torch_tensorrt[executorch]"

Once this branch pins an ExecuTorch release, its PyPI wheel's executorch
extra brings ExecuTorch's CUDA 13.2 build and the delegate built for
CUDA 13.2, so naming either in the command would only let them drift
apart. A new test checks that the guide's release install is exactly
this command.

The install command test also gains a case that refuses a command which
names the nightly index without --pre.
@shoumikhin
[2.15] point the .pte saving section at the install guide
71ed19a 
The saving section still printed a nightly install command and said the
integration supports Linux. On this branch that command takes main's
nightly in a fresh environment, and with 2.15 installed it adds ExecuTorch
but no delegate, because a release's extra leaves the delegate out. It
also disagreed with the guide linked just above it.

Replace the platform sentence and the command with a pointer to the
guide's Installation section.
@shoumikhin
[2.15] fix contradictions in the ExecuTorch guide and READMEs
51b5958 
The 2.15 guide and the pages it links disagreed with each other and with the code:

- The saving section said each delegate of a coalesced program runs on its own CUDA stream, and
  that a zero-copy TensorRT delegate returns before its engine finishes. Both delegates use the
  calling thread's default stream, and the TensorRT delegate waits for its work. The section
  now says so, and names the one case that needs a sync: the CUDA backend running last.
- Two pages described an overwritten coalesced weight file differently. Both now say it can give
  wrong answers or fail to load.
- The comparison table said a .pt2 runs on Windows. AOTInductor packaging is Linux only.
- The guide promised dynamic shapes in a section that has none, and its example list missed two
  examples.
- A coalesced program's data file is passed when loading from Python, the JetPack build joins
  the builds without a delegate, and ATen, KV and SM are spelled out.
- The saving page's format list no longer calls ExecuTorch output Linux-only.
- The guide said a host tensor passed to a device-resident program only costs a copy, and that
  Python cannot run a program with unplanned outputs. ExecuTorch's Python runtime refuses the host
  tensor and allocates device memory for unplanned outputs, so the guide and the runtime README
  now say so.
- Two save() error messages said the integration supports only Linux. It supports Windows x86-64
  too.
@shoumikhin
[2.15] update the ExecuTorch guide for pytorch#4833 and pytorch#4881
325a66a 
On this branch the guide was picked after the change that stops save()
from retracing, so its examples still pass retrace=False and
arg_inputs, and the text presents them as two extra arguments save
needs. For the ExecuTorch format save() now keeps the compiled graph as
it is and needs no example inputs. The examples now use one extra
argument, output_format="executorch", as on main.

The CUDA graph paragraph also said that engines on the shared
activation scratch never replay. With pytorch#4833 they do, each with its own
buffers for its inputs and outputs, and they record again when the
pool moves to a larger buffer. The paragraph now says so.
@shoumikhin shoumikhin changed the title [2.15] Backport #4919 and bring the ExecuTorch delegate version and guide up to date [2.15] Bump the ExecuTorch delegate to 0.2.1 and fix the ExecuTorch guide Oct 27, 2026
shoumikhin added a commit to shoumikhin/TensorRT that referenced this pull request Oct 27, 2026
@shoumikhin
docs(executorch): show one install command per channel
6307a77 
The install section showed a long command for 2.15 that named the delegate package and four
indexes. Each channel now has one command:

    pip install "torch_tensorrt[executorch]"
    pip install --pre "torch_tensorrt[executorch]" --index-url https://download.pytorch.org/whl/nightly/cu132 --extra-index-url https://pypi.org/simple

A release from PyPI is built for CUDA 13.2, the CUDA version of PyTorch's own PyPI wheel, and its
executorch extra brings ExecuTorch's CUDA 13.2 build and the delegate. A nightly's extra brings
the matching nightlies from the nightly index. The nightly command keeps PyPI as a second index,
because TensorRT's wheels and a few other dependencies are only there, the same as the README.

On Windows, PyPI has only the CPU builds of PyTorch and ExecuTorch, so the release section says to
add the PyTorch CUDA index there.

The pages also said that 2.15 re-exports in save() by default, never replays an engine on the
shared activation scratch, has no Windows delegate, and cannot pass runtime options from Python.
The 2.15 backport (pytorch#4909) brings the save() and replay fixes and the Windows delegate, and the
ExecuTorch release it pins has the Python option API, so those notes go.

The install command test let a docs command that names the delegate skip --pre. Only the old
2.15 command needed that, so the exception and its test cases go. The case that refuses a
nightly command without --pre now uses the guide's --index-url form.
shoumikhin added a commit to shoumikhin/TensorRT that referenced this pull request Oct 27, 2026
@shoumikhin
docs(executorch): show one install command per channel
e188095 
The install section showed a long command for 2.15 that named the delegate package and four
indexes, and a nightly command with two indexes. Each channel now has one command:

    pip install "torch_tensorrt[executorch]"
    pip install --pre "torch_tensorrt[executorch]" --index-url https://download.pytorch.org/whl/nightly/cu132

A release from PyPI is built for CUDA 13.2, the CUDA version of PyTorch's own PyPI wheel, and its
executorch extra brings ExecuTorch's CUDA 13.2 build and the delegate. A nightly's extra brings
the matching nightlies from the nightly index.

On Windows, PyPI has only the CPU builds of PyTorch and ExecuTorch, so the release section says to
add the PyTorch CUDA index there.

The pages also said that 2.15 re-exports in save() by default, never replays an engine on the
shared activation scratch, has no Windows delegate, and cannot pass runtime options from Python.
The 2.15 backport (pytorch#4909) brings the save() and replay fixes and the Windows delegate, and the
ExecuTorch release it pins has the Python option API, so those notes go.

The install command test let a docs command that names the delegate skip --pre. Only the old
2.15 command needed that, so the exception and its test cases go. The case that refuses a
nightly command without --pre now uses the guide's --index-url form.
shoumikhin added a commit to shoumikhin/TensorRT that referenced this pull request Oct 27, 2026
@shoumikhin
docs(executorch): show one install command per channel
42bdcaf 
The install section showed a long command for 2.15 that named the delegate package and four
indexes, and a nightly command with two indexes. Each channel now has one command:

    pip install "torch_tensorrt[executorch]"
    pip install --pre "torch_tensorrt[executorch]" --index-url https://download.pytorch.org/whl/nightly/cu132

A release from PyPI is built for CUDA 13.2, the CUDA version of PyTorch's own PyPI wheel, and its
executorch extra brings ExecuTorch's CUDA 13.2 build and the delegate. A nightly's extra brings
the matching nightlies from the nightly index.

On Windows, PyPI has only the CPU builds of PyTorch and ExecuTorch, so the release section says to
add the PyTorch CUDA index there.

The pages also said that 2.15 re-exports in save() by default, never replays an engine on the
shared activation scratch, has no Windows delegate, and cannot pass runtime options from Python.
The 2.15 backport (pytorch#4909) brings the save() and replay fixes and the Windows delegate, and the
ExecuTorch release it pins has the Python option API, so those notes go.

The install command test let a docs command that names the delegate skip --pre. Only the old
2.15 command needed that, so the exception and its test cases go. The case that refuses a
nightly command without --pre now uses the guide's --index-url form.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@narendasan narendasan Awaiting requested review from narendasan
@lanluo-nvidia lanluo-nvidia Awaiting requested review from lanluo-nvidia

Assignees

No one assigned

Labels

cla signed component: api [C++] Issues re: C++ API component: api [Python] Issues re: Python API component: build system Issues re: Build system component: tests Issues re: Tests documentation Improvements or additions to documentation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@shoumikhin