[2.15] Bump the ExecuTorch delegate to 0.2.1 and fix the ExecuTorch guide

docsrc/user_guide/runtime_performance/executorch.rst

@@ -21,8 +21,8 @@ program, so the whole model stays on the GPU.
 
 .. note::
 
-    The ExecuTorch delegate runs on Linux, x86_64 and aarch64, and on Windows x86_64. It
-    requires CUDA 13. Ordinary Torch-TensorRT builds keep their separate CUDA 12 support.
+    The prebuilt ExecuTorch delegate runs on Linux, x86_64 and aarch64, and on Windows x86_64.
+    It requires CUDA 13. Ordinary Torch-TensorRT builds keep their separate CUDA 12 support.
 
 ----
 
@@ -68,20 +68,18 @@ README shows.
 
 The delegate is built for the standard Linux wheels (x86-64 and aarch64), and for Windows x86-64.
 On aarch64 it needs glibc 2.35 or newer, the same as TensorRT itself.
-The TensorRT-RTX and Python-only builds have no delegate. To export a coalesced program you also
-need a CUDA toolkit, because ExecuTorch's CUDA backend compiles the leftover operators with
-``nvcc``.
+The JetPack, TensorRT-RTX and Python-only builds have no delegate. To export a coalesced program
+you also need a CUDA toolkit, because ExecuTorch's CUDA backend compiles the leftover operators
+with ``nvcc``.
 
 ----
 
 Compile and Save
 -----------------
 
-The usual ``ir="dynamo"`` path, with two extra arguments to ``torch_tensorrt.save``:
-
-* ``output_format="executorch"`` selects the ``.pte`` writer.
-* ``retrace=False`` keeps the compiled graph as it is. Re-exporting would drop the TensorRT
-  engines before ExecuTorch's partitioner sees them.
+The usual ``ir="dynamo"`` path, with one extra argument to ``torch_tensorrt.save``:
+``output_format="executorch"`` selects the ``.pte`` writer. For this format ``save`` keeps
+the compiled graph as it is rather than re-exporting it, so it needs no example inputs.
 
 .. code-block:: python
 
@@ -98,24 +96,19 @@ The usual ``ir="dynamo"`` path, with two extra arguments to ``torch_tensorrt.sav
         min_block_size=1,
     )
 
-    torch_tensorrt.save(
-        trt_gm,
-        "model.pte",
-        output_format="executorch",
-        arg_inputs=example_inputs,
-        retrace=False,
-    )
+    torch_tensorrt.save(trt_gm, "model.pte", output_format="executorch")
 
-:ref:`executorch_save` covers the rest: dynamic shapes, several methods in one file, a
-zero-copy KV cache, and the two-step ``torch_tensorrt.executorch.export()`` path for
-programs that need work before they are written.
+:ref:`executorch_save` covers the rest: several methods in one file, a zero-copy key-value
+(KV) cache, and the two-step ``torch_tensorrt.executorch.export()`` path for programs that
+need work before they are written.
 
 Coalescing the leftover operators onto the CUDA backend
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-TensorRT has no converter for every ATen operator, and by default the ones it rejects run on
-the CPU, costing a copy each way on every call. Pass a ``CudaPartitioner`` and ExecuTorch's
-CUDA backend compiles them instead, so nothing leaves the GPU:
+TensorRT has no converter for every ATen operator (PyTorch's core operator set), and by
+default the ones it rejects run on the CPU, costing a copy each way on every call. Pass a
+``CudaPartitioner`` and ExecuTorch's CUDA backend compiles them instead, so nothing leaves
+the GPU:
 
 .. code-block:: python
 
@@ -126,8 +119,6 @@ CUDA backend compiles them instead, so nothing leaves the GPU:
         trt_gm,
         "coalesced.pte",
         output_format="executorch",
-        arg_inputs=example_inputs,
-        retrace=False,
         partitioners=[
             CudaPartitioner([CudaBackend.generate_method_name_compile_spec("forward")])
         ],
@@ -140,9 +131,9 @@ where TensorRT cannot take ``erfinv``, the saved program's delegate list reads
 .. warning::
 
     The CUDA backend names its external weight file per device, not per model, so saving two
-    coalesced programs into one directory overwrites the first one's weights. That program
-    still loads, still reports finding its weights, and returns a wrong answer with no error.
-    Give each export its own directory.
+    coalesced programs into one directory overwrites the first one's weights. The first program
+    can then return wrong answers with no error, or fail to load. Give each export its own
+    directory.
 
 .. _executorch_external_engine_data:
 
@@ -211,8 +202,6 @@ copies are pure overhead:
         trt_gm,
         "device_resident.pte",
         output_format="executorch",
-        arg_inputs=example_inputs,
-        retrace=False,
         backend_config=ExecutorchBackendConfig(
             propagate_device_config=PropagateDeviceConfig(
                 skip_h2d_for_method_inputs=True,
@@ -223,20 +212,21 @@ copies are pure overhead:
         ),
     )
 
-Three things are easy to miss:
+Three things to know:
 
 * **Both skip flags need** ``enable_non_cpu_memory_planning=True``, because the copies are
   inserted during device-aware memory planning. Asking for a skip with it off raises.
 * **Inputs must be unplanned**, through ``MemoryPlanningPass(alloc_graph_input=False)``.
   Otherwise the program reserves its own input buffer and the runtime fills it from the
   caller, which puts the copy straight back.
-* **Leave outputs planned if Python will run the program**, so the program's device arena
-  owns the output. Add ``alloc_graph_output=False`` only for a C++ caller that supplies the
-  address with ``Module::set_output``. Python cannot.
+* **The outputs can stay in the program's own GPU memory**, its device arena. To give each output
+  its own buffer instead, also pass ``alloc_graph_output=False``. Then each output needs a GPU
+  buffer before the call: in C++ you set it with ``Module::set_output``, and the Python runtime
+  allocates one for you.
 
-The choice is baked into the file, so feed such a program CUDA tensors. A host tensor still
-gives the right answer, but it copies on every call, which is the cost you exported to
-avoid, and nothing warns you.
+The choice is baked into the file, so feed such a program CUDA tensors. ExecuTorch's Python
+runtime refuses a CPU tensor. From C++, a CPU tensor costs a copy on every call when the first
+delegate to use that input is TensorRT, and fails when it is the CUDA backend.
 
 ----
 
@@ -261,14 +251,16 @@ registers the backend. Everything after it is ExecuTorch's own API:
 If the delegate cannot load, that import raises at once, instead of surfacing later as a
 program that will not load.
 
-A coalesced program needs nothing extra: both backends are registered and the program records
-which parts go where. Torch-TensorRT is only needed to export, not to run.
+A coalesced program needs no extra import: both backends are registered and the program
+records which parts go where. If its export wrote a data file such as ``aoti_cuda_blob.ptd``,
+pass it with ``load_program(..., data_path=Path("aoti_cuda_blob.ptd"))``. Torch-TensorRT is only
+needed to export, not to run.
 
 .. note::
 
     ``torch_tensorrt.load(path, format="executorch")`` still works but is deprecated. It
     copies CUDA inputs to the CPU and supports embedded weights only. Use the Runtime API
-    above instead. A device-resident program requires it.
+    above instead. A device-resident program runs only through the Runtime API.
 
 ----
 
@@ -417,12 +409,12 @@ To put every delegate on a stream of your own, scope a guard over the whole exec
   :ref:`Running a coalesced .pte <executorch_single_stream>` has the same rule for a decode
   loop.
 
-That stream can be a green context, which holds a fixed number of streaming multiprocessors,
+That stream can be a green context, which holds a fixed number of streaming multiprocessors (SMs),
 so the model stays inside that partition and the rest of the GPU is free. Create it with
 ``cuGreenCtxStreamCreate`` and scope the same guard over it. The limit rides the stream, so
 the green context does not have to be made current.
 
-The reference runner has this built in. Pass the SM count:
+The reference runner has this built in. Pass the number of SMs:
 
 .. code-block:: bash
 
@@ -444,8 +436,6 @@ launch many short kernels, where CPU launch work dominates each call.
         trt_gm,
         "model.pte",
         output_format="executorch",
-        arg_inputs=example_inputs,
-        retrace=False,
         use_cuda_graphs=True,
     )
 
@@ -457,17 +447,23 @@ It is not free. Each engine keeps a device buffer per input and output, and ever
 call copies each one in and out. With few kernels, or large inputs and outputs, those copies
 can cost more than the launches they save.
 
-Some engines never replay and still pay the copies: changing shapes, aliased outputs, the
-shared activation scratch below, a green-context stream, GPUs without stream-ordered memory,
-and drivers older than CUDA 12.5. The delegate logs the reason once.
+Some engines never replay: changing shapes, aliased outputs, a green-context stream, GPUs
+without stream-ordered memory, and drivers older than CUDA 12.5. Only changing shapes still pay
+the copies; the others take the ordinary path without the extra buffers. The delegate logs the
+reason once for aliased outputs, missing stream-ordered memory and an old driver.
+
+Engines on the shared activation scratch below do replay, each with its own buffers for its
+inputs and outputs. When the pool moves to a larger buffer, an engine records again, as it does
+after a shape change.
 
 .. warning::
 
     While a recording runs, no other thread may create or destroy a TensorRT engine or
-    execution context, or call a whole-device sync such as ``cudaDeviceSynchronize`` or
-    ``torch.cuda.synchronize()``. Every input shape change starts another recording, so
-    loading everything up front does not end the risk. Leave replay off when other threads
-    may do any of that.
+    execution context outside this delegate, or call a whole-device sync such as
+    ``cudaDeviceSynchronize`` or ``torch.cuda.synchronize()``. Every input shape change, and
+    every growth of the shared activation scratch, starts another recording, so loading
+    everything up front does not end the risk. Leave replay off when other threads may do any
+    of that.
 
 A C++ host can refuse a program's saved request for one load:
 
@@ -525,10 +521,10 @@ and a device that has run a pooled engine must not be reset with ``cudaDeviceRes
 Shared engines
 ^^^^^^^^^^^^^^^
 
-Loading the same program twice in one process, say once per robot arm, used to hold its
-weights in device memory twice. Handles built from the same engine bytes, on the same device
-and with the same weight streaming request, now share one engine. Each still gets its own
-execution context, buffers and lock.
+Loading the same program twice in one process, say once per robot arm, keeps one copy of its
+weights on the GPU. Handles built from the same engine bytes, on the same device and with the
+same weight streaming request, share one engine. Each still gets its own execution context,
+buffers and lock.
 
 This is on by default. On an 8 GB Jetson Orin Nano, loading one robot policy twice grew
 reported GPU memory by 806 MiB without sharing and by 122 MiB with it. The match is found by
@@ -557,19 +553,23 @@ Comparison: .pte vs .pt2
    * - libtorch at runtime
      - Not required
      - Required
-   * - Non-TRT ops
+   * - Operators TensorRT cannot run
      - ExecuTorch CUDA backend, or CPU fallback
      - Compiled by AOTInductor
    * - Platform
-     - Linux + Windows x86_64, CUDA 13
-     - Linux + Windows
+     - Linux (x86_64, aarch64) and Windows x86_64, CUDA 13
+     - Linux
 
 ----
 
 Examples
 ---------
 
 * ``examples/torchtrt_executorch_example/export_static_shape.py`` writes a minimal ``.pte``.
+* ``examples/torchtrt_executorch_example/export_dynamic_shape.py`` exports a model whose batch
+  size can change.
+* ``examples/torchtrt_executorch_example/export_kv_cache_decode.py`` exports a decoder step
+  whose key-value cache stays in memory you own.
 * ``examples/torchtrt_executorch_example/export_coalesced.py`` splits one graph across the
   TensorRT delegate and ExecuTorch's CUDA backend.
 * ``examples/torchtrt_executorch_example/export_device_resident.py`` does the same with no

docsrc/user_guide/runtime_performance/saving_models.rst

@@ -22,7 +22,7 @@ specifying the `output_format` flag. Here are the options `output_format` will a
 * `exported_program` : This is the default. We perform transformations on the graphmodule first and use `torch.export.save` to save the module.
 * `torchscript` : We trace the graphmodule via `torch.jit.trace` and save it via `torch.jit.save`.
 * `PT2 Format` : This is a next generation runtime for PyTorch models, allowing them to run in Python and in C++
-* `executorch` : We lower the graphmodule to an ExecuTorch ``.pte`` program, delegating the TensorRT engines to the ExecuTorch backend. Linux-only; requires the ``executorch`` package.
+* `executorch` : We lower the graphmodule to an ExecuTorch ``.pte`` program, delegating the TensorRT engines to the ExecuTorch backend. Runs on Linux (x86-64 and aarch64) and Windows x86-64; see :ref:`executorch_deployment` to install it.
 
 a) ExportedProgram
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -233,24 +233,7 @@ c) ExecuTorch (.pte)
 
 The ``executorch`` output format lowers the compiled module to an ExecuTorch
 ``.pte`` program, delegating the TensorRT engines to the Torch-TensorRT ExecuTorch
-backend. This CUDA integration links CUDA 13 libraries, so it supports Linux with
-any CUDA 13 PyTorch build and needs an ExecuTorch nightly wheel. Install from the
-channel matching your CUDA, for example a CUDA 13.2 environment:
-
-.. code-block:: bash
-
-    pip install --pre "torch_tensorrt[executorch]" \
-      --extra-index-url https://download.pytorch.org/whl/nightly/cu132
-
-Substitute the channel for your CUDA, such as ``cu134`` for CUDA 13.4. ExecuTorch
-publishes a channel later than PyTorch does, so a very new CUDA minor may not have
-ExecuTorch wheels yet. Keep PyTorch, ExecuTorch and Torch-TensorRT on the
-same CUDA channel. ``--pre`` permits prereleases; it does not request an upgrade
-of an installed package. An existing version that declares the extra can gain
-its missing dependencies, and dependency resolution may change other packages.
-If an older version does not declare the extra, deliberately install the intended
-compatible Torch-TensorRT wheel first. A fresh virtual environment avoids changing
-an existing working stack.
+backend. To install it, follow the Installation section of :ref:`executorch_deployment`.
 
 There are two ways to produce a ``.pte``, and they suit different needs:
 
@@ -499,16 +482,10 @@ the third is refused where it is visible in the ``.pte``:
   surface as wrong results *or* as an illegal memory access. See
   :ref:`Running a coalesced .pte <executorch_single_stream>`.
 
-* **Synchronizing that stream before reading a cache on the host**, for any
-  zero-copy ``.pte`` a runner drives on a caller stream, coalesced or not. A
-  delegate whose aliased outputs are threaded through it reflects each one into
-  its delegate output and so waits for the engine before returning; zero-copy
-  elides those outputs, so there is nothing to reflect and ``execute()`` returns
-  with the engine still running. A single-delegate decode loop owes the
-  synchronization as much as a coalesced program does. It is only the caller
-  stream that brings the duty: with none installed ``execute()`` synchronizes
-  before it returns, as it does whenever a delegate input or output is staged
-  through the host.
+* **Synchronizing that stream before reading a cache on the host**, if the ``.pte`` is
+  coalesced and ExecuTorch's CUDA backend runs after the engine. The TensorRT delegate waits
+  for its engine before it returns, but the CUDA backend may return early. See
+  :ref:`Running a coalesced .pte <executorch_single_stream>`.
 
 * **Sharing one cache between methods.** Zero-copy is per method: it makes each
   method's engine write that method's buffer. Giving a prefill and a decode
@@ -556,43 +533,29 @@ must be pointed at those data files to load them.
 
 .. warning::
 
-    The CUDA backend names its external weight blob per-device (e.g.
-    ``aoti_cuda_blob.ptd``), not per-model, so saving two different coalesced
-    ``.pte`` into the same directory overwrites the blob and the first ``.pte``
-    will fail to load. Save each coalesced model into its own directory.
+    The CUDA backend names its external weight file per device (for example
+    ``aoti_cuda_blob.ptd``), not per model, so saving two coalesced programs into
+    one directory overwrites the first one's weights. The first program can then
+    return wrong answers with no error, or fail to load. Save each coalesced model
+    into its own directory.
 
 .. _executorch_single_stream:
 
-**Running a coalesced .pte: use a single CUDA stream**
-
-A coalesced ``.pte`` runs on more than one backend delegate (the TensorRT delegate
-plus the CUDA backend). By default each backend enqueues its GPU work on its *own*
-CUDA stream, and delegate execution is asynchronous -- a delegate returns after
-*enqueuing* its work, not after it completes. Because separate CUDA streams are not
-ordered relative to one another, at a delegate boundary the consuming delegate can
-begin before the producing delegate's writes have finished, reading incomplete
-data. This is a race: it is intermittent and can surface as wrong results or an
-illegal memory access.
-
-The runtime does not impose a shared stream across delegates, so it is the
-**runner's responsibility** to run all delegates on one CUDA stream. Create a
-single stream and scope ``executorch::extension::cuda::CallerStreamGuard`` over it
-for the duration of execution. That one guard reaches every CUDA-capable
-delegate: they resolve a single shared ``libextension_cuda``, so the TensorRT
-backend and the CUDA backend read the same caller-stream storage. All GPU work is
-then enqueued in order and every cross-boundary dependency is satisfied, while
-execution stays asynchronous.
-
-If the runner reads a delegate's outputs between calls (for example, an
-autoregressive decode loop), synchronize the shared stream before reading: the
-work may still be in flight when ``execute()`` returns, and a host-side copy on
-the default stream will not wait for a non-blocking stream. A model that threads
-its aliased outputs through the delegate is insulated from this in practice:
-reflecting each aliased output into its delegate output makes the delegate wait
-for the engine before it returns. Under
-:ref:`zero-copy KV <executorch_zero_copy_kv>` those outputs are elided, so there
-is nothing to reflect and the delegate returns with the engine still running --
-the synchronization is then the only thing making a host read see the new values.
+**Running a coalesced .pte on one CUDA stream**
+
+A coalesced ``.pte`` runs on two delegates: the TensorRT delegate and ExecuTorch's CUDA
+backend. With no stream chosen, both use ``cudaStreamPerThread``, the calling thread's default
+stream, so a program run from one thread is already in the right order.
+
+To use a stream of your own, put one ``executorch::extension::cuda::CallerStreamGuard`` around
+the whole call. It reaches both delegates, because they share one ``libextension_cuda``. Do not
+give the two delegates different streams: work on separate streams is not ordered, and the
+result is a race that can show up as wrong values or an illegal memory access.
+
+The TensorRT delegate waits for its own work before it returns. ExecuTorch's CUDA backend can
+return while its work is still queued, so when it runs last, ``execute()`` can return before the
+outputs are ready. Synchronize the stream before you read an output on the host, for example
+between the steps of a decode loop.
 
 **ExecuTorch lowering options**