From 13e0a807595df6c7971f3d4e28206d8e06e0113a Mon Sep 17 00:00:00 2001 From: Anthony Shoumikhin Date: Tue, 27 Oct 2026 13:30:13 -0700 Subject: [PATCH] Drop the "newer than 2.15" marks from the TensorRT backend page The TensorRT backend page marked two features as needing a Torch-TensorRT build newer than 2.15: keeping the engines in a separate .ptd data file, and optimization profiles. Both are in Torch-TensorRT 2.15, so the marks sent readers to a nightly build they do not need. The page now lists both as plain features, like the Torch-TensorRT guide does, and says that the engines go in the .pte by default. --- .../backends/tensorrt/tensorrt-overview.md | 31 +++++++++---------- 1 file changed, 14 insertions(+), 17 deletions(-) diff --git a/docs/source/backends/tensorrt/tensorrt-overview.md b/docs/source/backends/tensorrt/tensorrt-overview.md index 9b4ab7b43ca..411cc262961 100644 --- a/docs/source/backends/tensorrt/tensorrt-overview.md +++ b/docs/source/backends/tensorrt/tensorrt-overview.md @@ -3,8 +3,8 @@ The TensorRT backend runs ExecuTorch models on NVIDIA GPUs with [TensorRT](https://developer.nvidia.com/tensorrt), NVIDIA's inference library. At export time, TensorRT compiles the parts of the model it supports into engines (programs optimized for your -GPU), and the engines are stored in the `.pte` file. At run time, the TensorRT delegate hands each -engine back to TensorRT. +GPU), and by default the engines are stored in the `.pte` file. At run time, the TensorRT delegate +hands each engine back to TensorRT. The delegate is built and shipped by the [Torch-TensorRT](https://github.com/pytorch/TensorRT) project, not by ExecuTorch. You export with Torch-TensorRT, and you run with the normal ExecuTorch @@ -12,13 +12,11 @@ runtime and its usual APIs. The [Torch-TensorRT ExecuTorch guide](https://docs.pytorch.org/TensorRT/user_guide/runtime_performance/executorch.html) covers every option in detail. -This page describes Torch-TensorRT 2.15 and newer. Features marked "newer than 2.15" need a later -Torch-TensorRT build, such as a -[nightly build](https://docs.pytorch.org/TensorRT/user_guide/runtime_performance/executorch.html#nightly-builds). +This page describes Torch-TensorRT 2.15 and newer. ## Features -- **TensorRT engines inside a `.pte`.** One file holds the program and its engines. +- **TensorRT engines inside a `.pte`.** By default, one file holds the program and its engines. - **Works with the CUDA backend in one program.** Operators that TensorRT cannot compile can go to the ExecuTorch [CUDA backend](../cuda/cuda-overview.md) instead of the CPU, so the whole model stays on the GPU. @@ -36,8 +34,9 @@ Torch-TensorRT build, such as a - **Weight streaming.** An engine can keep part of its weights in CPU memory. - **Zero-copy KV cache.** The engine of a large language model (LLM) can update its key-value (KV) cache in place. -- **Newer than 2.15:** engines in a separate data file, and optimization profiles (tunings for - different input sizes) chosen per call. +- **Engines in a separate data file.** The engines can be kept in a `.ptd` file next to the `.pte`. +- **Optimization profiles.** One engine can carry tunings for different input sizes, and a C++ + application picks one per call. ## Target Requirements @@ -429,15 +428,15 @@ shows how to set the options that apply at run time, from C++ or from Python. Set how many bytes stay on the GPU with `weight_streaming_budget_per_engine` in `torch_tensorrt.save`, or with the `weight_streaming_budget` load option. The load option takes the number of bytes as a decimal string, or as an integer below 2 GiB. -- [**Engines in a separate data file**](https://docs.pytorch.org/TensorRT/user_guide/runtime_performance/executorch.html#keep-the-engines-in-a-separate-file) - (newer than 2.15). Pass `external_engine_data="engines"` to `torch_tensorrt.save` to write the - engines to `engines.ptd` instead of the `.pte`, and pass that file when you load the program. +- [**Engines in a separate data file.**](https://docs.pytorch.org/TensorRT/user_guide/runtime_performance/executorch.html#keep-the-engines-in-a-separate-file) + Pass `external_engine_data="engines"` to `torch_tensorrt.save` to write the engines to + `engines.ptd` instead of the `.pte`, and pass that file when you load the program. The Python loader takes one data file, so for a program that also uses the CUDA backend, pass `external_engine_data="aoti_cuda_blob"` to share its file. -- [**Optimization profiles**](https://docs.pytorch.org/TensorRT/user_guide/runtime_performance/executorch.html#optimization-profiles) - (newer than 2.15). One engine can be tuned for several ranges of input shapes, for example one - for decoding one token at a time and one for a long prompt. A C++ application picks the profile for - each call with `torch_tensorrt::executorch_backend::OptimizationProfileGuard`, from the +- [**Optimization profiles.**](https://docs.pytorch.org/TensorRT/user_guide/runtime_performance/executorch.html#optimization-profiles) + One engine can be tuned for several ranges of input shapes, for example one for decoding one + token at a time and one for a long prompt. A C++ application picks the profile for each call + with `torch_tensorrt::executorch_backend::OptimizationProfileGuard`, from the `torch_tensorrt/executorch/TensorRTBackend.h` header. Without the guard, every engine runs profile 0, and Python cannot choose a profile. The Python packages do not carry that header, and the guide explains where to get it. @@ -450,6 +449,4 @@ shows how to set the options that apply at run time, from C++ or from Python. - [Export a model that keeps its data on the GPU](https://github.com/pytorch/TensorRT/blob/main/examples/torchtrt_executorch_example/export_device_resident.py) - [Export a decoder step whose KV cache stays in memory you own](https://github.com/pytorch/TensorRT/blob/main/examples/torchtrt_executorch_example/export_kv_cache_decode.py) - [Export one engine with two optimization profiles](https://github.com/pytorch/TensorRT/blob/main/examples/torchtrt_executorch_example/export_multi_profile.py) - (newer than 2.15) - [C++ and Python reference runners](https://github.com/pytorch/TensorRT/tree/main/examples/executorch_reference_runner) - (the two multi-profile programs need a build newer than 2.15)