[pytorchbot]

This is a draft because the page describes the intended end state of Torch-TensorRT 2.15. Some links go to sections of the Torch-TensorRT guide that exist only after pytorch/TensorRT#4927 merges and its docs site is rebuilt. And the examples rely on 2.15 changes from pytorch/TensorRT#4909, such as torch_tensorrt.save keeping the compiled graph by default.

What is wrong today

The TensorRT backend page has drifted away from Torch-TensorRT, the project that builds and ships the TensorRT delegate.

• It points readers at a nightly build from early October.
• It says CUDA graph replay never runs for engines on the shared activation scratch. Newer builds run it.
• It repeats most of the Torch-TensorRT guide, so every delegate change is made twice.
• The backends table lists TensorRT as Linux only.

The CUDA backend page and the getting started page also disagree with the TensorRT page and with the code: on outputs kept on the GPU, on what happens when a planned input gets GPU memory, on CUDA versions, GPUs and Python versions, and on Windows.

What this change does

The TensorRT page keeps the usual backend sections, says less in each, and shows more code:

• Features, target requirements, and development requirements with the one-line install command.
• Export, including sending the operators TensorRT cannot compile to the CUDA backend, and keeping inputs and outputs on the GPU.
• Runtime integration from Python, and a complete C++ project with the commands to build and run it.
• A short list of performance options that link to the Torch-TensorRT guide.

It describes Torch-TensorRT 2.15 and newer, and marks the two features that need a newer build.

A second commit fixes the CUDA backend page, the getting started page and one docstring on the points above. To keep data on the GPU, all pages now leave inputs and outputs unplanned, which copies the least.

What was tested

On Linux x86-64 with an H100 GPU, an earlier version of the TensorRT page was run end to end: every Python block in order with nightly Torch-TensorRT, and the C++ example built and run against the installed packages.

With nightly packages, the GPU input and output recipe was run from Python for both backends, and the CUDA page's C++ lines were built and run. A profiler showed one GPU copy fewer per Python call than with planned outputs.

The other fixes were checked only by reading them against the sources. Windows was not tested.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24211

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 783c959 with merge base 4db778e ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.