Torch-TensorRT 2.13.0 Linux x86-64 and Windows targets

PyTorch 2.13, CUDA 13.0/13.2, TensorRT 11.0, Python 3.10~3.13

Torch-TensorRT Wheels are available:

x86-64 Linux and Windows: CUDA 13.0/13.2 + Python 3.10-3.13 is Available via PyPI

• https://pypi.org/project/torch-tensorrt/
• Available via PyTorch index: https://download.pytorch.org/whl/torch-tensorrt

aarch64 SBSA Linux and Jetson Thor and Orin: CUDA 13.0/13.2 + Python 3.10–3.13 + Torch 2.13 + TensorRT 11.0

• Available via PyPI: https://pypi.org/project/torch-tensorrt/
• Available via PyTorch index: https://download.pytorch.org/whl/torch-tensorrt

Torch-TensorRT-RTX 2.13.0 Linux x86-64 and Windows targets

PyTorch 2.13, CUDA 13.0/13.2, TensorRT-RTX 1.5, Python 3.10~3.13

Torch-TensorRT-RTX Wheels are available:

x86-64 Linux and Windows: CUDA 13.0/13.2 + Python 3.10-3.13 is Available via PyPI

• https://pypi.org/project/torch-tensorrt-rtx/

CUDA 13.0/13.2 + Python 3.10-3.13 is also Available via Pytorch Index

• https://download.pytorch.org/whl/torch-tensorrt-rtx

New Features:

GQA/MQA and Decode-Phase Attention Support

Extends Group Query Attention (GQA) and Multi-Query Attention (MQA) alongside traditional Multi-Head Attention (MHA), enabling efficient inference for modern large language models. This release relaxes SDPA validator constraints to allow variable head-count configurations, adds native support for decode-phase attention where query and key sequences differ, and introduces comprehensive HLO-level test coverage across all three attention kernel variants (standard, flash, efficient) with multiple precision levels (fp16, bf16, fp32). These changes unlock optimized inference patterns for LLMs using GQA/MQA while maintaining full backward compatibility with existing MHA-based models.

FP8 Attention Quantization Support

Supports FP8 quantization of the attention in both TensorRT's IAttention path and decomposed attention path (decompose_attention=True). Working with NVIDIA ModelOpt, it detects when attention inputs (Q, K, V) are quantized to FP8 and configures TensorRT's native FP8 MHA kernel for optimal performance on FP8-capable hardware (Hopper and newer GPUs). A ViT quantization example demonstrates both IAttention and decomposed attention paths.

Caller-Selected CUDA Streams and Green Context Support

Enables TensorRT inference on caller-selected CUDA streams across both Torch-TensorRT and ExecuTorch. The PyTorch runtime now respects active non-default streams, while ExecuTorch introduces a scoped CudaStreamGuard API. This supports CUDA Green Contexts, NCCL streams, and custom GPU scheduling without changing default-stream behavior. CUDA Graphs are recaptured when streams change, and completion events safely protect asynchronous execution-context reuse.

TensorRT 11 Support

Torch-TensorRT 2.13 adds support for NVIDIA TensorRT 11, with release packages updated to TensorRT 11.0. The integration has been updated for TensorRT 11 APIs across compilation, runtime execution, packaging, and distributed inference. Highlights:

• TensorRT 11.0 support for Python and C++ workflows.
• Updated CUDA 12 and CUDA 13 package dependencies for TensorRT 11.
• Updated distributed and NCCL-based operators for TensorRT 11.
• Improved scaled dot-product attention accuracy using TensorRT 11 fixes.
• Updated build configurations, CI coverage, and prebuilt release artifacts for TensorRT 11. TensorRT 11 is the default supported TensorRT version for Torch-TensorRT 2.13 release packages.

ExecuTorch Support

This release delivers a pure ExecuTorch TensorRT backend for deploying TensorRT-accelerated models in ExecuTorch .pte format. The native runtime path no longer depends on Torch or LibTorch, enabling lightweight C++ deployment using only ExecuTorch and TensorRT runtime components. Users can continue exporting TensorRT-compiled ExportedProgram and FX models with: torch_tenso

Torch-TensorRT 2.12.1 Linux x86-64 and Windows targets

PyTorch 2.12, CUDA 13.0, TensorRT 10.16, Python 3.10~3.13

Torch-TensorRT Wheels are available:

x86-64 Linux and Windows: CUDA 13.0 + Python 3.10-3.13 is Available via PyPI

• https://pypi.org/project/torch-tensorrt/

aarch64 SBSA Linux and Jetson Thor CUDA 13.0 + Python 3.10–3.13 + Torch 2.12 + TensorRT 10.16

• Available via PyPI: https://pypi.org/project/torch-tensorrt/
• Available via PyTorch index: https://download.pytorch.org/whl/torch-tensorrt

Jetson Orin

• no torch_tensorrt 2.9/2.10/2.11/2.12 release for Jetson Orin
• please continue using torch_tensorrt 2.8 release

Torch-TensorRT-RTX 2.12.1 Linux x86-64 and Windows targets

PyTorch 2.12, CUDA 13.0, TensorRT-RTX 1.4, Python 3.10~3.13

Torch-TensorRT-RTX Wheels are available:

x86-64 Linux and Windows: CUDA 13.0 + Python 3.10-3.13 is Available via PyPI

• https://pypi.org/project/torch-tensorrt-rtx/

CUDA 13.0 + Python 3.10-3.13 is also Available via Pytorch Index

• https://download.pytorch.org/whl/torch-tensorrt-rtx

What's Changed

• release 2.12 MD TRT fix by @apbose in https://github.com/pytorch/TensorRT/pull/4291
• fix: cap tensorrt_rtx requirement below 1.5.0 on release/2.12 by @tp5uiuc in https://github.com/pytorch/TensorRT/pull/4298
• fix(lowering): use factored sqrt form for complex acosh rewrite by @tp5uiuc in https://github.com/pytorch/TensorRT/pull/4300
• setup_nccl_for_torch_tensorrt change by @apbose in https://github.com/pytorch/TensorRT/pull/4303

Full Changelog: https://github.com/pytorch/TensorRT/compare/v2.12.0...v2.12.1

Torch-TensorRT 2.12.0 Linux x86-64 and Windows targets

PyTorch 2.12, CUDA 13.0, TensorRT 10.16, Python 3.10~3.13

Torch-TensorRT Wheels are available:

x86-64 Linux and Windows: CUDA 13.0 + Python 3.10-3.13 is Available via PyPI

• https://pypi.org/project/torch-tensorrt/

aarch64 SBSA Linux and Jetson Thor CUDA 13.0 + Python 3.10–3.13 + Torch 2.12 + TensorRT 10.16

• Available via PyPI: https://pypi.org/project/torch-tensorrt/
• Available via PyTorch index: https://download.pytorch.org/whl/torch-tensorrt

Jetson Orin

• no torch_tensorrt 2.9/2.10/2.11/2.12 release for Jetson Orin
• please continue using torch_tensorrt 2.8 release

Torch-TensorRT-RTX 2.12.0 Linux x86-64 and Windows targets

PyTorch 2.12, CUDA 13.0, TensorRT-RTX 1.4, Python 3.10~3.13

Torch-TensorRT-RTX Wheels are available:

x86-64 Linux and Windows: CUDA 13.0 + Python 3.10-3.13 is Available via PyPI

• https://pypi.org/project/torch-tensorrt-rtx/

CUDA 13.0 + Python 3.10-3.13 is also Available via Pytorch Index

• https://download.pytorch.org/whl/torch-tensorrt-rtx

Native Distributed Collectives

In prior versions of Torch-TensorRT, distributed operators were backed by kernels provided by TensorRT-LLM that the user needed to manually install. With Torch-TensorRT 2.12, many of these operations are natively supported which means in deployment, only Torch-TensorRT needs to be installed.

The distributed infrastructure is designed to operate on top of torch.distributed. Once a graph is sharded, traced and compiled. The torch.distributed device mesh can be passed to torch-tensorrt compiled modules using the following API:

trt_model = torch.compile(model, backend="torch_tensorrt", ...)
_ = trt_model(inp)  # warmup — triggers engine build
dist.barrier()

with torch_tensorrt.distributed.distributed_context(dist.group.WORLD, trt_model) as dmodel:
    output = dmodel(inp)

dist.destroy_process_group()
os._exit(0)

This can be done at compile time as well

with torch_tensorrt.distributed.distributed_context(tp_group):
    trt_model = torch.compile(model, backend="torch_tensorrt", ...)
    output = trt_model(inp)

Note: use_distributed_trace is no longer necessary to compile multi-device models, torch-tensorrt will automatically recognize distributed collectives and set the setting for the user.

torchtrtrun

The distributed operations utilize the NCCL version distributed by PyTorch which must be added to LD_PRELOAD before importing torch-tensorrt. As a convience we provide a tool torchtrtrun which is analogous to torchrun that configures these libraries correctly in addition to allowing users to launch models distributed across multiple nodes.

For example:

# Node 0 (rank 0):
torchtrtrun --nproc_per_node=1 --nnodes=2 --node_rank=0 \
  --rdzv_endpoint=<node0-ip>:29500 \
  tensor_parallel_llama_multinode.py

# Node 1 (rank 1):
torchtrtrun --nproc_per_node=1 --nnodes=2 --node_rank=1 \
  --rdzv_endpoint=<node0-ip>:29500 \
  tensor_parallel_llama_multinode.py

Serialization and torch.export

Models sharded and then exported can be compiled and saved to disk before being loaded on a deployment system. By default these modules attempt to bind to the default torch distributed device mesh. If there are multiple valid device meshes availble, the above API can be used to set a specific one to execute the engine.

More information on torch-tensorrt distributed collective support can be found here: https://docs.pytorch.org/TensorRT/tutorials/deployment/distributed_inference.html#multinode-inference

More information on native multi-device collectives can be found here: https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/work-with-transformers.html#multi-device-attention-preview-feature

ExecuTorch Support

Torch-TensorRT 2.12 introduces initial ExecuTorch integration for exporting and ru

Torch-TensorRT 2.11.0 Linux x86-64 and Windows targets

PyTorch 2.11, CUDA 12.6 12.8 12.9, 13.0, TensorRT 10.15, Python 3.10~3.13

Torch-TensorRT Wheels are available:

x86-64 Linux and Windows: CUDA 13.0 + Python 3.10-3.13 is Available via PyPI

• https://pypi.org/project/torch-tensorrt/

CUDA 12.6/12.8/12.9/13.0 + Python 3.10-3.13 is also Available via Pytorch Index

• https://download.pytorch.org/whl/torch-tensorrt

aarch64 SBSA Linux and Jetson Thor CUDA 13.0 + Python 3.10–3.13 + Torch 2.11 + TensorRT 10.15

• Available via PyPI: https://pypi.org/project/torch-tensorrt/
• Available via PyTorch index: https://download.pytorch.org/whl/torch-tensorrt

Jetson Orin

• no torch_tensorrt 2.9/2.10/2.11 release for Jetson Orin
• please continue using torch_tensorrt 2.8 release

Torch-TensorRT-RTX 2.11.0 Linux x86-64 and Windows targets

PyTorch 2.11, CUDA 12.9, 13.0, TensorRT-RTX 1.3, Python 3.10~3.13

Torch-TensorRT-RTX Wheels are available:

x86-64 Linux and Windows: CUDA 13.0 + Python 3.10-3.13 is Available via PyPI

• https://pypi.org/project/torch-tensorrt-rtx/

CUDA 12.9/13.0 + Python 3.10-3.13 is also Available via Pytorch Index

• https://download.pytorch.org/whl/torch-tensorrt-rtx

note: tensorrt-rtx 1.3 wheel is not in pypi yet, so please download the tarball from https://developer.nvidia.com/tensorrt-rtx install the wheel from tarball

IAttention Layer

In this release, TensorRT's native IAttention layer is used by default to handle various attention-related ATen ops, including SDPA, Flash-SDPA, Efficient-SDPA, and cuDNN-SDPA. This integration enables more efficient execution and can improve model performance. To explicitly enable this behavior, set decompose_attention=False in the compile() function. When enabled, the native TensorRT implementation is utilized to achieve optimized attention computation. However, due to current TensorRT limitations, certain operations such as compute_log_sumexp and Grouped Query Attention (GQA) are not yet supported. If these cases are encountered, an informational prompt will be displayed during compilation. Alternatively, you can set decompose_attention=True to decompose the attention ops into multiple basic ATen ops. Although this approach may not achieve the same level of performance optimization, it offers broader operator coverage and greater compatibility across different model architectures.

Improvements to the Symbolic Shape System

Two key improvements have been made to the symbolic shape system used to track mutations on dynamic dimensions through out the body of the graph.

1. A shape prop formula is recorded as metadata in for every compiled engine.

Now instead of needing to instantiate the engine in order to do key tasks like serialization, retracing or similar tasks which require fake tensor propagation, we record the shape relation between inputs and outputs for every TRT subgraph at compile time and store it as metadata. The torch.ops.tensorrt.execute_engine meta kernel now just replays this function in the new shape environment. This should enable more seamless integration with the rest of the torch.compile ecosystem as meta kernels in Torch-TensorRT will now work in the same way as other meta kernels.

2. For unbounded shape ranges, we now select sane defaults

Dynamic shapes are lazily inserted into Dynamo graphs. This is most noticeable when using Torch-TensorRT as a backend for torch.compile. Here when the first inference call is made to a boxed function

trt_mod = torch.compile(mod, backend="tensorrt")
trt_mod(*inputs)

The shapes of intermediate tensors are considered static and derived eagerly from the shapes of input tensors.

If input shapes then change

trt_mod(*other_sized_inputs)

TorchDynamo will start marking dimensions where shapes differ as dynamic dimensions. However, it does not assume upper bounds and criticall