Triton Inference Server

The Triton Inference Server provides a cloud inferencing solution optimized for both CPUs and GPUs. The server provides an inference service via an HTTP or GRPC endpoint, allowing remote clients to request inferencing for any model being managed by the server. For edge deployments, Triton Server is also available as a shared library with an API that allows the full functionality of the server to be included directly in an application.

• The 26.09 DLFW containers introduce initial support for the NVIDIA Rubin GPU architecture.
• Core: Flagged the developer_tools C API wrapper as deprecated in the documentation.
• Client: Fixed reading of cudaIpcMemHandle_t bytes via getPtr() for compatibility with cuda-bindings 13.x.
• Restored preserve_ordering behavior by reserving the completion-queue slot at enqueue time.
• Fixed default_model_filename autofill incorrectly applying to Python-runtime PyTorch models.
• OpenVINO backend: Fixed the OpenVINO GPU plugin build to ship libOpenCL.so.1.
• Python backend: Fixed a segfault during shutdown when using Triton metrics, and fixed shared-memory cleanup on forced shutdown.
• PyTorch backend: Fixed swapped ReportStatistics timestamp arguments and added a guard to the AOTI package loader in the PT2/AOTI backend.
• Excluded libtritonserver from auditwheel processing in the tritonfrontend wheel build.
• Fixed a crash: shared-memory inference requests now return a graceful UNAVAILABLE error instead of a NULL-dereference crash.
• SageMaker integration: Fixed cross-model invoke/unload handling for multi-model endpoints.
• OpenAI-compatible frontend: Improved cancellation handling for response streams.
• Fixed an unhandled exception when parsing inference header length in the HTTP frontends.
• Build: Updated the C++ standard from C++17 to C++20.
• Build: Migrated the tritonfrontend wheel to hatchling with PEP 639 metadata.

• The Triton TensorRT-LLM Backend container image is not included in this release.
• vLLM's v0 API and Ray are affected by vulnerabilities. Users should consider their own architecture and mitigation steps which may include but should not be limited to: *

Triton Inference Server

The Triton Inference Server provides a cloud inferencing solution optimized for both CPUs and GPUs. The server provides an inference service via an HTTP or GRPC endpoint, allowing remote clients to request inferencing for any model being managed by the server. For edge deployments, Triton Server is also available as a shared library with an API that allows the full functionality of the server to be included directly in an application.

• Added a log callback option to the tritonserver C and Python APIs and to the common Logger, allowing an embedding application to route Triton log records into its own logging stack instead of a file or stream.
• Fixed dynamic-batcher starvation caused by waiting_consumer_count drift, where the scheduler could stop dispatching work despite queued requests.
• Restored preserve_ordering semantics by reserving the completion-queue slot at enqueue time rather than at completion.
• Core: Improved model-readiness reporting in TRITONSERVER_ServerModelIsReady — non-availability errors are now returned and unresolved models correctly report ready=false.
• Model loading now catches exceptions rather than propagating them out of the load path, and uses logic_error so both invalid_argument and out_of_range are handled uniformly.
• TensorRT backend: Fixed CUDA graph execution for non-batching models.
• vLLM backend: Corrected JSON input parsing.
• OpenVINO backend: Enabled OpenVINO model generation on SBSA (aarch64).
• Python client: Removed the upper bound on the grpcio package requirement, so client installations are no longer pinned to an outdated gRPC.
• Build: Added experimental build presets, allowing a build to be driven from a file via --build-presets-file instead of a long command line.
• Build: Default build parallelism is now capped by available memory, with matching caps for the ONNX Runtime and OpenVINO backend builds, preventing out-of-memory failures during container builds.
• Aligned protobuf and related dependencies with the gRPC 1.81.1 bump across the server, core, client, third-party and Model Analyzer components.
• Documentation: Added a RHEL / manylinux build tutorial.

• The

Triton Inference Server

The Triton Inference Server provides a cloud inferencing solution optimized for both CPUs and GPUs. The server provides an inference service via an HTTP or GRPC endpoint, allowing remote clients to request inferencing for any model being managed by the server. For edge deployments, Triton Server is also available as a shared library with an API that allows the full functionality of the server to be included directly in an application.

• Added a streaming tool-call parse buffer limit to the OpenAI-compatible frontend to prevent excessive memory usage during streaming tool calls.
• PyTorch backend: Enabled PyTorch 2 batching and added support for loading NV embedding layers.
• HSTU generative-recommender is supported using the PyTorch AOTI serving path.
• TensorRT backend: Added multi-device (multi-GPU) inference support.
• Python client: Preserved the output datatype for empty (zero-element) tensors.
• Core: Improved model-readiness reporting in TRITONSERVER_ServerModelIsReady — non-availability errors are now returned, unresolved models correctly report ready=false, and the readiness lookup is * resolved by ModelIdentifier.
• Python backend: Fixed a deadlock in decoupled BLS and a use-after-free in the async_exec pybind callback.
• Security: Restriction header values are now redacted from the OpenAI frontend startup log, and the inference restriction is enforced in the SageMaker MME invoke handler.

• Avoid the specific APIs/flows (low-level OCB, BIO_f_linebuffer with short writes, CMS password-based decryption, low-level GF(2^m) with untrusted params) for many linux binaries, in order to avoid exposure to known issues in OpenSSL 1.1.1.
• vLLM's v0 API and Ray are affected by vulnerabilities. Users should consider their own architecture and mitigation steps which may include but should not be limited to:
  • Do not expose Ray executors and vLLM hosts to a network where any untrusted connections might reach the host.
  • Ensure that only the other vLLM hosts are able to connect to the TCP port used for the XPUB socket. Note that the port used is random.
• When using Valgrind or other leak detection tools on AGX-

Triton Inference Server

The Triton Inference Server provides a cloud inferencing solution optimized for both CPUs and GPUs. The server provides an inference service via an HTTP or GRPC endpoint, allowing remote clients to request inferencing for any model being managed by the server. For edge deployments, Triton Server is also available as a shared library with an API that allows the full functionality of the server to be included directly in an application.

• (Breaking) Removed deprecated Windows support: Dropped the Windows server build and removed Windows from the core build and documentation.
• (Breaking) Python client BF16 handling: The client now requires ml_dtypes.bfloat16 arrays for BF16 input/output tensors and no longer casts to/from np.float32. Scripts that passed BF16 data as np.float32 arrays must switch to ml_dtypes.bfloat16.
• TensorRT backend: Added multi-device (multi-GPU) inference support.
• PyTorch backend: Enabled PyTorch 2 dynamic batching.
• OpenAI-compatible frontend: Added an HTTP request body size limit.
• Fixed a server crash when loading models from S3 after an idle-connection timeout.
• Security: Hardened model environment unpacking and file-path handling against path traversal.
• Improved request validation in the SageMaker endpoint.
• Changed the container install tree under /opt/tritonserver/ to stay root-owned, with the triton-server user accessing it via group/other permissions.
• Fixed a server crash caused by improper error handling.

• Avoid the specific APIs/flows (low-level OCB, BIO_f_linebuffer with short writes, CMS password-based decryption, low-level GF(2^m) with untrusted params) for many linux binaries, in order to avoid exposure to known issues in OpenSSL 1.1.1.
• vLLM's v0 API and Ray are affected by vulnerabilities. Users should consider their own architecture and mitigation steps which may include but should not be limited to:
  • Do not expose Ray executors and vLLM hosts to a network where any untrusted connections might reach the host.
  • Ensure that only the other vLLM hosts are able to connect to the TCP port used for the XPUB socket. Note that the por