Python package version

The Python wheels for this release are published as 1.4.0rc2.post1. Container images and documentation remain at 1.4.0rc2. Use 1.4.0rc2.post1 when installing the Python package.

The wheel-only patch removes a direct Git dependency that prevented PyPI publication (#21128) and updates the Boost download mirror used during builds (#21133). Patched wheel source: v1.4.0rc2.post1, commit 43f8f70da6415a3b06f91f9ebaa4fb7611ab62cc.

Highlights

• Known Issues

  • Tensor-parallel serving with RPC orchestration may hang during model initialization when reusing an MPI worker pool across threads.

• Model Support

  • Add Cosmos3-Edge Reasoner support for visual generation workloads #20307
  • Add GLM-5.2 NVFP4 low-latency decode on Rubin through Modeling V2 #20517
  • Enable MiniMax-M3 on Rubin with NVFP4 KV exemptions and paged FMHA #20167
  • Expand Qwen3.8-Flash-Next Modeling V2 support for MTP, video, and quantized KV caches #20276
  • Add SM103 Modeling V2 defaults for GPT-OSS-120B and DeepSeek-R1-0528 NVFP4 #20267
  • Qualify dense Phi3 models for ModelExpress integration #19997
  • Support static FP8 Cosmos3 parallelism and four-step distilled checkpoints #20007
  • Support FLUX.1-dev static NVFP4 with per-tensor FP8 attention on SM103 #18396

• API

  • Migrate LlmRequest fields from C++ to Python (BREAKING) #20388
  • Drop KV cache manager V1 support from transceiver V2 (BREAKING) #20129
  • Remove obsolete executor backend compatibility paths (BREAKING) #20043
  • Enable zero-copy mode for Mooncake Store (BREAKING) #20108
  • Deprecate n-gram and user-provided speculative decoding modes #20438
  • Add configurable FP32 LM-head logits through lm_head_output_dtype #19850
  • Consolidate prompt rendering into standalone serve/render support for Dynamo #19964
  • Define backend contracts and share lifecycle enforcement for KV cache transfers #20117, #19822
  • Add a blob-store driver boundary with an in-memory test implementation #20217

• Feature

  • Add native retention and cross-turn generation-block reuse to KV cache V2 #20086, #19710
  • Commit sliding-window prefixes at fork points and account for connector-provided reuse blocks #19877, #20180
  • Add first-N/last-N mixed-precision KV caching using native sliding-window attention #20452
  • Support FP4 MLA in disaggregated serving deployments #20544
  • Add MXFP8 refitting with piecewise CUDA graphs and Ray collectives #20075
  • Optimize DeepSeek-V4 and DSpark on Rubin, including fused RMSNorm/RoPE kernels #19961, #20366
  • Add Kimi K3 fabric gathering, TCPStore rendezvous, and managed KDA replay buffers #20576
  • Integrate FlashInfer KDA prefill with fused convolution #20653
  • Enable shared CUDA graph setup for Qwen3-family vision-language encoders #20213
  • Schedule multimodal encoders by prefill chunk and reduce Gemma attention-mask memory #19884, #20405
  • Improve Qwen-Image Ulysses performance and add MiniMax-H3 asynchronous Ulysses execution #17147, #20333
  • Add PrimsTS SageAttention for VisualGen VSA and SOL workloads #20224
  • Parallelize MiniMax-M3 indexer scoring and use BF16 tensor-core router GEMMs #19975, #20699
  • Accelerate GLM-5.3-Flash sparse attention, verification offsets, mHC, quantization, and routing #20454, #20389, #20390
  • Improve DSA paged-MQA logits kernels and default indexing to CuTe DSL #19976
  • Optimize one-model speculative decoding host scheduling, target logits, and KDA draft records #19989
  • Optimize GVR V2 for large batches and short sequences #20106
  • Fuse SwiGLU into BF16 gate/up GEMM epilogues on SM100 #19773
  • Compile Mamba replay-input preparation and vectorize SID-GR final beam assembly #20688, #20166
  • Support required tool choice for Nemotron Super 3.5 VL #20422

• Fix

  • Prevent global stop-word buffer growth in the PyTorch sampler #20679
  • Preserve o

Highlights

• Known Issues

  • Disaggregated serving with the default Python KV cache transceiver may fail generation requests with a KV cache transfer error after streaming requests are cancelled (observed on A100).
  • The OpenAI JSON-schema serving example (examples/serve/openai_completion_client_json_schema.py) may return an unbounded response with Qwen3 models.

• Model Support

  • Add DeepSeek-V4.1-Flash support #19727
  • Support NVFP4 GLM-5.3-Flash checkpoints #20127
  • Add Qwen3.8-Flash-Next NVFP4 as an experimental Modeling V2 target #20143
  • Support LiLiCorr draft checkpoints in DFlash #19856
  • Remove the automatic W4A16 fallback for Nemotron-H NVFP4 checkpoints on Hopper #15772

• API

  • Default to KV cache manager V2 and the Python KV cache transceiver (BREAKING) #17495
  • Remove unused LlmRequest fields (BREAKING) #19967
  • Replace the C++ ArrayView API with std::span #20205
  • Select Modeling V2 through LLM configuration and arguments #20077
  • Publish serving configuration schemas #19906
  • Add a KV cache lending API for transfer backends #19722
  • Expose native KV event streaming and routing load #19440
  • Support runtime sleep and wakeup through LLM and trtllm-serve, including MiniMax-H3 #19016, #19604
  • Unify compiled-artifact cache configuration through TRTLLM_CACHE_DIR #18897
  • Make KV cache V2 pool rebalancing parameters configurable #19077
  • Reorganize VisualGen attention modules, keeping deprecated import paths and lazy exports #20214, #20215, #20218

• Feature

  • Integrate the Mooncake connector with KV cache manager V2 #19171
  • Fill the KV cache V2 host tier progressively #19650
  • Add RoPE precision options for NVFP4 cold-page compression #19391
  • Honor per-request seeds in batched TorchSampler and one-model speculative acceptance #19719, #20147
  • Add ragged DSpark verification with confidence-based policy and calibrated scoring #19707, #19818, #20071
  • Add deterministic FP32 GVR V2 top-K decoding and reduce its overflow overhead #20100, #20290
  • Add an opt-in setting to compile only piecewise CUDA graph forwards #18366
  • Support Rubin locality-domain execution and extend SM107 and multi-node MegaMoE initialization #19272, #19183
  • Optimize Kimi K3 on Rubin and avoid cuBLAS fallback in KDA prefill #19960, #20084
  • Support packed query heads and compact inputs in Blackwell CuTe DSL MLA #19338
  • Add PrimTS dense BF16, mixed BF16/FP8, and FP8 context attention paths #20006
  • Use paged attention for MiniMax-M3 dense layers and cut NVFP4 KV host overhead #20159, #19974
  • Limit GLM-5.3-Flash indexer scoring to visible pools with BF16 tensor cores #20251
  • Add token-sharded tensor parallelism for Wan in VisualGen #19713
  • Support MiniMax-H3 parallel tiled VAE decoding and Ulysses parallelism #19506
  • Speed up MiniMax-H3 video-VAE decoding and fuse AdaLN RMSNorm modulation #19774, #19772
  • Unify multimodal encoder caching and reduce encoder output copies #17876, #19799
  • Optimize Qwen vision encoders and decode JPEGs through torchvision #19797, #19794
  • Reduce per-response overhead in LLM API streaming #20244
  • Enable SwiGLU clamping and update NVFP4 FC1 weight layout in Rubin CuTe DSL MoE #19488, #20181
  • Overhaul the NVLink one-sided all-to-all #19610

• Fix

  • Align MiniMax-M3 and GLM reasoning and tool parsing between streaming and non-streaming #20442, #19935
  • Flush the Hugging Face streaming detokenizer on the final response #20424
  • Restore trtllm-serve streaming throughput by constraining uvloop below 0.23.0 #19992
  • Constrain FastAPI to versions below 0.140.0 #20161
  • Keep the overlap scheduler responsive while waiting for disaggregated KV receives #19857, #19940
  • Reserve retiring-request slot headroom for overlap scheduling without attention data parallelism #20250
  • Correct draft KV scheduling and prompt-tail recomputation on cache hits with overlap speculation #19745

Highlights

• Known Issues

  • Multi-GPU (TP2) streaming or RPC executor workloads may hang during worker initialization when MPI worker pools are reused across LLM instances (observed on H100).
  • DeepSeek MLA models on SM120/SM121 GPUs (for example RTX PRO 6000 Blackwell) may crash with an illegal memory access when CUDA graphs are captured with BF16 BMM.
  • GPT-OSS models on SM120 GPUs may fail with attention data parallelism when automatic communication selects NCCL EP for unsupported hidden sizes.
  • Qwen3 models with tied embeddings (for example Qwen3-0.6B) may fail at model construction when attention data parallelism is enabled with TP greater than 1.
  • OpenTelemetry tracing may export additional FastAPI spans when FastAPI 0.142 or newer is installed.

• Model Support

  • Add GLM-5.3-Flash (glm5_next) support #19136
  • Support K-EXAONE-2 in the PyTorch backend #17067
  • Add Qwen-Image-Layered with FP8 and Cache-DiT support #16889
  • Load static FP8 Cosmos3 Nano/Super checkpoints without re-quantization #17476
  • Enable EVS for Nemotron Super 3.5 VL and EVS handoff for Nemotron-H in EPD #19648, #19702
  • Load quantized Nemotron-H MTP heads #19558, #19597
  • Add MiniMax-M3 NVFP4 KV cache support with query-length buffer reuse and a native draft KV view #19422, #19768, #19731
  • Add MiniMax H3 ref2va mode and Turbo LoRA tensor-name mapping #19657, #19701
  • Route Qwen3.6/3.8 vision-language checkpoints to the generic Qwen3.5 VLM wrapper #19892

• API

  • Remove the Python backend of KVCacheManagerV2 (BREAKING) #19154
  • Migrate LlmRequest fields to Python (BREAKING) #19484
  • Add command-line configuration overrides #19555
  • Report per-request speculative-decoding acceptance in responses #19311
  • Return OpenAI-style errors when a request exceeds the context length #19457
  • Add a per-model serving-extension registry to the OpenAI server #19499
  • Ship the OpenEngine gRPC bindings in the wheel, so no separate package install is needed #19371

• Feature

  • Add a Mooncake store pool, CLI, and V2 scheduler preemption (part 1) #19235
  • Add sparse KV cache configuration, tier-aware locking, and offload with GPU metadata publication; prepare DeepSeek V4 attention for sparse KV offload #19580, #19821, #19502
  • Add VisualGen VSA and SOL sparse attention on PrimTS block-sparse FMHA and integrate updated SM90 SageAttention kernels #18079, #18496
  • Support beam search with the C++ KVCacheManagerV2 #17306
  • Stream C++ KV events with multimodal payloads #19359
  • Add model startup stage timing metrics, sampled iteration statistics, and partial telemetry context for startup failures #18060, #19717, #19598
  • Support model_kwargs with encoder CUDA graphs #19439
  • Speed up MoE routing for batches of up to eight tokens and cache MegaMoE kernel descriptors across forwards #19682, #19901
  • Enable tiled FP8 context FMHA for head dimension 256 on SM120 #19825
  • Add a static tile scheduler override for context FMHA #19873
  • Include KV connector prefixes in V2 scheduler budgeting and add configurable sliding-window attention endpoint retention priority #17974, #19670
  • Weight the speculative draft KV budget split by per-request spend #19744
  • Support post-SiLU clamp in CUTLASS MoE and add an optional finalize-before-combine path to the MoE scheduler #19217, #19749
  • Accept flat int32 arrays as GenerationRequest prompts without converting them to lists #19658
  • Allow the prefix-tokenization cache for MiniMax-M3 text-only prompts #19613
  • Reduce MTP draft-loop and multimodal fusion overhead #19785, #18636

• Fix

  • Fix host out-of-memory failures by releasing asynchronous IPC payloads when garbage collection is disabled #19869
  • Fix dummy-slot handling for Kimi K2.5 disaggregated EAGLE3 inference with attention data parallelism #19891
  • Harden disaggregated KV transfer: retire ownership after late completion, require verified write coverage bef

Highlights

• Model Support

  • Support Qwen3.5 checkpoints with global FP8 scales across model loading paths #19519
  • Optimize Gemma4 vision rotary embeddings without a GEMM operation #19571
  • Enable the CuTeDSL MoE backend for Kimi K3 NVFP4 SiTU #19003
  • Expose Nemotron-H vision-language LoRA configuration for supported inference paths #19151
  • Extend MiniMax-M3 piecewise CUDA graphs through model and executor paths #19423

• API

  • Remove the AutoDeploy integration and its public entry points (BREAKING) #19028
  • Remove the obsolete cpp_only build option from CMake (BREAKING) #19421
  • Accept parallel_tool_calls and cap tool enum values in OpenAI requests #19459
  • Treat explicit null optional fields as unset in the Responses API #19454
  • Validate media data URIs and require opt-in for private URL fetches #19458

• Feature

  • Add an MLA-backed standalone DSpark drafter for Inferact and Kimi K3 #19040
  • Enable Helix speculative verification for FP8, FP4 MLA, and DSpark groups #19273
  • Add a non-MTP KDA decode kernel and dispatch path #18938
  • Add an NVFP4 MLA residual switch for DeepSeek V4 #19305
  • Support MNNVL all-reduce through the Ray orchestrator #18231
  • Add opt-in cached KV token counts and per-rank routing logs #19375, #16323
  • Add quiet PyExecutor hang diagnostics for stalled inference requests #19045
  • Support backend-independent agent tools and a required-tool policy #19236
  • Update scaffolding MCP transport and KV-cache truncation behavior #19055
  • Support W4A16 NVFP4 group size 32 with explicit validation #19091
  • Add Triton top-k combine for Marlin NVFP4 MoE #18999
  • Fuse MLA context FP8 Q quantization into the absorb BMM epilogue #19082
  • Enable PDL on additional decode edges and FlashInfer GDN prefill on SM120/121 #19446, #19376
  • Defer fused-GELU MLP NVFP4 scale finalization until needed #18906
  • Avoid KV-cache V2 prefix probes when context chunk budget is insufficient #19366
  • Publish MoE all-to-all counters before dispatch and consolidate C++ MoE ops #19425, #19216

• Fix

  • Compute Cosmos3 rotary tables with broadcast multiplication to avoid K=1 matmul failures #19532
  • Restore Qwen3.5 gather and scatter behavior after a performance regression #19578
  • Preserve disaggregated KV transfer outcomes and synchronize receive destinations #19377, #19444
  • Authenticate disaggregated subagent affinity and reject unbindable IPv6 listener addresses #19175, #19544
  • Make disaggregated load-balancing routing global and repair cache-reuse adapter behavior #19399, #19480
  • Extend startup and warmup timeouts for Blackwell pipeline-parallel tests #19402
  • Keep one-sided MoE combine offsets and FP4 permute buffers correctly sized #19312, #19251
  • Reject paged-context attention when no fused kernel exists #19353
  • Share speculative CUDA-graph capture buffers and correct RNG corner cases #19350, #19505
  • Correct DFlash buffer sizing, positions, and FlashInfer speculation gating #19466
  • Guard PyExecutor model-engine access while handling responses #19559
  • Route zero-token causal convolution through the channel-major kernel #18571
  • Correct VBWS width tracking and terminal-step finalization #19407
  • Drain guided-decoder host functions before forward calls that hold the GIL #19455
  • Prevent out-of-bounds indices in vanilla top-p sampling #19496
  • Reuse pending MNNVL split communicators across workspace construction retries #19101

• Documentation

  • Document GVR V2 self-sampling and multi-threshold exact top-k methods #19244

• Test & Infra

  • Upgrade public PyTorch to 2.14, Triton to 3.8, and C++ to 20 #19477, #19249
  • Add Helix zero-KV, disaggregated closure, and speculative acceptance regression coverage #18995, #19478, #19176
  • Expand Qwen3.8 coverage across GPU configurations #19530
  • Disable context overlap for the DeepSeek V4 Pro disaggregated test configuration #19424