onnxruntime

ONNX Runtime WebGPU Plugin EP 0.5.0 expands attention, mixture-of-experts, and quantized operator support, improves GPU kernel performance, and strengthens runtime integration and correctness.

Highlights

Performance

• Enabled subgroup-matrix kernels on AMD GPUs (#32667).
• Added an FP16 GEMV specialization for single-row decode projections (#33070).
• Improved data movement in Concat, Slice, Transpose, and pooling, and vectorized NHWC depthwise convolution across channels (#32497, #32498).
• Enabled local-window flash attention for GroupQueryAttention (#32462).
• Optimized recurrent and chunked GatedDeltaNet execution (#32645, #33143).
• Optimized sparse attention indexers and SparsePagedAttention softmax, and cached sparse-indexer QSA visibility and scores (#32613, #32656, #33017).

Operator and Model Support

• Added native GatedDeltaNet and VarlenCausalConvWithState kernels, including subsequent state-update-capacity support for GatedDeltaNet (#32510, #32477, #32767).
• Added SparseAttentionIndexer, DynamicSparseAttention, and SparsePagedAttention, plus a packed sparse attention indexer for continuous batching (#32528, #32529, #32533, #32618).
• Extended PagedAttention with full-context and local non-causal attention, and per-head Q/K normalization (#32763, #32792, #32808).
• Added unquantized MoE and expanded QMoE activation, gating, routing, and packed-input support (#32447).
• Added block-scaled FP8 QMoE, INT2 expert weights, and mixed-width integer QMoE execution (#32794, #32878).
• Added Qwen4-Exp Engram support and VarlenNGramHashMapping for packed and variable-length batching (#32285, #32358, #32467).
• Extended GatherBlockQuantized to FP8 and FP4 data. Out-of-range indices now produce zero output slices for both floating-point and existing integer formats (#32480).
• Added GroupNorm, SkipGroupNorm, and 3D ConvTranspose support (#32463, #32688).
• Added weightless hyper-connection primitives: BranchwiseRMSNorm, ScaledSiLU, HyperConnectionPreMix, and HyperConnectionPostMix (#32687).
• Added SiLU/Swish/Sigmoid activation modes to GatedRMSNorm and QuickGelu fusion in the im2col-matmul convolution path (#32512, [#32560](https://github.com/microsoft/onnxruntim

ONNX Runtime 1.31.1 is a patch release that fixes an OpenVINO-enabled Python build regression and corrects WASM Debug external-data tests. These notes cover changes since ONNX Runtime 1.31.0.

Highlights

• Fixed a provider-info namespace mismatch that caused compilation failures when building with both OpenVINO and Python enabled (#33327).
• Fixed WASM Debug test failures by skipping file-backed external-data cases unsupported by the WASM test module, while retaining compile-output checks, buffer-destination coverage, and native test coverage. This is a test-only change and does not change runtime behavior (#33226).

Contributors

Thanks to our 3 human contributors for this release!

@eserscor, @mustjab, @tianleiwu

Full Changelog: rel-1.31.0...rel-1.31.1

Release highlights were prepared with AI assistance.

ONNX Runtime 1.31.0 stabilizes model-package and EPContext data APIs, improves CPU model loading and quantized MoE inference, expands device-based execution-provider selection, and strengthens model-loading and runtime reliability. These notes cover changes since ONNX Runtime 1.30.0. CUDA and WebGPU kernel updates are summarized here; detailed provider notes are covered by their separate plugin EP releases.

Highlights

• Promoted the model-package API and EPContext data callbacks to stable C and C++ APIs, with EPContext callback support added across language bindings (#33166, #32265).
• Reduced CPU session-initialization overhead by parallelizing eligible weight prepacking, and accelerated block-wise INT4/INT8 QMoE experts with MLAS QNBit kernels and grouped expert dispatch (#31691, #32644, #32668).
• Added x86 FP16 LayerNorm/RMSNorm acceleration, Arm KleidiAI SVE2.1 FP16 GEMM support, and additional RISC-V vector kernels (#32715, #32670, #32710).
• Enabled CoreML participation in device-based EP selection, including Apple Neural Engine selection through PREFER_NPU (#31975).
• Added workspace-memory accounting and verification for constrained-memory graph partitioning, alongside broader model, graph, and tensor validation (#31962, #32189).

Announcements & Compatibility

• CUDA Plugin EP v0.3.0 and WebGPU Plugin EP v0.5.0 are matched with ONNX Runtime 1.31.0. For contrib-op compatibility, core and the plugin must use the same contributed-operator schemas; building both from the same ONNX Runtime revision is recommended. Schema mismatches can cause incorrect execution or crashes and may not be detected during plugin registration.
• GatherBlockQuantized behavior change: out-of-range indices now produce zero output slices across CPU, CUDA, and WebGPU for both integer and floating-point quantized formats. This deliberately differs from ONNX Gather error semantics (#32480).
• Experimental API migration: model-package consumers should use OrtApi::GetModelPackageApi and the stable OrtModelPackageApi table. EPContext callback registration now uses stable APIs; the corresponding experimental lookup entries were removed. The stable native callback contract carries callback/state pointers, with payload limits enforced by application callbacks, bindings, or EPs rather than a native read-options policy (#33166, #32265).
• ACL EP is deprecated and will be removed in a future release. Source builds using --use_acl now receive a deprecation warning (#32564).
• The ONNX dependency remains at 1.22.0. The initial ONNX 1.23 integration was reverted pending upstream fixes; this release does not ship that integration's new opset or FLOAT6 support (#33073).
• Supported native builds now default to 1DS telemetry. Windows TraceLogging remains selectable with --use_windows_telemetry or onnxruntime_USE_WINDOWS_TELEMETRY=ON; telemetry can be disabled with --no_telemetry (#32384).

New F

This release brings sparse attention and packed continuous-batching support, INT2 and mixed-width QMoE execution, decode-kernel tuning, and reliability fixes to the ONNX Runtime CUDA Plugin EP. Highlights include new QSA/CSA attention operators, expanded paged speculative decoding, opt-in MatMul auto-tuning, and improved MXFP4 support on Blackwell consumer GPUs.

Compatibility

The minimum ONNX Runtime core version remains 1.24.4, but meeting that minimum alone does not guarantee compatibility for contributed operators. Core and the plugin must use the same contributed-operator schemas; building both from the same ONNX Runtime revision is recommended. Schema mismatches can cause incorrect execution or crashes and may not be detected when registering the plugin (#32855).

The matched ONNX Runtime version is 1.31.0 for this release.

Highlights

Plugin Runtime and CUDA Graphs

• Added checked GroupQueryAttention workspace recipes for KV-cache preparation and XQA, FlashAttention, memory-efficient, and unfused routes, plus conservative workspace estimation for resource-constrained graph partitioning. cuDNN workspace is reported as unavailable rather than incorrectly estimated as zero (#32446, #32453, #32454, #32617).
• Bounded sliding-window GQA workspace by the resident or staged KV extent rather than the unbounded absolute sequence length, and consolidated runtime scratch sizing and sequence-length-independent route eligibility into shared helpers (#32602, #32945, #33001).
• Replaced eligible decode-path memset operations with small clearing kernels for fpA_intB split-K semaphores, paged XQA semaphores, and compact convolution state updates, reducing CUDA Graph dependency overhead (#32884).
• Replaced shared ones-buffer bias broadcasts in Gemm and DecoderAttention with direct broadcasts on the consuming CUDA stream, removed the constant-ones cache, and corrected zero-K Gemm bias scaling (#32852).

Attention and KV Caching

• Added a cuDNN paged SDPA decode tier to PagedAttention for eligible FP16/BF16 causal workloads with unquantized KV caches. It is selected ahead of FlashAttention when XQA is not the winner, automatically considered on SM90+, and can be disabled with ORT_ENABLE_CUDNN_FLASH_ATTENTION=0. Follow-up fixes stabilize default-scale cache keys and backend selection during CUDA Graph execution (#32493, #32624).
• Expanded non-causal PagedAttention beyond FlashAttention, including portable paths for small cache pages and supported XQA speculative paths; added coverage for head size 128 with 16/32/64-token pages and ragged batches (#32619).
• Added causal paged speculative XQA for FP16 query/output, INT8 KV caches, head size 128, GQA group size 6, and query widths 2-8. Unsupported widths and non-causal H128 workloads retain existing fallbacks (#32740).
• Documented the block-eviction layout contract for sliding-window GQA caches, including how to index resident tokens after multi-token speculative or chunked-prefill steps (#32419).
• Allowed GQA to borrow unquantized past K/V without allocating or copying present caches when new K/V is empty and