ort

2.0.0-rc.14

💖 If you find ort useful, please consider supporting us on Open Collective 💖

🤔 Need help upgrading? Ask questions in GitHub Discussions or in the pyke.io Discord server!

✨ Supporters get priority (<24h) issue resolution & support!

---

⚠️ Manual environments

ort replaced the manual Environment struct with ort::init all the way back in Nov 2023 with 2.0.0-alpha.1 (yes, before rc hell, we used to be in alpha hell). This was done because only one environment was allowed per process anyway. However, almost every other release since then, there has been some kind of issue stemming from the environment being stored as a global variable.

The trouble caused by this now far outweighs the inconvenience of lugging around another variable everywhere, so in rc.14, you must always first create an Environment to start building sessions:

// pre-rc.14
{
    // optional, even!
    ort::init().commit()?;

    let mut model = Session::builder()?
        .with_optimization_level(GraphOptimizationLevel::Level3)?
        .with_intra_threads(4)?
        .commit_from_file("yolov8m.onnx")?;
}

// rc.14
{
    let env = ort::init().build()?; // <-- must build an environment first
    let mut model = Session::builder(&env)?
        .with_optimization_level(GraphOptimizationLevel::Level3)?
        .with_intra_threads(4)?
        .commit_from_file("yolov8m.onnx")?;
}

Note that you don't need to hold onto env unless you intend to create more sessions. Please refrain from storing environments or sessions as global variables -- we can't fix the issues that causes within ONNX Runtime! And library authors, please remember to allow the user to pass their own Environment.

✨ Rust 1.100 compatibility

rc.14 adds support for the upcoming v2 build dir layout that will ship in Rust 1.100. If you intend to update to Rust 1.100, you should upgrade to rc.14 as soon as possible, since older ort versions will fail to build when copy-dylibs is enabled.

Note that, due to the new layout, tests will no longer be able to see dylibs copied by copy-dylibs; you can try to use ort::util::preload_dylib to manually set the path to provider dylibs instead.

📦 You get a binary! You get a binary! Everyone gets binaries!

🟩 rc.14's prebuilt CUDA binaries add support for RTX 50xx/Blackwell cards on x64. We now also ship aarch64 builds for Jetson Orin/Thor, Grace Hopper/Blackwell, & DGX Spark! Turing & Ampere GPUs remain fully supported.

🟥 Team Red gets some love as well: we now have MIGraphX binaries on x64 Linux! In the future, we'll also ship Windows builds with the Vitis AI EP for NPU acceleration on Ryzen AI chips.

🟦 We didn't forget about Team Blue either! We're also shipping OpenVINO builds for Windows & Linux x64.

🤖 x64 Android binaries were also added to make simulation easier.

⚙️ Miscellaneous features

• https://github.com/pykeio/ort/commit/f2a5f73c9dd9ff3682303481cc792bf4669e3e94 Update to ONNX Runtime v1.31.0.
  • ort-web is updated to v1.30.0.
• https://github.com/pykeio/ort/commit/c1f6e09b39830c502ed42a1c87dd3d36aa113d28 Add ort::util::probe to load-dynamic to check the version of ONNX Runtime before you try to load it.
• https://github.com/pykeio/or

2.0.0-rc.13

💖 If you find ort useful, please consider sponsoring us on Open Collective 💖

🤔 Need help upgrading? Ask questions in GitHub Discussions or in the pyke.io Discord server!

---

🎯 Execution provider clarity

EP structs like ep::CUDA are now compile-time gated behind their respective Cargo feature flags. The EP feature flags only modifying runtime behavior was a common pain point, and this change brings ort in line with, like, every other Rust crate in existence.

⚠️ ort will now throw an error at link time if download-binaries is enabled and no binaries contain all of the requested EPs. Previously, if you enabled a nonsensical combination like cuda + coreml, ort would silently fall back to a CPU-only build. This caused great confusion as the only indicator of what went wrong was in log messages that don't show by default. A new lax-feature-matching Cargo feature will fallback to the closest fit instead of erroring.

🍎 ONNX Runtime 1.28

rc.13 skips ahead 4 ONNX Runtime versions to v1.28, bringing new operator support, bug & security fixes, and performance improvements.

✖️ No CUDA 12

rc.13 only ships CUDA 13 binaries, as ONNX Runtime has deprecated CUDA 12.

🧩 Custom operator ergonomics

Custom operators have been reworked to have a more Rusty interface:

 struct MyOperator;
 
 impl Operator for MyOperator {
+	type Kernel<'attr> = ort::operator::BoxedKernel<'attr>;
 
 	fn name(&self) -> &str {
 		"MyOperator"
 	}
 
-	fn inputs(&self) -> Vec<OperatorInput> {
-		vec![OperatorInput::required(TensorElementType::Float32), OperatorInput::required(TensorElementType::Float32)]
+	fn inputs(&self) -> impl IntoIterator<Item = OperatorInput> {
+		[OperatorInput::required(TensorElementType::Float32), OperatorInput::required(TensorElementType::Float32)]
 	}
 
-	fn outputs(&self) -> Vec<OperatorOutput> {
-		vec![OperatorOutput::required(TensorElementType::Float32)]
+	fn outputs(&self) -> impl IntoIterator<Item = OperatorOutput> {
+		[OperatorOutput::required(TensorElementType::Float32)]
 	}
 
-	fn create_kernel(&self, _: &KernelAttributes) -> ort::Result<Box<dyn Kernel>> {
-		Ok(Box::new(|ctx: &KernelContext| {
+	fn create_kernel<'attr>(&self, _: &KernelContext<'attr>) -> ort::Result<Self::Kernel<'attr>> {
+		Ok(Box::new(|ctx: &ComputeContext| {
 			...
 		}))
 	}
 }

ℹ️ Custom diagnostic messages

ort now uses the wonderful #[diagnostic] attributes to drastically improve the clarity of commonly encountered errors.

☑️ Miri-approved

Test coverage was expanded and ort was ran against Miri, and a few unsoundness bugs were fixed.

✨ Other features

• https://github.com/pykeio/ort/commit/8db51e7684f36615fb2a822f90e235a70d765b53 Add VSINPU EP
• https://github.com/pykeio/ort/commit/5688cce9a01f17fa88b613f603766e2b6052304d (ort-web): Allow creating tensors from images.
• https://github.com/pykeio/ort/commit/726dc7c47d8e0b2548885f796062b0b6d32d892e Expose the DirectML EP API.
• https://github.com/pykeio/ort/commit/5e669f6944911bb3fc34014680ec91f74cee30a0 Support string array operator attributes
• https://github.com/pykeio/ort/commit/5f99738620a1ee5d40aca7d90ab65bc4d295538f Support float16 tensor usage with the native f16 type on nightly Rust
• https://github.com/pykeio/ort/commit/62ad710a5861b1722e9041ae730837547f85375e (ort-candle): Update candle to 0.11
• https://github.com/pykeio/ort/commit/d5dc28f2f50c96d0e0cdcc668f3a23462f2b94dc (ort-tract): Update tract to 0.23

🔧 Other fixes

• https://github.com/pykeio/ort/commit/d5eb59f2ff5a597e0666b2d0cb8a901ffe54742e Committing a session which has a map input/output will

2.0.0-rc.12

💖 If you find ort useful, please consider sponsoring us on Open Collective 💖

🤔 Need help upgrading? Ask questions in GitHub Discussions or in the pyke.io Discord server!

---

This release was made possible by Rime.ai!

Authentic AI voice models for enterprise.

---

📍 Multiversioning

🚨 If you used ort with default-features = false, enable the api-24 feature to use the latest features.

The big highlight of this release is multiversioning: ort can now use any minor version of ONNX Runtime from v1.17 to v1.24. New features are gated behind api-* feature flags, like api-20 or api-24. These flags will set the minimum version of ONNX Runtime required by ort.

More info 👉 https://ort.pyke.io/setup/multiversion

🪄 Automatic device selection

With ONNX Runtime 1.22 or later, ort will now automatically use an NPU if one is available for maximum efficiency & power savings! Setting your own execution providers will override this.

This is thanks to the super cool new SessionBuilder::with_auto_device API! There's also SessionBuilder::with_devices for finer control.

👁️ CUDA 13

ort now ships builds for both CUDA 12 & CUDA 13! It should automatically detect which CUDA you're using, but if it gets it wrong, you can override it by setting the ORT_CUDA_VERSION environment variable to 12 or 13.

🩹 SessionBuilder error recovery

You can now recover from errors when building a session by calling .recover() on the error type to get the SessionBuilder back.

🛡️ Build attestations

Prebuilt binaries are now attested via GitHub Actions, so you can verify that they are untampered builds of ONNX Runtime coming straight from pyke.io.

To verify, download your binary package of choice and use the gh CLI to verify:

➜  gh attestation verify --owner pykeio ./x86_64-pc-windows-msvc+cu13.tar.lzma2
Loaded digest sha256:e96616510082108be228ad6ea026246a31650b7d446b330c6b9671fcb9ae6267 for file://./x86_64-pc-windows-msvc+cu13.tar.lzma2
Loaded 1 attestation from GitHub API

The following policy criteria will be enforced:
- OIDC Issuer must match:................... https://token.actions.githubusercontent.com
- Source Repository Owner URI must match:... https://github.com/pykeio
- Predicate type must match:................ https://slsa.dev/provenance/v1
- Subject Alternative Name must match regex: (?i)^https://github.com/pykeio/

✓ Verification succeeded!

sha256:e96616510082108be228ad6ea026246a31650b7d446b330c6b9671fcb9ae6267 was attested by:
REPO                  PREDICATE_TYPE                  WORKFLOW
pykeio/ort-artifacts  https://slsa.dev/provenance/v1  .github/workflows/build-runner.yml@refs/heads/main

(Also note that the SHA-256 hash lines up with the one defined in dist.txt.)

---

Moving stuff around

• The ORT_LIB_LOCATION environment variable has been renamed to ORT_LIB_PATH.
  • Same with all other env vars ending in _LOCATION.
  • The old names will continue to work, but they won't make it into v2.0.0, so it's a good idea to change them now!
• Everything that used to be in ort::tensor is now in ort::value, because why have a tensor module if the Tensor<T> type actually comes from the value module?
• IoBinding and Adapter were moved from their own modules into ort::session. All sub-modules

---

💖 If you find ort useful, please consider sponsoring us on Open Collective 💖

🤔 Need help upgrading? Ask questions in GitHub Discussions or in the pyke.io Discord server!

---

I'm sorry it took so long to get to this point, but the next big release of ort should be, finally, 2.0.0 🎉. I know I said that about one of the old alpha releases (if you can even remember those), but I mean it this time! Also, I would really like to not have to do another major release right after, so if you have any concerns about any APIs, please speak now or forever hold your peace!

A huge thank you to all the individuals who have contributed to the Collective over the years: Marius, Urban Pistek, Phu Tran, Haagen, Yunho Cho, Laco Skokan, Noah, Matouš Kučera, mush42, Thomas, Bartek, Kevin Lacker, & Okabintaro. You guys have made these past rc releases possible.

If you are a business using ort, please consider sponsoring me. Egress bandwidth from pyke.io has quadrupled in the last 4 months, and 90% of that comes from just a handful of businesses. I'm lucky enough that I don't have to pay for egress right now, but I don't expect that arrangement to last forever. pyke & ort have been funded entirely from my own personal savings for years, and (as I'm sure you're well aware 😂) everything is getting more expensive, so that definitely isn't sustainable.

Seeing companies that raise tens of millions in funding build large parts of their business on ort, ask for support, and then not give anything back just... seems kind of unfair, no?

---

ort-web

ort-web allows you to use the fully-featured ONNX Runtime on the Web! This time, it's hack-free and thus here to stay (it won't be removed, and then added back, and then removed again like last time!)

See the crate docs for info on how to port your application to ort-web; there is a little bit of work involved. For a very barebones sample application, see ort-web-sample.

Documentation for ort-web, like the rest of ort, will improve by the time 2.0.0 comes around. If you ever have any questions, you can always reach out via GitHub Discussions or Discord!

Features

• 5d85209 Add WebNN & WASM execution providers for ort-web.
• #430 (💖 @jhonboy121) Support statically linking to iOS frameworks.
• #433 (💖 @rMazeiks) Implement more traits for GraphOptimizationLevel.
• 6727c98 Make PrepackedWeights Send + Sync.
• 15bd15c Make the TLS backend configurable with new tls-* Cargo features.
• f3cd995 Allow overriding the cache dir with the ORT_CACHE_DIR environment variable.
• 🚨 8b3a1ed Load the dylib immediately when using ort::init_from.
  • You can now detect errors from dylib loading and let your program react accordingly.
• 🚨 #484 (💖 @michael-p) Update ndarray to v0.17.
  • This means you'll need to upgrade your ndarray dependency to v0.17, too.
• [**`0