usls

usls is a cross-platform Rust library powered by ONNX Runtime for efficient inference of SOTA vision and vision-language models (typically under 1B parameters).

🌟 Highlights

• ⚡ High Performance: Multi-threading, SIMD, and CUDA-accelerated processing
• ✨ Cross-Platform: Linux, macOS, Windows with ONNX Runtime execution providers (CUDA, TensorRT, CoreML, OpenVINO, DirectML, etc.)
• 🎯 Precision Support: FP32, FP16, INT8, UINT8, Q4, Q4F16, BNB4, and more
• 🛠️ Full-Stack Suite: DataLoader, Annotator, and Viewer for complete workflows
• 🏗️ Unified API: Single Model trait inference with run()/forward()/encode_images()/encode_texts() and unified Y output
• 📥 Auto-Management: Automatic model download (HuggingFace/GitHub), caching and path resolution
• 📦 Multiple Inputs: Image, directory, video, webcam, stream and combinations
• 🌱 Model Ecosystem: 50+ SOTA vision and VLM models

🚀 Quick Start

Run the YOLO-Series demo to explore models with different tasks, precision and execution providers:

• Tasks: detect, segment, pose, classify, obb
• Versions: v5, v6, v7, v8, v9, v10, 11, 12, v13, 26
• Scales: n, s, m, l, x
• Precision: fp32, fp16, q8, int8, q4, q4f16, bnb4, and more
• Execution Providers: CPU, CUDA, TensorRT, TensorRT-RTX, CoreML, OpenVINO, and more

cargo run -r --example yolo -- --task detect --ver 26 --scale n --dtype fp16

cargo run -r -F cuda --example yolo -- --task segment --ver 11 --scale m --device cuda:0 --processor-device cuda:0

cargo run -r -F tensorrt-full --example yolo -- --device tensorrt:0 --processor-device cuda:0

cargo run -r -F nvrtx-full --example yolo -- --device nvrtx:0 --processor-device cuda:0

cargo run -r -F coreml --example yolo -- --device coreml

cargo run -r -F openvino -F ort-load-dynamic --example yolo -- --device openvino:CPU

Environment: NVIDIA RTX 3060Ti (TensorRT-10.11.0.33, CUDA 12.8, TensorRT-RTX-1.3.0.35) / Intel i5-12400F

Setup: YOLO26 Detection, COCO2017-val (5,000 images), 640x640, Conf thresholds: [0.35, 0.3, ..]

Results are for rough reference only.

Scale	EP	ImageProcessor	DType	Batch	Preprocess	Inference	Postprocess	Total
n	TensorRT	CUDA	FP16	1	~233µs	~1.3ms	~14µs	~1.55ms
n	TensorRT-RTX	CUDA	FP32	1	~233µs	~2.0ms	~10µs	~2.24ms
n	TensorRT-RTX	CUDA	FP16	1	❓	❓	❓	❓
n	CUDA	CUDA	FP32	1	~233µs	~5.0ms	~17µs	~5.25ms
n	CUDA	CUDA	FP16	1	~233µs	~3.6ms	~17µs	~3.85ms
n	CUDA	CPU	FP32	1	~800µs	~6.5ms	~14µs	~7.31ms
n	CUDA	CPU	FP16	1	~800µs	~5.0ms	~14µs	~5.81ms
n	CPU	CPU	FP32	1	~970µs	~20.5ms	~14µs	~21.48ms
n	CPU	CPU	FP16	1	~970µs	~25.0ms	~14µs	~25.98ms
n	TensorRT	CUDA	FP16	8	~1.2ms	~6.0ms	~55µs	~7.26ms
n	TensorRT	CPU	FP16	8	~18.0ms	~25.5ms	~55µs	~43.56ms
m	TensorRT	CUDA	FP16	1	~233µs	~3.6ms	~14µs	~3.85ms
m	TensorRT	CUDA	Int8	1	~233µs	~2.6ms	~14µs	~2.84ms
m	CUDA	CUDA	FP32	1	~233µs	~16.1ms	~17µs	~16.35ms
m	CUDA	CUDA	FP16	1	~233µs	~8.8ms	~17µs	~9.05ms

🔍 What's Next?

• 📖 Online Documentation
• 📚 API Reference
• 📦 Model Zoo
• 🚀 Examples

📦 Model Zoo

[!NOTE]

Status: ✅ Supported  |  ❓ Unknown  |  ❌ Not Supported For Now

🔍 All ONNX models are available from the ONNX Models Repository

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
YOLOv5	Image ClassificationObject DetectionInstance Segmentation	demo	✅	✅	✅	✅	✅	❌	❌
YOLOv6	Object Detection	demo	✅	✅	✅	✅	✅	❌	❌
YOLOv7	Object Detection	demo	✅	✅	✅	✅	✅	❌	❌
YOLOv8	Object DetectionInstance SegmentationImage ClassificationOriented Object DetectionKeypoint Detection	demo	✅	✅	✅	✅	✅	❌	❌
YOLO11	Object DetectionInstance SegmentationImage ClassificationOriented Object DetectionKeypoint Detection	demo	✅	✅	✅	✅	✅	❌	❌
YOLOv9	Object Detection	demo	✅	✅	✅	✅	✅	❌	❌
YOLOv10	Object Detection	demo	✅	✅	✅	✅	✅	❌	❌
YOLOv12	Image ClassificationObject DetectionInstance Segmentation	demo	✅	✅	✅	✅	✅	✅	✅
YOLOv13	Object Detection	demo	✅	✅	✅	✅	✅	✅	✅
YOLO26	Object DetectionInstance SegmentationImage ClassificationOriented Object DetectionKeypoint Detection	demo	✅	✅	✅	✅	✅	✅	✅

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
BEiT	Image Classification	demo	✅	✅	✅	✅	❌	❌	❌
ConvNeXt	Image Classification	demo	✅	✅	✅	✅	❌	❌	❌
FastViT	Image Classification	demo	✅	✅	✅	✅	❌	❌	❌
MobileOne	Image Classification	demo	✅	✅	✅	✅	❌	❌	❌
DeiT	Image Classification	demo	✅	✅	✅	✅	❌	❌	❌
RAM	Image Tagging	demo	✅	❓	✅	✅	✅	✅	✅
RAM++	Image Tagging	demo	✅	❓	✅	✅	✅	✅	✅

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
RT-DETRv1	Object Detection	demo	✅	✅	✅	✅	✅	✅	✅
RT-DETRv2	Object Detection	demo	✅	✅	✅	✅	✅	✅	✅
RT-DETRv4	Object Detection	demo	✅	✅	✅	✅	✅	✅	✅
RF-DETR	Object Detection	demo	✅	✅	✅	✅	✅	✅	✅
PP-PicoDet	Object Detection	demo	❌	❓	✅	❌	❌	❌	❌
D-FINE	Object Detection	demo	✅	❓	✅	❌	❌	❌	❌
DEIM	Object Detection	demo	✅	❓	✅	❌	❌	❌	❌
DEIMv2	Object Detection	demo	✅	❓	✅	✅	✅	✅	✅

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
SAM	Segment Anything	demo	✅	❓	✅	❌	❌	❌	❌
SAM-HQ	Segment Anything	demo	✅	❓	✅	❌	❌	❌	❌
MobileSAM	Segment Anything	demo	✅	❓	✅	❌	❌	❌	❌
EdgeSAM	Segment Anything	demo	✅	❓	✅	❌	❌	❌	❌
YOLOE-v8/11-Prompt-Free	Open-Set Detection And Segmentation	demo	✅	✅	✅	✅	✅	✅	✅
YOLOE-26-Prompt-Free	Open-Set Detection And Segmentation	demo	✅	✅	✅	✅	✅	✅	✅
FastSAM	Instance Segmentation	demo	✅	✅	✅	✅	✅	✅	✅
EdgeCrafter (ECDetSeg)	Instance Segmentation	demo	❓	✅	✅	✅	❌	❌	❌
SAM2	Segment Anything	demo	✅	❓	✅	❌	❌	❌	❌
SAM3-Tracker	Segment Anything	demo	✅	✅	✅	✅	✅	✅	✅
BiRefNet - COD	Camouflaged Object Detection	demo	✅	❓	✅	✅	✅	✅	✅
BiRefNet - DIS	Dichotomous Image Segmentation	demo	✅	❓	✅	✅	✅	✅	✅
BiRefNet - HRSOD	High-Resolution Salient Object Detection	demo	✅	❓	✅	✅	✅	✅	✅
BiRefNet - Massive	Multi-Dataset Robust Segmentation	demo	✅	❓	✅	✅	✅	✅	✅

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
RMBG	Image SegmentationBackground Removal	demo	✅	❓	✅	✅	✅	✅	✅
BEN2	Image SegmentationBackground Removal	demo	✅	❓	✅	✅	❌	❌	❌

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
MobileGaze	Eye Gaze Estimation	demo	✅	❓	✅	✅	✅	✅	✅

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
MODNet	Image Matting	demo	✅	❓	✅	✅	✅	❌	❌
MediaPipe Selfie	Image Segmentation	demo	✅	❓	✅	✅	✅	❌	❌
BiRefNet - Portrait	Portrait Background Removal	demo	✅	❓	✅	✅	✅	✅	✅
BiRefNet - Matting	Portrait Matting & Background Removal	demo	✅	❓	✅	✅	✅	✅	✅
BiRefNet - HR Matting	High-Resolution Portrait Matting	demo	✅	❓	✅	✅	✅	✅	✅
BiRefNet - General	General Purpose Segmentation	demo	✅	❓	✅	✅	✅	✅	✅
BiRefNet - HR General	High-Resolution General Segmentation	demo	✅	❓	✅	✅	✅	✅	✅
BiRefNet - Lite General	Lightweight General Segmentation (2K)	demo	✅	❓	✅	✅	✅	✅	✅
BiRefNet - General Tiny	Lightweight General Segmentation with Swin-V1-Tiny	demo	✅	❓	✅	✅	✅	✅	✅

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
GroundingDINO	Open-Set Detection With Language	demo	✅	❓	✅	✅	✅	✅	✅
MM-GDINO	Open-Set Detection With Language	demo	✅	❓	✅	✅	✅	✅	✅
LLMDet	Open-Set Detection With Language	demo	✅	❓	✅	✅	✅	✅	✅
OWLv2	Open-Set Object Detection	demo	✅	❓	✅	✅	❌	❌	❌
YOLO-World	Open-Set Detection With Language	demo	✅	✅	✅	✅	✅	✅	✅
YOLOE-Prompt-Based	Open-Set Detection And Segmentation	demo	✅	✅	✅	✅	✅	✅	✅
YOLOE-26-Prompt-Based	Open-Set Detection And Segmentation	demo	✅	✅	✅	✅	✅	✅	✅
SAM3-Image	Open-Set Detection And Segmentation	demo	✅	✅	✅	✅	✅	✅	✅

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
ByteTrack	Multi-Object Tracking	demo	❌	❌	❌	❌	❌	❌	❌

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
Swin2SR	Image Restoration	demo	✅	❓	✅	✅	✅	✅	✅
APISR	Anime Super-Resolution	demo	✅	❓	✅	✅	✅	✅	✅

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
RTMPose	Keypoint Detection	demo	✅	❓	✅	✅	✅	✅	✅
DWPose	Keypoint Detection	demo	✅	❓	✅	✅	✅	✅	✅
RTMW	Keypoint Detection	demo	✅	❓	✅	✅	✅	✅	✅
RTMO	Keypoint Detection	demo	✅	❓	✅	✅	✅	✅	❌
ECPose	Keypoint Detection	demo	✅	❓	✅	✅	❌	❌	❌

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
DB (PaddleOCR det v4 / v5)	Text Detection	demo	✅	❓	✅	✅	❌	❌	❌
FAST	Text Detection	demo	✅	❓	✅	✅	❌	❌	❌
LinkNet	Text Detection	demo	✅	❓	✅	✅	❌	❌	❌
SVTR (PaddleOCR rec v4 / v5)	Text Recognition	demo	✅	❓	✅	✅	❌	❌	❌
TrOCR	Text Recognition	demo	✅	❓	✅	✅	❌	❌	❌
SLANet (PaddleOCR tab v4 / v5)	Table Recognition	demo	✅	❓	✅	✅	❌	❌	❌
DocLayout-YOLO	Object Detection	demo	✅	❓	✅	✅	✅	❌	❌
PP-DocLayout-v1-Plus-L	Object Detection	demo	✅	✅	✅	✅	✅	❌	❌
PP-DocLayout-v2	Object Detection	demo	✅	❓	✅	✅	✅	✅	✅
PP-DocLayout-v3	Object Detection	demo	✅	✅	✅	✅	✅	✅	✅

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
BLIP	Image Captioning	demo	✅	❓	✅	❓	❌	❌	❌
Florence2	A Variety of Vision Tasks	demo	✅	❓	✅	✅	❌	❌	❌
Moondream2	Open-Set Object DetectionOpen-Set Keypoints DetectionImage CaptioningVisual Question Answering	demo	✅	❓	❌	❌	✅	✅	❌
SmolVLM	Visual Question Answering	demo	✅	❓	✅	❓	❓	❓	❓
SmolVLM2	Visual Question Answering	demo	✅	❓	✅	❓	❓	❓	❓
FastVLM	Vision Language Models	demo	✅	❓	✅	✅	✅	✅	✅

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
CLIP	Vision-Language Embedding	demo	✅	❓	✅	✅	✅	✅	✅
jina-clip-v1	Vision-Language Embedding	demo	✅	❓	✅	✅	✅	✅	✅
jina-clip-v2	Vision-Language Embedding	demo	✅	❓	✅	✅	✅	✅	✅
mobileclip	Vision-Language Embedding	demo	✅	❓	✅	✅	✅	✅	✅
SigLIP	Vision-Language Embedding	demo	✅	❓	✅	✅	✅	✅	✅
SigLIPv2	Vision-Language Embedding	demo	✅	❓	✅	✅	✅	✅	✅
DINOv2	Vision Embedding	demo	✅	❓	✅	❌	❌	❌	❌
DINOv3	Vision Embedding	demo	✅	❓	✅	✅	✅	✅	✅

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
DepthAnything v1	Monocular Depth Estimation	demo	✅	❓	✅	✅	✅	✅	✅
DepthAnything v2	Monocular Depth Estimation	demo	✅	❓	✅	✅	✅	✅	✅
DepthPro	Monocular Depth Estimation	demo	✅	❓	✅	✅	✅	✅	✅
Depth-Anything-3	MonocularMetricMulti-View	demo	✅	❓	✅	✅	✅	✅	✅

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
Sapiens	Foundation for Human Vision Models	demo	✅	❓	✅	✅	✅	✅	✅
YOLOPv2	Panoptic Driving	demo	✅	❓	✅	❌	❌	❌	❌

🤝 Contributing

This is a personal project maintained in spare time, so progress on performance optimization and new model support may vary.

We highly welcome PRs for model optimization! If you have expertise in specific models and can help optimize their interfaces or post-processing, your contributions would be invaluable. Feel free to open an issue or submit a pull request for suggestions, bug reports, or new features.

🙏 Acknowledgments

• This project is built on top of ort (ONNX Runtime for Rust), which provides seamless Rust bindings for ONNX Runtime. Special thanks to the ort maintainers.

• Special thanks to @kadu-v for the jamtrack-rs project, which inspired our ByteTracker implementation

Thanks to all the open-source libraries and their maintainers that make this project possible. See Cargo.toml for a complete list of dependencies.

📜 License

This project is licensed under LICENSE.