标签：AWQ 量化

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

Faster Whisper transcription with CTranslate2

中文LLaMA&Alpaca大语言模型+本地CPU/GPU训练部署 (Chinese LLaMA & Alpaca LLMs)

[🔥updating ...] AI 自动量化交易机器人(完全本地部署) AI-powered Quantitative Investment Research Platform. 📃 online docs: https://ufund-me.github.io/Qbot ✨ :news: qbot-mini: https://github.com/Charmve/iQuant

A vector index built on TurboQuant, written in Rust with Python bindings

A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.

Accessible large language models via k-bit quantization for PyTorch.

Lossy PNG compressor — pngquant command based on libimagequant library

Fast inference engine for Transformer models

[ICLR2025, ICML2025, NeurIPS2025 Spotlight] Quantized Attention achieves speedup of 2-5x compared to FlashAttention, without losing end-to-end metrics across language, image, and video models.

[ICLR2025 Spotlight] SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models

State-of-the-art LLM compression, built for production inference with vLLM

Your Cheat Sheet for AI Engineering Interview – Questions and Answers.

🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools

Sparsity-aware deep learning inference runtime for CPUs

PyTorch native quantization for training and inference

ComfyUI Plugin of Nunchaku

Run GLM-5.3-Flash (321B), DeepSeek-V4.1-Flash (748B) and MiniMax-M3.1-Flash (428B) on your own GPUs - one engine, one dashboard, OpenAI- and Anthropic-compatible API.

AIMET is a library that provides advanced quantization and compression techniques for trained neural network models.

SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime

Build, personalize and control your own LLMs. From data pre-processing to fine-tuning, xTuring provides an easy way to personalize open-source LLMs. Join our discord community: https://discord.gg/TgHXuSJEk6

Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.

Mastering Applied AI, One Concept at a Time

Run a 1-billion parameter LLM on a $10 board with 256MB RAM

A Python package for extending the official PyTorch that can easily obtain performance on Intel platform

Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率，量化感知训练的精度

PaddleSlim is an open-source library for deep model compression and architecture search.

Brevitas: neural network quantization in PyTorch

A toolkit to optimize ML models for deployment for Keras and TensorFlow, including quantization and pruning.

Build, train, optimize, and run computer vision models locally, from raw images to live inference. Open source, optimized for Intel® XPU (CUDA also supported).

LLM model quantization (compression) toolkit with HW acceleration support for Nvidia (CUDA), AMD (ROCm), Intel GPU (XPU), Huawei Ascent (NPU), and Intel/AMD/Apple CPU via HF, vLLM, and SGLang.

[ICML2025] SpargeAttention: A training-free sparse attention that accelerates any model inference.

Neural Network Compression Framework for enhanced OpenVINO™ inference

Z80-μLM is a 2-bit quantized language model small enough to run on an 8-bit Z80 processor. Train conversational models in Python, export them as CP/M .COM binaries, and chat with your vintage computer.

Dataflow compiler for QNN inference on FPGAs

A pytorch quantization backend for optimum

A tool for converting ONNX files to LiteRT/TFLite/TensorFlow, PyTorch native code (nn.Module), TorchScript (.pt), state_dict (.pt), Exported Program (.pt2), and Dynamo ONNX. It also supports direct conversion from LiteRT to PyTorch.

[ICLR2024 spotlight] OmniQuant is a simple and powerful quantization technique for LLMs.

Official implementation of Half-Quadratic Quantization (HQQ)

Palette quantization library that powers pngquant and other PNG optimizers

Halo is an open-source framework built by White Circle for training large language and multimodal models

RDNA-native LLM inference engine in Rust.

[EMNLP 2024 & AAAI 2026] A powerful toolkit for compressing large models including LLMs, VLMs, and video generative models.

The Hailo Model Zoo includes pre-trained models and a full building and evaluation environment

37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.

Bilingual (中文+EN) ML / LLM / diffusion / agent interview cheat sheets for AI 秋招 — generated by ARIS /interview-cheatsheet, rendered by /render-html into single-file HTML, reads anywhere — plus a CV→DBLP-fact-checked academic homepage generator and hand-authored long-form blogs 🌱

Welcome to the official repository of SINQ! A novel, fast and high-quality quantization method designed to make any Large Language Model smaller while preserving accuracy [ICML 2026]

QKeras: a quantization deep learning library for Tensorflow Keras

Tool for onnx->keras or onnx->tflite. Hope this tool can help you.