标签：GPTQ 量化

把 100+ LLM/VLM 的 SFT、DPO、PPO 收敛到一套配置和 CLI 上，省掉每个模型重写训练脚本

把 Whisper 从 PyTorch 换成 CTranslate2 推理，同精度下省显存、提吞吐，直接顶替 openai-whisper 的转写层

在 LLaMA 上扩中文词表再预训练的中文基座，产出的是 LoRA 增量而不是可直接加载的模型

本地部署的量化投研脚手架，把数据、策略、回测、实盘串成一条流水线，省去自己搭骨架

进程内全量扫描的量化向量索引，用 2/4 bit 压缩顶替 FAISS 在小内存机器上放不下的 embedding 集合

在单 CPU、8GB 内存上跑 2.78T 参数 MoE 推理的纯 C99 参考实现，用来验证流式权重加载这条路走不走得通

PyTorch 生态里做 4/8-bit 量化训练与推理的底层算子库，顶替的是手写量化 kernel 和显存不够时的模型裁剪

把 PNG 压成带 alpha 的 8-bit 调色板图，在构建流水线里顶替 optipng/zopfli 这类无损压缩器

Fast inference engine for Transformer models

[ICLR2025, ICML2025, NeurIPS2025 Spotlight] Quantized Attention achieves speedup of 2-5x compared to FlashAttention, without losing end-to-end metrics across language, image, and video models.

State-of-the-art LLM compression, built for production inference with vLLM

[ICLR2025 Spotlight] SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models

Your Cheat Sheet for AI Engineering Interview – Questions and Answers.

🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools

Sparsity-aware deep learning inference runtime for CPUs

PyTorch native quantization for training and inference

ComfyUI Plugin of Nunchaku

Run GLM-5.3 (744B), GLM-5.3-Flash (321B), DeepSeek-V4.1-Flash (748B) and MiniMax-M3.1-Flash (428B) on your own GPUs - one engine, one dashboard, OpenAI- and Anthropic-compatible API.

AIMET is a library that provides advanced quantization and compression techniques for trained neural network models.

SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime

Build, personalize and control your own LLMs. From data pre-processing to fine-tuning, xTuring provides an easy way to personalize open-source LLMs. Join our discord community: https://discord.gg/TgHXuSJEk6

Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.

Mastering Applied AI, One Concept at a Time

Run a 1-billion parameter LLM on a $10 board with 256MB RAM

A Python package for extending the official PyTorch that can easily obtain performance on Intel platform

Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率，量化感知训练的精度

PaddleSlim is an open-source library for deep model compression and architecture search.

Brevitas: neural network quantization in PyTorch

A toolkit to optimize ML models for deployment for Keras and TensorFlow, including quantization and pruning.

Build, train, optimize, and run computer vision models locally, from raw images to live inference. Open source, optimized for Intel® XPU (CUDA also supported).

LLM model quantization (compression) toolkit with HW acceleration support for Nvidia (CUDA), AMD (ROCm), Intel GPU (XPU), Huawei Ascent (NPU), and Intel/AMD/Apple CPU via HF, vLLM, and SGLang.

[ICML2025] SpargeAttention: A training-free sparse attention that accelerates any model inference.

Neural Network Compression Framework for enhanced OpenVINO™ inference

Z80-μLM is a 2-bit quantized language model small enough to run on an 8-bit Z80 processor. Train conversational models in Python, export them as CP/M .COM binaries, and chat with your vintage computer.

Dataflow compiler for QNN inference on FPGAs

A pytorch quantization backend for optimum

A tool for converting ONNX files to LiteRT/TFLite/TensorFlow, PyTorch native code (nn.Module), TorchScript (.pt), state_dict (.pt), Exported Program (.pt2), and Dynamo ONNX. It also supports direct conversion from LiteRT to PyTorch.

[ICLR2024 spotlight] OmniQuant is a simple and powerful quantization technique for LLMs.

Official implementation of Half-Quadratic Quantization (HQQ)

Palette quantization library that powers pngquant and other PNG optimizers

Halo is an open-source framework built by White Circle for training large language and multimodal models

RDNA-native LLM inference engine in Rust.

[EMNLP 2024 & AAAI 2026] A powerful toolkit for compressing large models including LLMs, VLMs, and video generative models.

The Hailo Model Zoo includes pre-trained models and a full building and evaluation environment

37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.

Bilingual (中文+EN) ML / LLM / diffusion / agent interview cheat sheets for AI 秋招 — generated by ARIS /interview-cheatsheet, rendered by /render-html into single-file HTML, reads anywhere — plus a CV→DBLP-fact-checked academic homepage generator and hand-authored long-form blogs 🌱

Welcome to the official repository of SINQ! A novel, fast and high-quality quantization method designed to make any Large Language Model smaller while preserving accuracy [ICML 2026]

QKeras: a quantization deep learning library for Tensorflow Keras

Tool for onnx->keras or onnx->tflite. Hope this tool can help you.