neural-compressor

intelApache-2.0活跃维护

面向 PyTorch、TensorFlow 和 ONNX Runtime 的模型压缩工具，支持 INT8/FP8/INT4 等低比特量化和稀疏化，帮大模型在低精度下保持精度并降低显存与推理成本。

SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime

v3.10发布于 2026-09-29 07:11 UTC2,730star +1 / 7 天332fork同步于 2026-11-10 11:43 UTC

模型压缩低比特量化稀疏化训练后量化剪枝

对比 GitHub 官网 auto-tuning awq fp4 gptq int4

怎么上手

由 README 提炼，执行前请确认

原文

pip install neural-compressor-pt
pip install neural-compressor-jax

硬件要求按 README 说明

README 未给出具体显存要求；主要面向 Intel Gaudi AI 加速器、Core Ultra、Xeon 可扩展处理器、Xeon CPU Max 系列及 Data Center GPU Flex/Max 系列，并有限测试 AMD CPU、ARM CPU 和 NVIDIA GPU。PyTorch 后端支持 CPU、GPU 和 HPU，需按硬件安装对应 PyTorch 版本。

支持 Apple Silicon有 Docker 镜像

下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼

没人工点评？写下你的经验

Intel 开源的模型压缩库，支持 PyTorch/JAX 上的低比特量化与稀疏化

面向 PyTorch、TensorFlow 和 ONNX Runtime 的模型压缩工具，支持 INT8/FP8/INT4 等低比特量化和稀疏化，帮大模型在低精度下保持精度并降低显存与推理成本。

核心能力

支持 INT8/FP8/MXFP8/INT4/MXFP4/NVFP4 低比特量化
提供静态、动态、SmoothQuant、仅权重量化与量化感知训练
集成 AutoRound 量化 LLaMA、Qwen、DeepSeek 等大模型与 VLM
提供 transformers 风格 API 加载 GPTQ 等预量化模型
覆盖 Intel Gaudi、Core Ultra、Xeon 及 Flex/Max GPU

容易踩的坑

PyTorch 后端需按硬件单独安装对应版本的 intel_extension_for_pytorch 或 torch；Gaudi 软件栈与 Neural Compressor 有版本映射，需按文档表格匹配；低精度函数多不支持 PT_HPU_LAZY_MODE=0，建议设为 1；首次加载 GPTQ 模型会转换格式并缓存，耗时较长。

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

用户点评

登录用户的真实使用记录，可按版本和硬件筛选

共 0 条

还没有人点评。第一个说出真实体验的人最有价值。

README 原文

未翻译

在 GitHub 打开

Intel® Neural Compressor

Architecture | Workflow | Documentations

Intel® Neural Compressor aims to provide popular model compression techniques such as Static Quantization, Dynamic Quantization, SmoothQuant, Weight-Only Quantization, Quantization-Aware Training, Mixed Precision, etc.

Support advanced quantization of Large Language Models (LLMs) and Vision-Language Models (VLMs) such as LLaMA, Qwen, DeepSeek, Flux, FramePack, etc., across diverse quantization techniques and low-precision data types through integration with AutoRound.
Support a wide range of Intel hardware such as Intel Gaudi Al Accelerators, Intel Core Ultra Processors, Intel Xeon Scalable Processors, Intel Xeon CPU Max Series, Intel Data Center GPU Flex Series, and Intel Data Center GPU Max Series with extensive testing; support AMD CPU, ARM CPU, and NVidia GPU with limited testing.

What's New

[2026/03] FP8 quantization support for Keras/JAX (experimental)
[2026/03] FP8 KV cache/Attention static quantization with AutoRound (experimental)
[2025/12] NVFP4 quantization experimental support
[2025/10] MXFP8 / MXFP4 quantization experimental support
[2025/09] FP8 dynamic quantization, including Linear, FusedMoE on Intel Gaudi AI Accelerators
[2025/05] FP8 static quantization of DeepSeek V3/R1 model on Intel Gaudi AI Accelerators
[2025/03] VLM quantization in transformers-like API on Intel CPU/GPU

Installation

Choose the necessary framework dependencies to install based on your deploy environment.

Install Framework for PyTorch Backend (on-demand)

Intel Neural Compressor supports PyTorch with CPU, GPU and HPU. Please install the corresponding PyTorch version based on your hardware environment.

Install intel_extension_for_pytorch for CPU
Install intel_extension_for_pytorch for Intel GPU
Use Docker Image with torch installed for HPU
Note: There is a version mapping between Intel Neural Compressor and Gaudi Software Stack, please refer to this table and make sure to use a matched combination.
Install torch for other platform

Install Neural Compressor from pypi

# Framework extension API + PyTorch dependency
pip install neural-compressor-pt
# Framework extension API + JAX dependency
pip install neural-compressor-jax

Note: Further installation methods can be found under Installation Guide. check out our FAQ for more details.

Getting Started

After successfully installing these packages, try your first quantization program. Following example code demonstrates FP8 Quantization, it is supported by Intel Gaudi2 AI Accelerator.
To try on Intel Gaudi2, docker image with Gaudi Software Stack is recommended, please refer to following script for environment setup. More details can be found in Gaudi Guide.

Run a container with an interactive shell, more info

docker run -it --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --net=host --ipc=host vault.habana.ai/gaudi-docker/1.24.0/ubuntu24.04/habanalabs/pytorch-installer-2.10.0:latest

Note: Since Habana software >= 1.21.0, PT_HPU_LAZY_MODE=0 is the default setting. However, most low-precision functions (such as convert_from_uint4) do not support this setting. Therefore, we recommend setting PT_HPU_LAZY_MODE=1 to maintain compatibility.

Run the example,

from neural_compressor.torch.quantization import (
    FP8Config,
    prepare,
    convert,
)

import torch
import torchvision.models as models

model = models.resnet18()
qconfig = FP8Config(fp8_config="E4M3")
model = prepare(model, qconfig)

# Customer defined calibration. Below is a dummy calibration
model(torch.randn(1, 3, 224, 224).to("hpu"))

model = convert(model)

output = model(torch.randn(1, 3, 224, 224).to("hpu")).to("cpu")
print(output.shape)

More FP8 quantization doc.

Following example code demonstrates weight-only large language model loading on Intel Gaudi2 AI Accelerator.

from neural_compressor.torch.quantization import load

model_name = "TheBloke/Llama-2-7B-GPTQ"
model = load(
    model_name_or_path=model_name,
    format="huggingface",
    device="hpu",
    torch_dtype=torch.bfloat16,
)

Note: Intel Neural Compressor will convert the model format from auto-gptq to hpu format on the first load and save hpu_model.safetensors to the local cache directory for the next load. So it may take a while to load for the first time.

Documentation

Selected Publications/Events

arXiv: Faster Inference of LLMs using FP8 on the Intel Gaudi (Mar 2025)
PyTorch landscape: PyTorch general optimizations (Mar 2025)
Blog on SqueezeBits: [Intel Gaudi] #4. FP8 Quantization (Jan 2025)

Note: View Full Publication List.

Additional Content

Communication

GitHub Issues: mainly for bug reports, new feature requests, question asking, etc.
Email: welcome to raise any interesting research ideas on model compression techniques by email for collaborations.