OneCompression

Fujitsu One Compression

Fujitsu One Compression (OneComp) is a Python package for LLM compression.

⚡ Just one line.

onecomp <generative AI>

That's all you need. OneComp detects your GPU VRAM, picks the best bit-width per layer, quantizes with error propagation, evaluates, and saves — fully automatic.

# Example
onecomp meta-llama/Llama-2-7b-hf

Or from Python:

from onecomp import Runner

Runner.auto_run(model_id="meta-llama/Llama-2-7b-hf")

📖 Documentation

Full documentation is available at https://FujitsuResearch.github.io/OneCompression/.

📦 Features

• Quantization Error Propagation (QEP): A post-training quantization method that corrects quantization errors by propagating them to subsequent layers, improving the accuracy of quantized LLMs. See Arai & Ichikawa, NeurIPS 2025 for details. The original reference implementation is available at FujitsuResearch/qep.
• Layer-Projected Coordinate Descent (LPCD): A unified Post Training Quantization (PTQ) framework that extends layer-wise quantization to arbitrary submodules by optimising relaxed objectives and projecting the solutions with layer-wise quantizers. See Ichikawa et al., 2025 for details.
• vLLM Plugin Integration: Serve OneComp-quantized models with vLLM via built-in plugins for DBF and Mixed-GPTQ quantization methods. Pair with Open WebUI for a ChatGPT-like chat experience on your local machine.
• AutoBit: Mixed-precision quantization with ILP-based bitwidth assignment. Automatically estimates the target bitwidth from available VRAM and assigns per-layer bitwidths to minimize quantization error under the memory budget.
• JointQ: Joint quantization method that optimizes weight assignments and scale parameters simultaneously for improved quantization accuracy. Supports group-wise quantization (e.g., 4-bit, groupsize=128).
• MDBF (Multi-Envelope Double Binary Factorization): A binary factorization quantizer that approximates each weight matrix as a sum of multi-path sign matrices with multi-scale FP16 envelopes, generalizing DBF and LittleBit for aggressive low-bit (sub-1-bit) compression. Supports ADMM/gradient refinement, activation-aware initialization, and a GemLite-accelerated 1-bit inference path. See the MDBF guide for details.
• Block-wise PTQ: Post-quantization block-wise distillation that minimises intermediate-representation MSE against an FP16 teacher model at Transformer-block granularity. Includes Phase 1 (greedy per-block optimisation) and Phase 2 CBQ (cross-block sliding-window optimisation). Supports GPTQ, DBF, and OneBit quantizers.
• Global PTQ: Globally optimise GPTQ scales/zeros, DBF scaling factors, and MDBF amplitude parameters through KL distillation from a full-precision teacher. Supports optional MDBF binary-factor training with STE, single-device and distributed execution with DeepSpeed, and configurable teacher device placement. See the Global PTQ guide.
• MoE Router Fine-Tuning: Adapt routing decisions after expert quantization by training only router parameters with next-token prediction loss, while experts and other weights remain frozen. Results use the normal quantized-model save/load workflow. See the router fine-tuning guide.
• LoRA SFT Post-Process: Fine-tune quantized models with LoRA adapters for accuracy recovery or domain-specific knowledge injection. Supports SFT loss, teacher distillation, and intermediate block alignment.
• Rotation Preprocessing: SpinQuant/OstQuant-based rotation preprocessing that reduces quantization error by learning optimal rotation matrices before quantization. Rotation/scaling matrices are absorbed into model weights, with online Hadamard hooks automatically registered at load time. Supports Llama and Qwen3 architectures.
• Web Dashboard (HPC): A browser-based dashboard for launching quantization jobs, deploying models, and validating chat-based inference in HPC environments. See dashboard/README.md for details.
• GGUF Export & Hugging Face Hub Integration: Convert models to the GGUF v3 format (F16) for llama.cpp/Ollama with a dependency-free built-in writer, generate model cards with quantization recipes and evaluation results, and push save directories to the Hugging Face Hub. Supports Llama (SentencePiece or Llama-3-style BPE) and Qwen2 (BPE) architectures, including multi-EOS stop-token mapping. See the GGUF Export guide.
• OpenVINO Export: Export supported OneComp GPTQ 4-bit text-generation checkpoints to OpenVINO IR using an isolated, dependency-locked OpenVINO 2026.3.1 environment and conversion example. Checkpoint metadata is normalized in a temporary copy without modifying the source. See the OpenVINO export guide.
• (TBD)

🤖 Supported Models

OneComp has been verified with the following model architectures. Other Hugging Face-compatible models may work but are currently untested.

#	Architecture	Verified Models	Status
1	Llama	TinyLlama, Llama-2, Llama-3	✅ Verified
2	Qwen3	Qwen3-0.6B ~ 32B, Qwen3-30B-A3B	✅ Verified
3	Gemma	Gemma 2, Gemma 3, Gemma 4, Gemma 4-26B-A4B	✅ Verified
4	GPT-OSS	openai/gpt-oss-20b, gpt-oss-120b	✅ Verified
5	Qwen3.6	Qwen3.6-27B, Qwen3.6-35B-A3B	✅ Verified
6	Qwen3.8	Qwen3.8-27B	✅ Verified

Note: Support for additional architectures is planned. Contributions and test reports are welcome.

🔧 Installation

for users (pip)

1. Install PyTorch

Please install the appropriate version of PyTorch.

✅ CPU-only

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu

✅ CUDA-enabled

Choose the appropriate CUDA version for your system:

CUDA Version	Installation Command
CUDA 11.8	pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
CUDA 12.1	pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
CUDA 12.4	pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
CUDA 12.6	pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126
CUDA 12.8	pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128
CUDA 13.0	pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu130

Check your CUDA version:

nvcc --version

or

nvidia-smi

Verify PyTorch GPU support:

import torch
print(torch.cuda.is_available())

✅ macOS (MPS)

On macOS, install PyTorch from PyPI (default wheels include MPS support). You do not need the CUDA index URLs above.

pip install torch torchvision torchaudio

Verify MPS:

import torch
print(torch.backends.mps.is_available())

Then install OneComp from PyPI (see step 2 below). GPTQ quantization and Hugging Face generate() inference on MPS are supported; vLLM serving requires Linux with an NVIDIA GPU. An editable install from a git clone is not required for MPS use — see for developers (pip) only if you are contributing to OneComp.

MPS device placement (GPTQ vs QEP)
With device="mps", calibration and model forward passes can run on the GPU. The bottleneck is usually not missing Cholesky ops on MPS (recent PyTorch builds implement them); the implementation splits work as follows:

• GPTQ (run_gptq): Hessian and weights are moved to CPU for the full column-wise loop (including inverse-Hessian Cholesky). If that loop stayed on MPS, quantize() would call maxq.item() once per column; each call triggers per-column host sync (wait for pending MPS ops, then read one scalar—not a full Hessian/weight copy every column)—often several times slower than CPU on Apple Silicon (e.g. ~4× in internal benchmarks with PyTorch 2.12). Keeping GPTQ on CPU avoids that overhead. With mse=True, find_params also calls quantize() in a grid loop and benefits from the same CPU placement.
• QEP weight correction (adjust_weight, when QEP correction runs—typically qep=True with error propagation enabled): Per-layer work stays on MPS (e.g. weight @ delta_hatX, diagonal damping). Only the Cholesky solve uses CPU via _safe_cholesky_and_solve (one solve per layer, not per column); moving all of QEP to CPU does not materially improve speed. The subsequent GPTQ step still uses the CPU path above.

DBF-based AutoBit fallback is not supported on MPS.

2. Install onecomp

Once PyTorch is installed, you can install onecomp:

pip install onecomp

To enable multi-GPU training for Global PTQ (DeepSpeed), install with the distributed extra:

pip install "onecomp[distributed]"

for developers (uv : recommended)

Install uv

uv is a fast Python package and project manager written in Rust. It offers a drop-in replacement for pip and pip-tools while also managing virtual environments and Python installations. With its Rust-based dependency resolver and the uv.lock lockfile, uv provides deterministic and reproducible environments across development machines and CI pipelines.

# install uv (for macOS or Linux)
curl -LsSf https://astral.sh/uv/install.sh | sh

git clone https://github.com/FujitsuResearch/OneCompression.git
cd OneCompression

The uv sync command creates a Python virtual environment and installs all dependent libraries.

Linux (CUDA quantization / vLLM)

uv sync --extra cu128 --extra dev --extra visualize

The --extra cu128 option installs the CUDA-enabled version of PyTorch (along with torchvision from the same CUDA index). Replace cu128 with the appropriate variant for your environment: cpu, cu118, cu121, cu124, cu126, cu128, or cu130. PyTorch will be automatically downloaded by uv, so you do not need to install it beforehand.

macOS (development / MPS inference)

uv sync --extra mps --extra dev --extra visualize

On macOS, use --extra mps only. CUDA extras (cu118–cu130), --extra cpu (Linux-only), and --extra vllm are not supported on macOS. After uv sync, you can run GPTQ quantization and Hugging Face generate() inference on MPS; vLLM serving still requires Linux with an NVIDIA GPU. See the MPS device placement (GPTQ vs QEP) note under macOS (MPS) above for why GPTQ runs on CPU while QEP correction uses MPS.

Adding --extra dev installs development tools (black, pre-commit, pytest, pylint). Adding --extra visualize installs matplotlib for visualization features. Adding --extra distributed installs DeepSpeed for Global PTQ multi-GPU training. Adding --extra hydra installs hydra-core for the example scripts and model_validation/ runners that use Hydra-based configuration.

To use vLLM for serving quantized models on Linux, add --extra vllm together with --extra cu130:

uv sync --extra cu130 --extra dev --extra visualize --extra vllm

Note: --extra vllm is only compatible with --extra cu130. Recent vLLM releases require torch>=2.10, whose wheels are only published for the cu130 index. Combining --extra vllm with cpu / mps / cu118 / cu121 / cu124 / cu126 / cu128 is rejected by uv at lock time.

Note: --extra vllm may take a long time on the first run if a pre-built xformers wheel is not available for your Python/CUDA combination (e.g. Python 3.13). Using Python 3.12 typically avoids this.

Running commands (uv environment)

In the environment created by uv sync, you can run commands in two ways:

Option 1: Use uv run (no activation needed)

uv run pytest tests/ -v
uv run python example/example_gptq.py
uv run black --check onecomp/

Option 2: Activate the virtual environment (traditional approach)

source .venv/bin/activate
pytest tests/ -v
python example/example_gptq.py
black --check onecomp/

for developers (pip)

Note: The editable install below is for developing OneComp from a local clone. macOS users who only want MPS inference or quantization should use the for users (pip) flow (pip install torch then pip install onecomp from PyPI); pip install -e is not needed for MPS.

git clone <git repository URL>
cd OneCompression

# First, install PyTorch with CUDA support for your environment
pip install torch --index-url https://download.pytorch.org/whl/cu128
# Then install onecomp with development dependencies
pip install -e ".[dev]"

Replace cu128 with the appropriate variant for your environment: cpu, cu118, cu121, cu124, cu126, cu128, or cu130.

Pre-commit

After installing development dependencies (--extra dev with uv, or pip install -e ".[dev]" with pip), register the Git hooks once:

pre-commit install

On every git commit, the following checks run automatically:

Hook	Description
black	Code formatting (line length 99)
isort	Import sorting
no-japanese	Forbid Japanese characters in text files (.md and .gitignore are excluded)
copyright-header	Verify the Fujitsu copyright header in Python files
no-email-address	Forbid email addresses in Python files

Common commands:

# Run hooks on staged files only
pre-commit run

# Run hooks on all files (useful after first install or config changes)
pre-commit run --all-files

# Run a specific hook
pre-commit run black --all-files

# With uv (no activation needed)
uv run pre-commit run --all-files

Building Documentation Locally

--extra docs alone is sufficient (no PyTorch mps / cu* extra required):

uv sync --extra docs
uv run mkdocs serve

Then open http://127.0.0.1:8000 in your browser.

📓 Tutorial Notebook

Interactive walkthrough in Jupyter or Google Colab — RTN visualization, Runner.auto_run, and vLLM chat inference.

See notebook/README.md for local setup, or the Tutorial Notebook guide in the docs.

🚀 Examples

Category	Script	Description
Tutorial	01_tutorial.ipynb	Interactive notebook (Jupyter / Colab)
Quantization	example_gptq.py	GPTQ quantization
	example_qep_gptq.py	GPTQ + QEP (error propagation)
	example_lpcd_gptq.py	GPTQ + QEP + LPCD quantization
	example_jointq.py	JointQ quantization
	example_mdbf.py	MDBF quantization
	example_autobit.py	AutoBit mixed-precision quantization
	example_auto_run.py	AutoBit with automatic VRAM estimation
Calibration	example_custom_calibration.py	Custom calibration dataset with CalibrationConfig
Save / Load	example_save_load.py	Save and load quantized models
	example_reload_post_process_resave.py	Reload, post-process, and re-save a quantized checkpoint
Rotation Preprocessing	example_llama_preprocess_rtn.py	Rotation preprocessing + RTN (TinyLlama)
	example_preprocess_save_load.py	Save and load rotation-preprocessed quantized models
Post-Process	example_blockwise_ptq.py	Block-wise PTQ via Runner.run() with packed buffers by default
	example_blockwise_global_ptq.py	BlockWisePTQ → GlobalPTQ in a single Runner, followed by safetensors save/load
	example_blockwise_global_ptq_staged.py	Staged BlockWisePTQ → GlobalPTQ across save/load boundaries with accumulated post-process metadata
	example_global_ptq.py	Global PTQ with packed buffers by default and HF-compatible safetensors output
	example_global_ptq_dbf.py	Global PTQ with the DBF backend and HF-compatible safetensors output
	example_global_ptq_mdbf.py	Global PTQ with the MDBF backend and HF-compatible safetensors output
	example_global_ptq_distributed.py	Multi-GPU Global PTQ with DeepSpeed / torchrun and safetensors output
	example_router_fine_tuning.py	Router-only next-token fine-tuning for a quantized MoE model
	example_lora_sft.py	LoRA SFT post-quantization fine-tuning
	example_lora_sft_knowledge.py	LoRA SFT knowledge injection
	example_lora_sft_knowledge_jointq.py	LoRA SFT knowledge injection on a JointQ-quantized model
	example_lora_gptq_vllm_inference.py	GPTQ + LoRA SFT, HF-compatible safetensors + PEFT sidecar, and vLLM inference
vLLM	example_gptq_vllm_inference.py	GPTQ + QEP quantization and vLLM inference
	example_gptq_vllm_qwen36_inference.py	GPTQ quantization and vLLM inference for Qwen3.6 (full-wrapper save)
	example_gptq_vllm_gptoss_inference.py	GPTQ quantization and vLLM inference for gpt-oss
	example_jointq_vllm_inference.py	JointQ quantization and vLLM inference
	example_autobit_vllm_inference.py	AutoBit quantization and vLLM inference
	example_dbf_vllm_inference.py	DBF quantization and vLLM inference
CPU Inference	example_gptq_gguf_cpu.py	Export GPTQ quantized models to GGUF and run CPU inference
	example_mixed_gptq_gguf_cpu.py	Export Mixed-GPTQ quantized models to GGUF and run CPU inference
	example_serve_cpu.py	Serve a GGUF model with llama.cpp on CPU

🔌 vLLM Inference

OneComp-quantized models can be served with vLLM: GPTQ, JointQ, and RTN models use vLLM's built-in GPTQ plugin, while DBF and Mixed-GPTQ models are served via OneComp's own plugins. Combined with Open WebUI, you can chat with your quantized model through a ChatGPT-like browser interface — entirely on your local machine.

# uv users (vLLM requires cu130; see Installation for details)
uv sync --extra cu130 --extra vllm

# pip users
pip install vllm

See the vLLM Inference guide for details, including Open WebUI setup instructions.

GPT-OSS (mixed_gptq)

GPT-OSS models (openai/gpt-oss-20b, openai/gpt-oss-120b) need extra steps beyond the standard vLLM plugin flow:

1. Quantize with GPTQ(wbits=4) and keep the MoE experts 4-bit via Runner(..., moe_quant_experts=True) with groupsize=64. GPT-OSS hidden_size (2880) is not divisible by 128, so the experts must use group_size=64 for vLLM's WNA16 MoE kernel.
2. Before LLM(...), apply vLLM runtime patches: python -m vllm_plugins.patches.apply_all
3. Set export VLLM_USE_DEEP_GEMM=0

The --extra vllm dependency set includes conch-triton-kernels. On NVIDIA Blackwell (B200, sm100), vLLM 0.20 needs Conch for mixed_gptq linear layers when the Marlin kernel cannot handle GPT-OSS weight shapes. If you install vLLM manually with pip install vllm, also run pip install conch-triton-kernels.

See the GPT-OSS guide for HF save/load, patch details, and verification scripts.

📬 Contact Us

• For technical questions and feature requests, please use GitHub Issues.
• For security vulnerabilities, please do not open a public Issue. See our Security Policy for how to report them privately.
• For collaborations, partnerships, and other inquiries, please contact us at [email protected].

📄 License

OneComp is licensed under the MIT License.

The dependencies installed with OneComp are separate open-source software (OSS) projects and are distributed under their respective licenses. Their licenses may change when the dependencies are updated, so please check the license terms of the installed versions as well.

Citation

OneComp technical report:

@misc{ichikawa2026onecomponelinerevolutiongenerative,
      title={OneComp: One-Line Revolution for Generative AI Model Compression}, 
      author={Yuma Ichikawa and Keiji Kimura and Akihiro Yoshida and Yudai Fujimoto and Hiroki Tokura and Yamato Arai and Yoshiyuki Ishii and Yusei Kawakami and Genki Shikada and Achille Jacquemond and Yoshihiko Fujisawa and Katsuki Fujisawa and Takumi Honda and Akira Sakai},
      year={2026},
      eprint={2603.28845},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2603.28845}, 
}

QEP (Quantization Error Propagation):

@inproceedings{
arai2025quantization,
title={Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization},
author={Yamato Arai and Yuma Ichikawa},
booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},
year={2025},
url={https://openreview.net/forum?id=a3l3K9khbL}
}

LPCD (Layer-Projected Coordinate Descent):

@article{ichikawa2025lpcd,
title={LPCD: Unified Framework from Layer-Wise to Submodule Quantization},
author={Yuma Ichikawa and Yudai Fujimoto and Akira Sakai},
journal={arXiv preprint arXiv:2512.01546},
year={2025},
url={https://arxiv.org/abs/2512.01546}
}

MDBF (Multi-Envelope Double Binary Factorization):

@misc{ichikawa2025bitsmultienvelopedoublebinary,
      title={More Than Bits: Multi-Envelope Double Binary Factorization for Extreme Quantization}, 
      author={Yuma Ichikawa and Yoshihiko Fujisawa and Yudai Fujimoto and Akira Sakai and Katsuki Fujisawa},
      year={2025},
      eprint={2512.24545},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2512.24545}, 
}