AngelSlim

English | 简体中文

📣Latest News

• [26/10/26] We've released the Q4_K_M and Q8_0 quantized model for Hy-SimulMT on llama.cpp, supporting both the translation-only model and source text and translation). We also propose a PR to support the architecture and ASR inference!🔥🔥🔥
• llama.cpp support for Hy-SimulMT! Enable. Using Prima.cpp, we ran this 214 GB model on a Laptop 4090 (32GB RAM, 16GB VRAM) + a Server with 4x A4000 (32GB RAM, 16GB VRAM each), hitting 1.02 tokens/s. See the blog and the deployment guideline! 🔥🔥🔥
• [26/09/01] We've released the hy4 preview MIX-STQ1_0 version! Compressed from 1.5TB to 214 GiB with only a 0.7% performance drop on SWE-bench Pro. Using Prima.cpp, we ran this 214 GB model on a Laptop 4090 (32GB RAM, 16GB VRAM) + a Server with 4x A4000 (32GB RAM, 16GB VRAM each), hitting 1.02 tokens/s. See the blog and the deployment guideline! 🔥🔥🔥
• [26/07/29] We have open-sourced AngelSpec, a torch-native, disaggregated speculative-decoding training framework with support for a variety of draft methods, led by DFly and MTP + TTT, and released the MTP and DFly drafter weights for Hy3-A21B — DFly delivers up to 2.40× average end-to-end speedup over AR, and D-cut adds up to +15.7% throughput at high concurrency. [Paper] | [GitHub] | [Docs] | [Hugging Face] 🔥🔥🔥
• [26/07/20] We now support scale-only quantization-aware distillation on Megatron-Core for Qwen3-MoE and Hy3, with TP/EP/CP/SP distributed training. [Docs]
• [26/07/06] We now support FP8-Static quantization , SmoothQuant for Hy3 (MoE A21B).[Docs]
• [26/06/04] We have released Stem, a sparse attention algorithm that accelerates the Prefill stage of long-context LLMs by dynamically selecting top-k key blocks for block-sparse attention, significantly reducing latency while preserving generation quality. [Docs]
• [26/06/01] We have released DFlare, a block-diffusion speculative decoding framework with layer-wise fusion that achieves up to 5.52× end-to-end speedup. [Docs]
• [26/05/27] We have released D-Cut, an adaptive verification depth pruning technique for speculative decoding. [Docs]
• [26/05/20] We support Distillation for full-precision HuggingFace models and quantized QAT-style models, as detailed in the distillation documentation.
• [26/05/08] We have released STQ1_0 kernel for 1.25-bit model and given a PR to llama.cpp PR #22836 ! If you have any questions or suggestions for STQ_0, welcome to comment under the PR !🔥🔥🔥
• [26/04/29] We have released 2-bit and 1.25-bit versions of Tencent Hy-MT1.5-1.8B Translation Model: Hy-MT1.5-1.8B-2bit and Hy-MT1.5-1.8B-1.25bit. Additionally, we have make an offline translation demo for you to try out. We invite you to give it a spin! 🔥🔥🔥
• [26/04/23] We now support FP8-Static quantization for Hy3-preview (MoE A20B).
• [26/03/25] We have released DAQ, the quantization algorithm that preserves the knowledge acquired while the update of parameters is relatively small during post-training training.[Paper] | [Docs]
• [26/02/09] We have released HY-1.8B-2Bit, 2bit on-device large language model,[Huggingface].
• [26/01/13] We have released v0.3. We support the training and deployment of Eagle3 for all-scale LLMs/VLMs/Audio models, as detailed in the guidance documentation. And We released Sherry, the hardware-efficient 1.25 bit quantization algorithm [Paper] | [Code]🔥🔥🔥

• [25/11/05] We have released v0.2. Quantization support for new models, such as GLM-4.6, Qwen3-VL and Qwen3-Omni, open-sources the Eagle3 speculative decoding training framework, and updates the Diffusion model quantization tools.
• [25/09/30] We have released SpecExit, the reasoning early-exit algorithm: [Paper] | [Docs] | [vLLM Code]
• [25/09/26] We have released TEQUILA, the ternary quantization algorithm [Paper] | [Code]
• [25/09/24] We now support the PTQ quantization of NVFP4 for the Qwen3 series models. We also opensource Qwen3-32B-NVFP4 and Qwen3-235B-A22B-NVFP4 weights.
• [25/09/01] We now support ​FP8 quantization​ of the Hunyuan-MT-7B translation model. And enabled ​Torch inference and Benchmark evaluation​ for Eagle3. And implemented support for ​quantization and Cache​ for FLUX. And support ​quantization​ for the Seed-OSS.
• [25/08/06] We now support quantization for Hunyuan 0.5B/1.8B/4B/7B and multimodal model Qwen2.5VL 3B/7B/32B/72B, including FP8/INT4 algorithms, and quantization for DeepSeek-R1/V3 and Kimi-K2, including FP8-Static and W4A8-FP8 algorithms. We also opensource Hunyuan 1.8B/4B/7B series Eagle3 model weight.
• [25/07/04] We now support quantization for Hunyuan/Qwen2.5/Qwen3/DeepSeek-R1-Distill-Qwen and other models, including INT8/FP8/INT4 algorithms. We also opensource Qwen3 series Eagle3 model weight.

🌟Key Features

• Highly Integrated: This toolkit integrates mainstream compression algorithms into a unified framework, offering developers one-click access with exceptional ease of use.
• Continuous Innovation: Beyond integrating widely-used industry algorithms, we are continuously researching better compression algorithms, which will be gradually open-sourced in the future.
• Performance-Driven: We continuously optimize end-to-end performance in model compression workflows and algorithm deployment, such as enabling quantization of models like Qwen3-235B and DeepSeek-R1 on a single GPU.

💼Technical Overview

🛎️How to Use

1. Install AngelSlim

We recommend using pip to install the latest stable version of AngelSlim:

pip install angelslim

Alternatively, you can clone the repository and install from source in editable mode:

cd AngelSlim && python setup.py install

Note: AngelSlim integrates the speculative-decoding training framework AngelSpec as a git submodule (under third_party/AngelSpec/). Clone with --recursive, or initialize in an existing clone:

# Clone with submodules
git clone --recursive https://github.com/Tencent/AngelSlim.git
# Or, in an already-cloned repo
git submodule update --init --recursive

For more detailed installation instructions and platform-specific guidance, please refer to the Installation Documentation.

2. Quick Start

2.1 Speculative Decoding

AngelSlim's speculative-decoding training is powered by AngelSpec (git submodule under third_party/AngelSpec/) — a torch-native framework for training speculative-decoding draft models, featuring a rich set of draft architectures, target-model support, and production-scale training. It supports a variety of draft methods, led by DFly and MTP + TTT.

It follows a disaggregated design: inference engines run the frozen target model and extract multi-layer hidden states, which a Mooncake store streams over RDMA (no disk staging) to the FSDP2 training workers, while a controller handles batching, backpressure, and evaluation. Inference and training run on separate GPU pools and scale independently.

Capabilities:

• Multi-backend inference — vLLM (first-class), SGLang, and HuggingFace.
• Long-sequence training — Ulysses sequence parallelism (USP) for 128k+ contexts.
• Document-aware sequence packing — with cross-document attention isolation.
• Online evaluation — spec-decode acceptance rate measured during training.
• Multi-node — large MoE target models sharded across nodes over Mooncake RDMA.
• Vocabulary pruning — shrink the draft lm_head to a smaller token set at train or convert time.

# Initialize the submodule (see Install section above)
git submodule update --init --recursive
cd third_party/AngelSpec
# Install AngelSpec + the vLLM backend
pip install -e ".[vllm]"
# Hidden-state transport (not pulled in by the extras; install separately)
pip install mooncake-transfer-engine
# Single-node quickstart (4 GPUs: 2 inference + 2 training)
./examples/qwen3-8b-single-node/run.sh

For more details, see the technical report, AngelSpec docs, AngelSpec README, and Hugging Face weights.

2.2 LLM/VLM/Audio Model Quantization

After installing AngelSlim, you can launch static FP8 quantization for the Qwen3-1.7B model with the following one-command script:

python3 tools/run.py -c configs/qwen3/fp8_static/qwen3-1_7b_fp8_static.yaml

This example produces quantized model weights by performing PTQ calibration on a model loaded from HuggingFace.

For Hy3-preview (MoE A20B) FP8-Static quantization:

python tools/run.py -c configs/hunyuan/fp8_static/hunyuanv3_a20b_fp8_static_c8.yaml

To perform dynamic FP8 quantization on Qwen3-1.7B:

from angelslim.engine import Engine

slim_engine = Engine()
# Prepare model
slim_engine.prepare_model(model_name="Qwen", model_path="Qwen/Qwen3-1.7B",)
# Initialize compressor
slim_engine.prepare_compressor("PTQ", default_method="fp8_dynamic")
# Compress model
slim_engine.run()
# Save compressed model
slim_engine.save("./output")

For more details, please refer to the Quick Start Documentation.

2.3 Diffusion Model Quantization

Use the scripts/diffusion/run_diffusion.py for quantization and inference:

# Online quantization and inference
python scripts/diffusion/run_diffusion.py \
  --model-name-or-path black-forest-labs/FLUX.1-schnell \
  --quant-type fp8-per-tensor \
  --prompt "A cat holding a sign that says hello world" \
  --height 1024 --width 1024 --steps 4 --guidance 0.0 --seed 0

For more quantization inference methods, please refer to the Diffusion Model Quantization Documentation.

2.4 Token Compression (VLM)

AngelSlim provides a universal metadata-driven framework for vision token pruning and merging. You can quickly verify a compression strategy (e.g., VisionZip) with a smoke test:

python tools/test_universal_pruning.py \
    --model_path "Qwen/Qwen2.5-VL-3B-Instruct" \
    --config "configs/qwen2_5_vl/pruning/visionzip_r0.9.yaml"

For more details on implementing new strategies, please refer to the Token Compressor Documentation.

3. Deployment and Testing

3.1 Offline Inference

To test offline inference with a quantized model loaded via transformers.

python scripts/deploy/offline.py $MODEL_PATH "Hello, my name is"

Where MODEL_PATH is the path to the quantized model output.

3.2 API Service Deployment

After specifying the quantized model path MODEL_PATH, you can deploy an OpenAI-compatible API service using vLLM and SGLang inference frameworks.

• vLLM

  Use the following script to launch a vLLM server, recommended version vllm>=0.8.5.post1. For MOE INT8 quantized models, vllm>=0.9.0 is required.

  bash scripts/deploy/run_vllm.sh --model-path $MODEL_PATH --port 8080 -d 0,1,2,3 -t 4 -p 1 -g 0.8 --max-model-len 4096
  

  Where -d is the visible devices, -t is tensor parallel size, -p is pipeline parallel size, and -g is the GPU memory utilization.

• SGLang

  Use the following script to launch a SGLang server, recommended version sglang>=0.4.6.post1.

  bash scripts/deploy/run_sglang.sh --model-path $MODEL_PATH --port 8080 -d 0,1,2,3 -t 4 -g 0.8
  

3.3 Service Invocation

Invoke requests via OpenAI's API format.

bash scripts/deploy/openai.sh -m $MODEL_PATH -p "Hello, my name is" --port 8080 --max-tokens 4096 --temperature 0.7 --top-p 0.8 --top-k 20 --repetition-penalty 1.05 --system-prompt "You are a helpful assistant."

where -p is the input prompt.

3.4 Performance Evaluation

Evaluate the performance of quantized model using lm-evaluation-harness, recommended versionlm-eval>=0.4.8.

bash scripts/deploy/lm_eval.sh -d 0,1 -t 2 -g 0.8 -r $RESULT_PATH -b "auto" --tasks ceval-valid,mmlu,gsm8k,humaneval -n 0 $MODEL_PATH

where RESULT_PATH is the directory for saving test results, -b is batch size, --tasks specifies the evaluation tasks, and -n is the number of few-shot examples.

For more detaileds, please refer to the Deployment Documentation.

📈 Benchmark

1. Speculative Decoding

The results below come from AngelSpec — a torch-native, disaggregated speculative-decoding training framework, integrated into AngelSlim as a git submodule (see §2.1), with support for a variety of draft methods, led by DFly and MTP + TTT. [Paper] | [Docs] | [Hugging Face]

✨ Highlights

• 🏗️ Full-stack open source — the AngelSpec training framework plus Hy3-A21B MTP / DFly drafter weights and training code, all released at once.
• 🚀 DFly leads across the board — on Hy3-A21B, DFly delivers the highest throughput across all concurrency levels (4–64) and all six benchmarks — 1.98–2.40× average speedup over the AR baseline (peak 2.86× on code / math), and 10.5–11.8% faster than DFlash.
• 📈 Substantially higher accepted length — DFly reaches a mean accepted length of 4.79 — +30% over DFlash (3.69) and ~1.6× over MTP (3.00) — up to 5.52 on HumanEval.
• ⚡ D-cut squeezes high concurrency — on live traffic of a 295B model, it adds up to +15.7% throughput in the high-concurrency regime at near-zero cost (mean accepted length 2.50 → 2.46).
• 💬 MTP + TTT cracks chat — fixes the train/inference mismatch: mean acceptance rate 52.8% → 66.4%, mean accepted length 2.58 → 2.99.

1.1 🏆 Drafter Showdown — DFly Leads the Pack

Draft quality — mean accepted length. Measured on Qwen3-8B and Hy3-A21B across math / code / chat (temperature = 1, no thinking); bold marks the best drafter per target per benchmark. DFly leads on average — 5.41 on Qwen3-8B and 4.79 on Hy3-A21B — and tops nearly every benchmark, well ahead of DFlash and MTP.

End-to-end throughput on Hy3-295B-A21B. Output-token throughput (Tok/s) and speedup (Spd.) over the AR baseline (TP=8, temperature 1, concurrency c4–c64; each cell = 3×120s windows). Bold marks the best speedup per benchmark at each concurrency; MTP-3 uses 3 speculative tokens, DFlash-8 / DFly-8 use block length 8. DFly-8 wins the average at every concurrency — 1.98×–2.40× over AR, peaking at 2.86× on HumanEval.

1.2 ⚡ D-cut — Breaking the Concurrency Ceiling

When concurrency climbs, target verification becomes the bottleneck and rejected draft suffixes clog the batch. D-cut treats verification as a shared batch budget — ranking each request's verification depth by expected gain / cost, deepening drafts when the system is idle and trimming them when it's slammed. The payoff lands exactly where DFly plateaus: past concurrency 48, D-cut keeps turning load into throughput, up to +15.7% over DFly at near-zero quality cost (accepted length 2.50 → 2.46).

2. Quantization

The performance test results for selected models are shown below. For the complete benchmark, refer to the Benchmark documentation

2.1 Hunyuan Series Models

Benchmark results for the Hunyuan-Instruct model with FP8, INT4-AWQ and INT4-GPTQ quantization algorithms on datasets includingOlympiadBench, AIME 2024 and DROP:

2.2 Qwen3 Series Models

Benchmark results for Qwen3 series models with FP8-Static, FP8-Dynamic, INT4-GPTQ, and INT4-AWQ quantization algorithms on datasets including CEVAL, MMLU, GSM8K, and HUMANEVAL:

2.3 DeepSeek Series Models

Benchmark results for DeepSeek-R1-0528 series models with FP8-Block-Wise and W4A8-FP8 quantization algorithms on datasets including GPQA Diamond、AIME 2024、SimpleQA and LiveCodeBench：

• The above results are based on the average of 5 test runs deployed with TRT-LLM
• The hyperparameters used during evaluation are as follows:

{
 "top_k": 20,
 "top_p": 0.6,
 "temperature": 0.7,
 "output_seq_len": 32768,
 "max_input_seq_len": 16384
}

2.4 Qwen-VL Series Models

Qwen3-VL Benchmark

Benchmark results for Qwen3VL series models with BF16、FP8-Static and FP8-Dynamic quantization algorithms on datasets including MMMU_VAL、DocVQA_VAL and ChartQA_TEST：

Benchmark results for Qwen2.5VL series models with BF16、FP8-Static、FP8-Dynamic、INT4-GPTQ、INT4-AWQ quantization algorithms on datasets including MMMU_VAL、DocVQA_VAL and ChartQA_TEST：

2.5 Qwen-Omni Series Models

Qwen3-Omni Text to Text Benchmark

Benchmark results for Qwen3-Omni series models in BF16, FP8-Static, and FP8-Dynamic on aime25, gpqa_diamond, and mmlu_redux are as follows:

• The above evaluation results were obtained by deploying with the vLLM framework and averaging over 5 runs (vLLM only supports the thinker component).
• The hyperparameters used during evaluation are as follows:

{
 "top_p": 0.95,
 "temperature": 0.6,
 "do_sample": true,
 "max-model-len 65536": 65536
}

2.6 Other Models

Other models such as GLM-4.6, Qwen2.5, and Seed-OSS have been evaluated on benchmarks like CEVAL, MMLU, and GSM8K using quantization strategies including FP8-Static, FP8-Dynamic, INT4-GPTQ, and INT4-AWQ.

3. Token Compression (VLM)

We evaluated various vision token compression strategies on the Qwen2.5-VL-3B-Instruct model across multiple multimodal benchmarks. You can replicate these results using the following command:

python tools/run_pruning_eval.py \
    --model_path "Qwen/Qwen2.5-VL-3B-Instruct" \
    --configs "configs/qwen2_5_vl/pruning/visionzip_r0.9.yaml" \
    --tasks "textvqa" \
    --output_dir "./results/visionzip_test"

📝 License

The code for this project is open-sourced under the License for AngelSlim.

🔗 Citation

@article{angelslim2026,
  title={AngelSlim: A more accessible, comprehensive, and efficient toolkit for large model compression},
  author={Hunyuan AI Infra Team},
  journal={arXiv preprint arXiv:2602.21233},
  year={2026}
}

💬 Technical Discussion

• AngelSlim is developed by the Tencent Hunyuan AI Infra team, with new features being iteratively updated. If you have any questions or suggestions, please submit them on GitHub Issues or join our WeChat discussion group.

• ⭐ Star this repo to follow our latest progress. And if you are interested in joining us for an internship or full-time position, send your resume to: [email protected].