SVD-LLM

Introduction

SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression

Xin Wang1, Yu Zheng2, Zhongwei Wan1, Mi Zhang1
1The Ohio State University, 2Michigan State University

International Conference on Learning Representations (ICLR) 2025

SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression

Xin Wang, Samiul Alam, Zhongwei Wan, Hui Shen, Mi Zhang
The Ohio State University

Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL) 2025

Quick Start

Installation

Please keep the version of the transformers package exactly equal to 4.35.2 since the svd-compressed version of LLM has a slight change of model structure (in the component/. folder). Create and set up a conda environment with python version 3.9 (newer versions break some dependencies)

conda create -n compress python=3.9
conda activate compress

Clone and navigate to the repository

git clone https://github.com/AIoT-MLSys-Lab/SVD-LLM.git

Install requirements.txt

pip install -r requirements.txt

Quick Example

bash compress_llama.sh

This script would compress the LLaMA-7B model under 20% compression ratio and automatically run the evaluation code, including both perplexity and efficiency of the compressed model.

Step-by-Step Instructions of SVD-LLM

1. Truncation-Aware Data Whitening + SVD Compression

Under the low compression ratio (recommended ratio <= 0.3), we first run the data whitening of the LLM and saved the weight along with the whitening information.

python SVDLLM.py \
--step 1  \
--ratio COMPRESSION_RATIO \
--model HUGGINGFACE_MODEL_REPO \
--whitening_nsamples WHITENING_SAMPLE_NUMBER \
--dataset WHITENING_DATASET \
--seed SAMPLING_SEED \
--model_seq_len MODEL_SEQ_LEN \
--save_path WHITENING_INFO_SAVING_PATH

2. Parameter Update with Sequential Low-rank Approximation

We first update the compressed weight matrix U and then V with LoRA fine-tuning.

python LoRA.py \
--prune_model COMPRESSED_MODEL_PATH \
--data_path yahma/alpaca-cleaned \
--output_dir LORA_OUTPUT_PATH  \
--lora_r 8 \
--num_epochs 2 \
--learning_rate 1e-4 \
--batch_size 64

3. SVD-LLM + GPTQ

SVD-LLM can also be integrated with quantization methods to achieve a better compression. Here is the example of how to integrate SVD-LLM (20% compression ratio) with GPTQ-4bit to compress LLaMA-7B

bash svdllm_gptq.sh

4. Evaluation

• Perplexity Evaluation:

python SVDLLM.py \
--step 4 \
--model_path COMPRESSD_MODEL_SAVING_PATH  \

We use the same c4 dataset as in SparseGPT. Since the original dowload link is invalid, please directly download it from this link and add the two json files under the utils/. folder.

• Efficiency Evaluation:

python SVDLLM.py \
--step 5 \
--model_path COMPRESSD_MODEL_SAVING_PATH  \

Citation

If you find this work useful, please cite

@inproceedings{wang2025svdllm,
  title={{SVD}-{LLM}: Truncation-aware Singular Value Decomposition for Large Language Model Compression},
  author={Xin Wang and Yu Zheng and Zhongwei Wan and Mi Zhang},
  booktitle={International Conference on Learning Representations (ICLR)},
  year={2025},
  url={https://openreview.net/forum?id=LNYIUouhdt}
}