awesome-data-llm

A Survey of LLM × DATA

English | 中文

A collection of papers and projects related to LLMs and corresponding data-centric methods.

Other publicly-available materials: [Slides]

If you find our survey useful, please cite the paper:

@article{LLMDATASurvey,
    title={A Survey of LLM × DATA},
    author={Xuanhe Zhou and Junxuan He and Wei Zhou and Haodong Chen and Zirui Tang and Haoyu Zhao and Xin Tong and Guoliang Li and Youmin Chen and Jun Zhou and Zhaojun Sun and Binyuan Hui and Shuo Wang and Conghui He and Zhiyuan Liu and Jingren Zhou and Fan Wu},
    year={2025},
    journal={arXiv preprint arXiv:2505.18458},
    url={https://arxiv.org/abs/2505.18458}
}
@article{tangllmasanalyst,
    title={LLM/Agent-as-Data-Analyst: A Survey},
    author={Zirui Tang and Weizheng Wang and Zihang Zhou and Yang Jiao and Bangrui Xu and Boyu Niu and Xuanhe Zhou and Guoliang Li and Yeye He and Wei Zhou and Yitong Song and Cheng Tan and Bin Wang and Conghui He and Xiaoyang Wang and Fan Wu},
    year={2025},
    journal={arXiv preprint arXiv:2509.23988},
    url={https://arxiv.org/abs/2509.23988}
}
@article{zhoullmpreparationsurvey,
    title={Can LLMs Clean Up Your Mess? A Survey of Application-Ready Data Preparation with LLMs},
    author={Wei Zhou and Jun Zhou and Haoyu Wang and Zhenghao Li and Qikang He and Shaokun Han and Guoliang Li and Xuanhe Zhou and Yeye He and Chunwei Liu and Zirui Tang and Bin Wang and Shen Tang and Kai Zuo and Yuyu Luo and Zhenzhe Zheng and Conghui He and Jingren Zhou and Fan Wu},
    year={2026},
    journal={arXiv preprint arXiv:2601.17058},
    url={https://arxiv.org/abs/2601.17058}
}

🌤 The IaaS Concept of DATA4LLM

The IaaS concept for LLM data (phonetically echoing Infrastructure as a Service) defines the characteristics of high-quality datasets along four key dimensions: (1) Inclusiveness ensures broad coverage across domains, tasks, sources, languages, styles, and modalities. (2) Abundance emphasizes sufficient and well-balanced data volume to support scaling, fine-tuning, and continual learning without overfitting. (3) Articulation requires clear, coherent, and instructive content with step-by-step reasoning to enhance model understanding and task performance. (4) Sanitization involves rigorous filtering to remove private, toxic, unethical, and misleading content, ensuring data safety, neutrality, and compliance.

🌟 LLM/Agent-as-Data-Analyst

We observe the evolution of LLM/Agent-as-Data-Analyst techniques follows a five-dimension trajectory: (1) Data Modality (homogeneous → heterogeneous); (2) Analysis Functionality (literal → semantic); (3) Knowledge Scope (closed-world →open-world); (4) Tool Integration (tool-coupled → tool-assisted); (5) Development Autonomy (manual → fully autonomous).

📤 LLM-Enhanced Application-Ready Data Preparation

We present a systematic survey of LLM-enhanced data preparation, an emerging paradigm driven by growing demands for application-ready data, advances in large language models, and agent-oriented data infrastructures.

• (1) Paradigm Shift Identification: We reveal the transition from rule-based, model-specific pipelines to prompt-driven, context-aware, and agentic preparation workflows;
• (2) Task-Centric Taxonomy: We organize existing methods into data cleaning, data integration, and data enrichment;
• (3) Methodological Survey: We review representative techniques for each task, highlighting their strengths and limitations;
• (4) Empirical Landscape Analysis: We examine commonly used datasets and evaluation practices;
• (5) Research Roadmap: We outline open challenges and future directions toward scalable, reliable, and robust LLM-based data preparation systems.

Table of Contents

• Datasets

• 0 Data Characteristics across LLM Stages

  • Data for Pretraining
  • Data for Continual Pre-training
  • Data for Supervised Fine-Tuning (SFT)
  • Data for Reinforcement Learning (RL)
  • Data for Retrieval-Augmented Generation (RAG)
  • Data for LLM Evaluation
  • Data for LLM Agents

• 1 Data Processing for LLM

  • 1.1 Data Acquisition
  • 1.2 Data Deduplication
  • 1.3 Data Filtering
  • 1.4 Data Domain Selection
  • 1.5 Data Mixing
  • 1.6 Data Distillation and Synthesis
  • 1.7 End-to-End Data Processing Pipelines

• 2 Data Storage for LLM

  • 2.1 Data Formats
  • 2.2 Data Distribution
  • 2.3 Data Organization
  • 2.4 Data Movement
  • 2.5 Data Fault Tolerance
  • 2.6 KV Cache

• 3 Data Serving for LLM

  • 3.1 Data Shuffling
  • 3.2 Data Compression
  • 3.3 Data Packing
  • 3.4 Data Provenance

• 4 LLM for Data Management

  • 4.1 LLM for Analysis
  • 4.2 LLM for Data System Optimization

• 5 LLM as Data Analyst

  • 5.1 LLM for Structured Data Analysis
  • 5.2 LLM for Semi-Structured Data Analysis
  • 5.3 LLM for Unstructured Data Analysis
  • 5.4 LLM for Heterogeneous Data Analysis

• 6 LLM-Enhanced Application-Ready Data Preparation

  • 6.1 Data Cleaning
  • 6.2 Data Integration
  • 6.3 Data Enrichment

Datasets

1. CommonCrawl: A massive web crawl dataset covering diverse languages and domains; widely used for LLM pretraining. [Source]

2. The Stack: A large-scale dataset of permissively licensed source code in multiple programming languages; used for code LLMs. [HuggingFace]

3. RedPajama: A replication of LLaMA’s training data recipe with open datasets; spans web, books, arXiv, and more. [Github]

4. SlimPajama-627B-DC: A deduplicated and filtered subset of RedPajama (627B tokens); optimized for clean and efficient training. [HuggingFace]

5. Alpaca-CoT: Instruction-following dataset enhanced with Chain-of-Thought (CoT) reasoning prompts; used for dialogue fine-tuning. [Github]

6. LLaVA-Pretrain: A multimodal dataset with image-text pairs for training visual language models like LLaVA. [HuggingFace]

7. Wikipedia: Structured and encyclopedic content; a foundational source for general-purpose language models. [HuggingFace]

8. C4: A cleaned version of CommonCrawl data, widely used in models like T5 for high-quality web text. [HuggingFace]

9. BookCorpus: Contains free fiction books; often used to teach models long-form language understanding. [HuggingFace]

10. Arxiv: Scientific paper corpus from arXiv, covering physics, math, CS, and more; useful for academic language modeling. [HuggingFace]

11. PubMed: Biomedical literature dataset from the PubMed database; key resource for medical domain models. [Source]

12. StackExchange: Community Q&A data covering domains like programming, math, philosophy, etc.; useful for QA and dialogue tasks. [Source]

13. OpenWebText2: A high-quality open-source web text dataset based on URLs commonly cited on Reddit; GPT-style training corpus. [Source]

14. OpenWebMath: A dataset of math questions and answers; designed to improve mathematical reasoning in LLMs. [HuggingFace]

15. Falcon-RefinedWeb: Filtered web data used in training Falcon models; emphasizes data quality through rigorous preprocessing. [HuggingFace]

16. CCI 3.0: A large-scale multi-domain Chinese web corpus, suitable for training high-quality Chinese LLMs. [HuggingFace]

17. OmniCorpus: A unified multimodal dataset (text, image, audio) designed for general-purpose AI training. [Github]

18. WanJuan3.0: A diverse and large-scale Chinese dataset including news, fiction, QA, and more; released by OpenDataLab. [Source]

0 Data Characteristics across LLM Stages

⬆️top

Data for Pretraining

1. OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents Hugo Laurençon, Lucile Saulnier, Léo Tronchon, et al. NeurIPS 2023. [Paper]
2. Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books Yukun Zhu, Ryan Kiros, Richard Zemel, et al. ICCV 2015. [Paper]

Data for Continual Pre-training

1. MedicalGPT: Training Medical GPT Model Ming Xu. 2025. [Github]
2. BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmark Dakuan Lu, Hengkui Wu, Jiaqing Liang, et al. arXiv 2023. [Paper]

Data for Supervised Fine-Tuning (SFT)

General Instruction Following

1. Free dolly: Introducing the world’s first truly open instruction-tuned llm Mike Conover, Matt Hayes, Ankit Mathur, et al. 2023. [Source]

Specific Domain Usage

1. MedicalGPT: Training Medical GPT Model [Github]
2. DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Services Shengbin Yue, Wei Chen, Siyuan Wang, et al. arXiv 2023. [Paper]

Data for Reinforcement Learning (RL)

RLHF

1. MedicalGPT: Training Medical GPT Model [Github]
2. UltraFeedback: Boosting Language Models with Scaled AI Feedback Ganqu Cui, Lifan Yuan, Ning Ding, et al. ICML 2024. [Paper]

RoRL

1. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning DeepSeek-AI. arXiv 2025. [Paper]
2. Kimi k1.5: Scaling Reinforcement Learning with LLMs Kimi Team. arXiv 2025. [Paper]

Data for Retrieval-Augmented Generation (RAG)

1. DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Services [Paper]
2. DH-RAG: A Dynamic Historical Context-Powered Retrieval-Augmented Generation Method for Multi-Turn Dialogue Feiyuan Zhang, Dezhi Zhu, James Ming, et al. arXiv 2025. [Paper]
3. Medical Graph RAG: Towards Safe Medical Large Language Model via Graph Retrieval-Augmented Generation Junde Wu, Jiayuan Zhu, Yunli Qi, et al. arXiv 2024. [Paper]
4. ERAGent: Enhancing Retrieval-Augmented Language Models with Improved Accuracy, Efficiency, and Personalization Yunxiao Shi, Xing Zi, Zijing Shi, et al. arXiv 2024. [Paper]
5. PersonaRAG: Enhancing Retrieval-Augmented Generation Systems with User-Centric Agents Saber Zerhoudi, Michael Granitzer. arXiv 2024. [Paper]

Data for LLM Evaluation

1. StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs Jialin Yang, Dongfu Jiang, Lipeng He, et al. TMLR 2025. [Paper] [Code] [Dataset]
2. MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI Xiang Yue, Yuansheng Ni, Kai Zhang, et al. CVPR 2024. [Paper]
3. LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Models Haitao Li, You Chen, Qingyao Ai, et al. NeurIPS 2024. [Paper]
4. What disease does this patient have? a large-scale open domain question answering dataset from medical exams Di Jin, Eileen Pan, Nassim Oufattole, et al. AAAI 2021. [Paper]
5. Evaluating Large Language Models Trained on Code Mark Chen, Jerry Tworek, Heewoo Jun, et al. arXiv 2021. [Paper]

Data for LLM Agents

1. STeCa: Step-level Trajectory Calibration for LLM Agent Learning Hanlin Wang, Jian Wang, Chak Tou Leong, Wenjie Li. arXiv 2025. [Paper]
2. Large Language Model-Based Agents for Software Engineering: A Survey Junwei Liu, Kaixin Wang, Yixuan Chen, et al. arXiv 2024. [Paper]
3. Advancing LLM Reasoning Generalists with Preference Trees Lifan Yuan, Ganqu Cui, Hanbin Wang, et al. arXiv 2024. [Paper]
4. Tool Learning in the Wild: Empowering Language Models as Automatic Tool Agents Zhengliang Shi, Shen Gao, Lingyong Yan, et al. arXiv 2024. [Paper]
5. Enhancing Chat Language Models by Scaling High-quality Instructional Conversations Ning Ding, Yulin Chen, Bokai Xu, et al. EMNLP 2023. [Paper]

1 Data Processing for LLM

1.1 Data Acquisition

⬆️top

Public Data Sources

Open-source or copyright-free resources

1. Common Crawl: A large-scale publicly accessible web crawl dataset that provides massive raw webpages and metadata. It serves as a crucial raw data source in typical pretraining data pipelines, where it undergoes multiple processing steps such as cleaning, deduplication, and formatting to produce high-quality corpora for downstream model training. [Source]
2. CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, et al. LREC-COLING 2024. [Paper]
3. Project Gutenberg: A large collection of free eBooks from the public domain; supports training language models on long-form literary text. [Source]
4. Open Library: A global catalog of books with metadata and some open-access content; useful for multilingual and knowledge-enhanced language modeling. [Source]
5. GitHub: The world’s largest open-source code hosting platform; supports training models for code generation and understanding. [Source]
6. GitLab: A DevOps platform for hosting both private and open-source projects; provides high-quality programming and documentation data. [Source]
7. Bitbucket: A source code hosting platform by Atlassian; suitable for mining enterprise-level software development data. [Source]
8. The Stack: 3 TB of permissively licensed source code Denis Kocetkov, Raymond Li, Loubna Ben Allal, et al. arXiv 2022. [Paper]
9. CodeSearchNet Challenge: Evaluating the State of Semantic Code Search Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, et al. arXiv 2019. [Paper]

Website Crawling

1. An Empirical Comparison of Web Content Extraction Algorithms Janek Bevendorff, Sanket Gupta, Johannes Kiesel, Benno Stein. SIGIR 2023. [Paper]
2. Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction Adrien Barbaresi. ACL 2021 Demo. [Paper]
3. Fact or Fiction: Content Classification for Digital Libraries Aidan Finn, N. Kushmerick, Barry Smyth. DELOS Workshops / Conferences 2001. [Paper]

Website Crawling Tools

1. Beautiful Soup: A Python-based library for parsing HTML and XML documents; supports extracting structured information from static web pages. [Source]
2. Selenium: A browser automation tool that enables interaction with dynamic web pages; suitable for scraping JavaScript-heavy content. [Github]
3. Playwright: A browser automation framework developed by Microsoft; supports multi-browser environments and is ideal for high-quality, concurrent web scraping tasks. [Source]
4. Puppeteer: A Node.js library that provides a high-level API to control headless Chrome or Chromium; useful for scraping complex pages, taking screenshots, or generating PDFs. [Source]

Layout Analysis Pipelines

Extract text from hand-written or non-textual data (e.g., scanned PDF documents)

Pipelines

Extract text using multiple components as a pipeline, including segmentation and OCR

1. PaddleOCR: An open-source Optical Character Recognition (OCR) toolkit based on the PaddlePaddle deep learning framework; supports multilingual text detection and recognition, ideal for extracting text from images and document layout analysis. [Github]
2. MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsing Junbo Niu, Zheng Liu, Zhuangcheng Gu, et al. arXiv 2025. [Paper]
3. MinerU: An Open-Source Solution for Precise Document Content Extraction Bin Wang, Chao Xu, Xiaomeng Zhao, et al. arXiv 2024. [Paper]

End-to-End Models

Extract text using multi-modal LLMs as a whole, from image to text directly

1. DeepSeek-OCR: Contexts Optical Compression Haoran Wei, Yaofeng Sun, Yukun Li. arXiv 2025. [Paper]
2. dots.ocr: Multilingual Document Layout Parsing in a Single Vision-Language Model Yumeng Li, Guang Yang, Hao Liu, Bowen Wang, Colin Zhang. arXiv 2025. [Paper]
3. General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model Haoran Wei, Chenglong Liu, Jinyue Chen, et al. arXiv 2024. [Paper]
4. Focus Anywhere for Fine-grained Multi-page Document Understanding Chenglong Liu, Haoran Wei, Jinyue Chen, et al. arXiv 2024. [Paper]

Entity Recognition & Linking

Identify and link entities for the extracted text

1. UMIE: Unified Multimodal Information Extraction with Instruction Tuning Lin Sun, Kai Zhang, Qingyuan Li, Renze Lou. AAAI 2024. [Paper]
2. ChatEL: Entity Linking with Chatbots Yifan Ding, Qingkai Zeng, Tim Weninger. LREC-COLING 2024. [Paper]
3. WebIE: Faithful and Robust Information Extraction on the Web Chenxi Whitehouse, Clara Vania, Alham Fikri Aji, et al. ACL 2023. [Paper]
4. Alignment-Augmented Consistent Translation for Multilingual Open Information Extraction Keshav Kolluru, Muqeeth Mohammed, Shubham Mittal, et al. ACL 2022. [Paper]

1.2 Data Deduplication

⬆️top

1. Analysis of the Reasoning with Redundant Information Provided Ability of Large Language Models Wenbei Xie. arXiv 2023. [Paper]
2. Scaling Laws and Interpretability of Learning from Repeated Data Danny Hernandez, Tom Brown, Tom Conerly, et al. arXiv 2022. [Paper]

Exact Substring Matching

Identify samples with identical patterns, such as hashes (MD5/SHA256) or substring.

Hashing

Identify samples with the same MD5/SHA256 hashes.

1. MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens Anas Awadalla, Le Xue, Oscar Lo, et al. NeurIPS 2024. [Paper]
2. BaichuanSEED: Sharing the Potential of ExtensivE Data Collection and Deduplication by Introducing a Competitive Large Language Model Baseline Guosheng Dong, Da Pan, Yiding Sun, et al. arXiv 2024. [Paper]
3. The llama 3 herd of models Llama Team, AI @ Meta. arXiv 2024. [Paper]

Suffix Array

1. The RefinedWeb dataset for falcon LLM: outperforming curated corpora with web data only Guilherme Penedo, Quentin Malartic, Daniel Hesslow, et al. NeurIPS 2023. [Paper]
2. Deduplicating Training Data Makes Language Models Better Katherine Lee, Daphne Ippolito, Andrew Nystrom, et al. ACL 2022. [Paper]

Bloom Filter

1. Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research Luca Soldaini, Rodney Kinney, Akshita Bhagia, et al. ACL 2024. [Paper]
2. DataComp: In search of the next generation of multimodal datasets Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, et al. NeurIPS 2023. [Paper]

Approximate Hash Matching

Identify samples with similar patterns. Algorithms include SimHash, MinHash and DotHash.

1. LSHBloom: Memory-efficient, Extreme-scale Document Deduplication Arham Khan, Robert Underwood, Carlo Siebenschuh, et al. arXiv 2025. [Paper]
2. SlimPajama-DC: Understanding Data Combinations for LLM Training Zhiqiang Shen, Tianhua Tao, Liqun Ma, et al. arXiv 2023. [Paper]
3. Noise-Robust De-Duplication at Scale Emily Silcock, Luca D'Amico-Wong, Jinglin Yang, Melissa Dell. arXiv 2022. [Paper]

Sample-Level:

1. The llama 3 herd of models [Paper]
2. Deduplicating Training Data Makes Language Models Better [Paper]
3. The RefinedWeb dataset for falcon LLM: outperforming curated corpora with web data only Guilherme Penedo, Quentin Malartic, Daniel Hesslow, et al. NeurIPS 2023. [Paper]

Line-Level:

1. BaichuanSEED: Sharing the Potential of ExtensivE Data Collection and Deduplication by Introducing a Competitive Large Language Model Baseline [Paper]

Code Data:

1. BaichuanSEED: Sharing the Potential of ExtensivE Data Collection and Deduplication by Introducing a Competitive Large Language Model Baseline [Paper]
2. DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence Daya Guo, Qihao Zhu, Dejian Yang, et al. arXiv 2024. [Paper]

Approximate Embedding Clustering

Identify samples with similar semantics.

1. FairDeDup: Detecting and Mitigating Vision-Language Fairness Disparities in Semantic Dataset Deduplication Eric Slyman, Stefan Lee, Scott Cohen, Kushal Kafle. CVPR 2024. [Paper]
2. SemDeDup: Data-efficient learning at web-scale through semantic deduplication Amro Abbas, Kushal Tirumala, Dániel Simig, et al. ICLR 2023. [Paper]
3. D4: Improving LLM Pretraining via Document De-Duplication and Diversification Kushal Tirumala, Daniel Simig, Armen Aghajanyan, Ari Morcos. NeurIPS 2023. [Paper]

Frequency-based Weighted Resampling

Identify and down-sample samples with high frequency:

1. SoftDedup: an Efficient Data Reweighting Method for Speeding Up Language Model Pre-training Nan He, Weichen Xiong, Hanwen Liu, et al. ACL 2024. [Paper]

Identify and up-sample samples with the removal rate after the entire filtering stage:

1. DataComp: In search of the next generation of multimodal datasets Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, et al. NeurIPS 2023. [Paper]

1.3 Data Filtering

⬆️top

Sample-level Filtering

(1) Statistical Evaluation

Filter samples based on statistical metrics (e.g., cosine similarity) or model characteristics (e.g., loss, perplexity).

Perplexity Measuring:

1. Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, et al. ICLR 2025. [Paper]
2. Smaller Language Models are capable of selecting Instruction-Tuning Training Data for Larger Language Models Dheeraj Mekala, Alex Nguyen, Jingbo Shang. ACL 2024. [Paper]
3. From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning Ming Li, Yong Zhang, Zhitao Li, et al. NAACL 2024. [Paper]
4. Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning Ming Li, Yong Zhang, Shwai He, et al. ACL 2024. [Paper]
5. Improving Pretraining Data Using Perplexity Correlations Tristan Thrush, Christopher Potts, Tatsunori Hashimoto. arXiv 2024. [Paper]

Influence Assessment: Assess sample impact on LLM performance or learning process (e.g., model parameters)

1. Data-efficient Fine-tuning for LLM-based Recommendation Xinyu Lin, Wenjie Wang, Yongqi Li, et al. SIGIR 2024. [Paper]
2. SHED: Shapley-Based Automated Dataset Refinement for Instruction Fine-Tuning Yexiao He, Ziyao Wang, Zheyu Shen, et al. NeurIPS 2024. [Paper]

Clustering: Cluster semantically similar samples - select within the clusters reduces redundancy, while select across the clusters increases diversity

1. SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Models Yu Yang, Siddhartha Mishra, Jeffrey Chiang, et al. NeurIPS 2024. [Paper]
2. Effective Pruning of Web-Scale Datasets Based on Complexity of Concept Clusters Amro Abbas, Evgenia Rusak, Kushal Tirumala, et al. arXiv 2024. [Paper]

(2) Model Scoring

Filter samples based on sample quality evaluated by LLM.

1. SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selection Han Shen, Pin-Yu Chen, Payel Das, Tianyi Chen. ICLR 2025. [Paper]
2. SCAR: Data Selection via Style-Consistency-Aware Response Ranking for Efficient Instruction Tuning of Large Language Models Zhuang Li, Yuncheng Hua, Thuy-Trang Vu, et al. ACL 2025. [Paper]
3. QuRating: Selecting High-Quality Data for Training Language Models Alexander Wettig, Aatmik Gupta, Saumya Malik, Danqi Chen. ICML 2024. [Paper]
4. What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning Wei Liu, Weihao Zeng, Keqing He, et al. ICLR 2024. [Paper]

(3) Hybrid Methods

Filter samples using multiple types of methods.

1. When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale Max Marion, Ahmet Üstün, Luiza Pozzobon, et al. arXiv 2023. [Paper]
2. Instruction Mining: Instruction Data Selection for Tuning Large Language Models Yihan Cao, Yanbin Kang, Chi Wang, Lichao Sun. arXiv 2023. [Paper]
3. MoDS: Model-oriented Data Selection for Instruction Tuning Qianlong Du, Chengqing Zong, Jiajun Zhang. arXiv 2023. [Paper]

Content-level Filtering

Filter samples with specific information, including

Meaningless Text (e.g., special characters and unnecessary texts)

1. BaichuanSEED: Sharing the Potential of ExtensivE Data Collection and Deduplication by Introducing a Competitive Large Language Model Baseline [Paper]
2. The llama 3 herd of models [Paper]
3. Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research [Paper]
4. When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale Max Marion, Ahmet Üstün, Luiza Pozzobon, et al. arXiv 2023. [Paper]

Language Filtering (using fastText, langdetect, or CLD2)

1. Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research [Paper]
2. The RefinedWeb dataset for falcon LLM: outperforming curated corpora with web data only Guilherme Penedo, Quentin Malartic, Daniel Hesslow, et al. NeurIPS 2023. [Paper]
3. Exploring the limits of transfer learning with a unified text-to-text transformer Colin Raffel, Noam Shazeer, Adam Roberts, et al. arXiv 2023. [Paper]
4. The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao, Stella Biderman, Sid Black, et al. arXiv 2021. [Paper]
5. CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data Guillaume Wenzek, Marie - Anne Lachaux, Alexis Conneau, et al. LREC 2020. [Paper]

Toxic Information (e.g., spam, gambling, and pornographic content)

1. BaichuanSEED: Sharing the Potential of ExtensivE Data Collection and Deduplication by Introducing a Competitive Large Language Model Baseline [Paper]
2. Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research [Paper]
3. The RefinedWeb dataset for falcon LLM: outperforming curated corpora with web data only Guilherme Penedo, Quentin Malartic, Daniel Hesslow, et al. NeurIPS 2023. [Paper]
4. Llama 2: Open Foundation and Fine-Tuned Chat Models GenAI, Meta. arXiv 2023. [Paper]

Sensitive information (e.g., personally identifiable information)

1. DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4 Zhengliang Liu, Yue Huang, Xiaowei Yu, et al. arXiv 2023. [Paper]
2. Analyzing Leakage of Personally Identifiable Information in Language Models Nils Lukas, Ahmed Salem, Robert Sim, et al. IEEE S&P 2023. [Paper]

1.4 Data Domain Selection

⬆️top

1. A Survey on Data Selection for Language Models Alon Albalak, Yanai Elazar, Sang Michael Xie, et al. arXiv 2024. [Paper]

2. A Survey on Data Selection for LLM Instruction Tuning Jiahao Wang, Bolin Zhang, Qianlong Du, et al. arXiv 2024. [Paper]

Similarity-based Data Selection

Select subset of data with similar characteristics as the target data

Cosine Similarity

1. Efficient Continual Pre-training for Building Domain Specific Large Language Models Yong Xie, Karan Aggarwal, Aitzaz Ahmad. Findings of ACL 2024. [Paper]

Bag-of-Words Similarity

1. Data Selection for Language Models via Importance Resampling Sang Michael Xie, Shibani Santurkar, Tengyu Ma, Percy Liang. NeurIPS 2023. [Paper]

Lexicon Set Overlap

1. Enabling On-Device Large Language Model Personalization with Self-Supervised Data Selection and Synthesis Ruiyang Qin, Jun Xia, Zhenge Jia, et al. DAC 2024. [Paper]

Bayes-based Selection

1. CoLoR-Filter: Conditional Loss Reduction Filtering for Targeted Language Model Pre-training David Brandfonbrener, Hanlin Zhang, Andreas Kirsch, et al. NeurIPS 2024. [Paper]

Optimization-based Data Selection

Select subset of data towards improving model performance on target tasks

Linear Search (Datamodel)

1. DSDM: model-aware dataset selection with datamodels Logan Engstrom, Axel Feldmann, Aleksander Mądry. ICML 2024. [Paper]
2. LESS: Selecting Influential Data for Targeted Instruction Tuning Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, et al. ICML 2024. [Paper]
3. TSDS: Data Selection for Task-Specific Model Finetuning Zifan Liu, Amin Karbasi, Theodoros Rekatsinas. arXiv 2024. [Paper]

Model-based Data Selection

Select subset of data with the help of LLM

1. Autonomous Data Selection with Language Models for Mathematical Texts Yifan Zhang, Yifan Luo, Yang Yuan, et al. ICLR 2024. [Paper]

1.5 Data Mixing

⬆️top

1. Mixtera: A Data Plane for Foundation Model Training Maximilian Böther, Xiaozhe Yao, Tolga Kerimoglu, Dan Graur, Viktor Gsteiger, Ana Klimovic. arXiv 2025. [Paper]
2. Scalable Data Ablation Approximations for Language Models through Modular Training and Merging Clara Na, Ian Magnusson, Ananya Harsh Jha, et al. EMNLP 2024. [Paper]
3. Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Models Wenfeng Feng, Chuzhan Hao, Yuewei Zhang, et al. COLING 2024. [Paper]

Before-Training Mixing (Human Experience)

Empirical set mixing ratio (e.g., complexity and diversity)

1. Exploring the limits of transfer learning with a unified text-to-text transformer [Paper]
2. Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretraining Steven Feng, Shrimai Prabhumoye, Kezhi Kong, et al. arXiv 2024. [Paper]
3. SlimPajama-DC: Understanding Data Combinations for LLM Training SlimPajama-DC: Understanding Data Combinations for LLM Training, arXiv 2024. [Paper]
4. BiMix: Bivariate Data Mixing Law for Language Model Pretraining Ce Ge, Zhijian Ma, Daoyuan Chen, et al. arXiv 2024. [Paper]

Before-Training Mixing (Model-Based Optimization)

Logically decide the best mixing ratio based on other factors using models

1. RegMix: Data Mixture as Regression for Language Model Pre-training Qian Liu, Xiaosen Zheng, Niklas Muennighoff, et al. ICLR 2025. [Paper]
2. Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance Jiasheng Ye, Peiju Liu, Tianxiang Sun, et al. ICLR 2025. [Paper]
3. CMR Scaling Law: Predicting Critical Mixture Ratios for Continual Pre-training of Language Models Jiawei Gu, Zacc Yang, Chuanghao Ding, et al. EMNLP 2024. [Paper]
4. BiMix: Bivariate Data Mixing Law for Language Model Pretraining Ce Ge, Zhijian Ma, Daoyuan Chen, et al. arXiv 2024. [Paper]
5. D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models Haoran Que, Jiaheng Liu, Ge Zhang, et al. arXiv 2024. [Paper]
6. Data Proportion Detection for Optimized Data Management for Large Language Models Hao Liang, Keshi Zhao, Yajie Yang, et al. arXiv 2024. [Paper]

During-Training Mixing (Bilevel Optimization)

Find the best mixing ratio that minimizes both training and validation loss

1. ScaleBiO: Scalable Bilevel Optimization for LLM Data Reweighting Rui Pan, Jipeng Zhang, Xingyuan Pan, et al. ACL 2025. [Paper]
2. DoGE: Domain Reweighting with Generalization Estimation Simin Fan, Matteo Pagliardini, Martin Jaggi. ICML 2024. [Paper]

During-Training Mixing (Distributionally Robust Optimization).

Find the best mixing ratio with Distributionally Robust Optimization (DRO)

1. Task-level Distributionally Robust Optimization for Large Language Model-based Dense Retrieval Guangyuan Ma, Yongliang Ma, Xing Wu, et al. AAAI 2025. [Paper]
2. DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining Sang Michael Xie, Hieu Pham, Xuanyi Dong, et al. NeurIPS 2023. [Paper]

1.6 Data Distillation and Synthesis

⬆️top

1. How to Synthesize Text Data without Model Collapse? Xuekai Zhu, Daixuan Cheng, Hengli Li, et al. ICML 2025. [Paper]
2. Differentially Private Synthetic Data via Foundation Model APIs 2: Text Chulin Xie, Zinan Lin, Arturs Backurs, et al. ICML 2024. [Paper]
3. WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions Can Xu, Qingfeng Sun, Kai Zheng, et al. ICLR 2024. [Paper]
4. LLM See, LLM Do: Leveraging Active Inheritance to Target Non-Differentiable Objectives Luísa Shimabucoro, Sebastian Ruder, Julia Kreutzer, et al. EMNLP 2024. [Paper]
5. Augmenting Math Word Problems via Iterative Question Composing Haoxiong Liu, Yifan Zhang, Yifan Luo, et al. arXiv 2024. [Paper]

Pre-training Data Augmentation

Augment raw pre-training text into various formats to scale data

Rephrasing

Rephrasing raw corpora into various styles

1. Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling Pratyush Maini, Skyler Seto, Richard Bai, et al. ACL 2024. [Paper]

Instruction Augmentation

Convert raw corpora into instructions

1. Instruction Pre-Training: Language Models are Supervised Multitask Learners Daixuan Cheng, Yuxian Gu, Shaohan Huang, et al. EMNLP 2024. [Paper]

Image Caption Augmentation

1. VeCLIP: Improving CLIP Training via Visual-Enriched Captions Zhengfeng Lai, Haotian Zhang, Bowen Zhang, et al. ECCV 2024. [Paper]
2. Improving CLIP Training with Language Rewrites Lijie Fan, Dilip Krishnan, Phillip Isola, et al. NeurIPS 2023. [Paper]

SFT Task Data Augmentation

Augment fine-tuning text for specific use cases (e.g., domain augmentation, alignment, agentic LLM, etc.)

Domain Data Synthesis

1. Structural Entropy Guided Agent for Detecting and Repairing Knowledge Deficiencies in LLMs Yifan Wei, Xiaoyan Yu, Tengfei Pan, et al. NeurIPS 2025. [Paper]
2. Multistage Collaborative Knowledge Distillation from a Large Language Model for Semi-Supervised Sequence Generation Jiachen Zhao, Wenlong Zhao, Andrew Drozdov, et al. ACL 2024. [Paper]
3. Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models Haoran Li, Qingxiu Dong, Zhengyang Tang, et al. arXiv 2024. [Paper]

Instruction Data Synthesis

1. AgentInstruct: Toward Generative Teaching with Agentic Flows Arindam Mitra, Luciano Del Corro, Guoqing Zheng, et al. arXiv 2024. [Paper]
2. Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing Zhangchen Xu, Fengqing Jiang, Luyao Niu, et al. arXiv 2024. [Paper]
3. Self-Instruct: Aligning Language Models with Self-Generated Instructions Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, et al. ACL 2023. [Paper]

Agentic Data Synthesis

1. Kimi K2: Open Agentic Intelligence Kimi Team. arXiv 2025. [Paper]

Privacy-Preserving Distillation

1. PDSS: A Privacy-Preserving Framework for Step-by-Step Distillation of Large Language Models Tao Fan, Weijing Chen, Yan Kang, et al. arXiv 2025. [Paper]

Reasoning Data Augmentation

Synthesize reasoning steps using, e.g., CoT.

1. LIMO: Less is More for Reasoning Yixin Ye, Zhen Huang, Yang Xiao, et al. COLM 2025. [Paper]
2. Distilling System 2 into System 1 Ping Yu, Jing Xu, Jason Weston, Ilia Kulikov. arXiv 2024 [Paper]

Reasoning Ability Distillation

1. Distilling Reasoning Capabilities into Smaller Language Models Kumar Shridhar, Alessandro Stolfo, Mrinmaya Sachan. ACL 2024. [Paper]
2. Symbolic chain-of-thought distillation: Small models can also "think" step-by-step Liunian Harold Li, Jack Hessel, Youngjae Yu, et al. arXiv 2024. [Paper]
3. Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, et al. ACL 2023. [Paper]

Reasoning Step Augmentation

1. Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling Zhenyu Hou, Xin Lv, Rui Lu, et al. arXiv 2025. [Paper]
2. Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations Peiyi Wang, Lei Li, Zhihong Shao, et al. ACL 2024. [Paper]
3. Let's Verify Step by Step Hunter Lightman, Vineet Kosaraju, Yura Burda, et al. arXiv 2023. [Paper]

Reinforcement Learning

1.Chunks as Arms: Multi-Armed Bandit-Guided Sampling for Long-Context LLM Preference Optimization Shaohua Duan, Pengcheng Huang, Xinze Li, et al. ACL 2026. [Paper] 2. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, et al. NeurIPS 2023. [Paper] 3. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback Yuntao Bai, Andy Jones, Kamal Ndousse, et al. arXiv 2022. [Paper]

Retrieval-Augmentation Generation

1. LLMs Can Easily Learn to Reason from Demonstrations: Structure, Not Content, Is What Matters! Dacheng Li, Shiyi Cao, Tyler Griggs, et al. arXiv 2025. [Paper]
2. Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search Maohao Shen, Guangtao Zeng, Zhenting Qi, et al. arXiv 2025. [Paper]
3. MUSTARD: Mastering Uniform Synthesis of Theorem and Proof Data Yinya Huang, Xiaohan Lin, Zhengying Liu, et al. ICLR 2024. [Paper]
4. PaD: Program-aided Distillation Can Teach Small Models Reasoning Better than Chain-of-thought Fine-tuning Xuekai Zhu, Biqing Qi, Kaiyan Zhang, et al. NAACL 2024. [Paper]
5. Mitigating the Privacy Issues in Retrieval-Augmented Generation (RAG) via Pure Synthetic Data Shenglai Zeng, Jiankun Zhang, Pengfei He, et al. arXiv 2024. [Paper]
6. NuminaMath: The largest public dataset in AI4Maths with 860k pairs of competition math problems and solutions Jia Li, Edward Beeching, Lewis Tunstall, et al. 2024. [Paper]
7. QwQ: Reflect Deeply on the Boundaries of the Unknown Qwen Team. 2024. [Source]
8. Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Data Anup Shirgaonkar, Nikhil Pandey, Nazmiye Ceren Abay, et al. arXiv 2024. [Paper]
9. Knowledge-augmented reasoning distillation for small language models in knowledge-intensive tasks Minki Kang, Seanie Lee, Jinheon Baek, et al. NeurIPS 2023. [Paper]
10. Dialogue chain-of-thought distillation for commonsense-aware conversational agents Hyungjoo Chae, Yongho Song, Kai Tzu-iunn Ong, et al. arXiv 2023. [Paper]
11. MCC-KD: Multi-CoT consistent knowledge distillation Hongzhan Chen, Siyue Wu, Xiaojun Quan, et al. arXiv 2023. [Paper]
12. Large language models are reasoning teachers Namgyu Ho, Laura Schmid, Se-Young Yun. arXiv 2023. [Paper]
13. Leveraging training data in few-shot prompting for numerical reasoning Zhanming Jie, Wei Lu. arXiv 2023. [Paper]
14. Distilling reasoning capabilities into smaller language models Kumar Shridhar, Alessandro Stolfo, Mrinmaya Sachan. arXiv 2023. [Paper]
15. SCOTT: Self-consistent chain-of-thought distillation Peifeng Wang, Zhengyang Wang, Zheng Li, et al. arXiv 2023. [Paper]
16. Democratizing reasoning ability: Tailored learning from large language model Zhaoyang Wang, Shaohan Huang, Yuxuan Liu, et al. arXiv 2023. [Paper]
17. Explanations from large language models make small reasoners better Shiyang Li, Jianshu Chen, Yelong Shen, et al. arXiv 2022. [Paper]
18. Training Verifiers to Solve Math Word Problems Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, et al. arXiv 2021. [Paper]

Pre-training Data Augmentation

1. Llama 2: Open Foundation and Fine-Tuned Chat Models [Paper]
2. The Pile: An 800GB Dataset of Diverse Text for Language Modeling [Paper]
3. Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research [Paper]
4. BERT-Tiny-Chinese: A lightweight Chinese BERT pre-trained model released by CKIP Lab, with a small number of parameters; suitable for use as an encoder in pre-training data augmentation tasks to enhance efficiency for compact models. [Source]
5. Case2Code: Scalable Synthetic Data for Code Generation Yunfan Shao, Linyang Li, Yichuan Ma, et al. COLING 2025. [Paper]
6. Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages Zui Chen, Tianqiao Liu, Mi Tian, et al. ICLR 2025. [Paper]
7. Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks Bin Xiao, Haiping Wu, Weijian Xu, et al. CVPR 2024. [Paper]
8. DiffuseMix: Label-Preserving Data Augmentation with Diffusion Models Khawar Islam, Muhammad Zaigham Zaheer, Arif Mahmood, et al. CVPR 2024. [Paper]
9. Magicoder: Empowering Code Generation with OSS-Instruct Yuxiang Wei, Zhe Wang, Jiawei Liu, et al. ICML 2024. [Paper]
10. JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Models Kun Zhou, Beichen Zhang, Jiapeng Wang, et al. arXiv 2024. [Paper]
11. Diffusion Models and Representation Learning: A Survey Michael Fuest, Pingchuan Ma, Ming Gui, et al. arXiv 2024. [Paper]
12. CtrlSynth: Controllable Image Text Synthesis for Data-Efficient Multimodal Learning Qingqing Cao, Mahyar Najibi, Sachin Mehta. arXiv 2024. [Paper]
13. Qwen2 Technical Report An Yang, Baosong Yang, Binyuan Hui, et al. arXiv 2024. [Paper]
14. TinyLlama: An Open-Source Small Language Model Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, Wei Lu. arXiv 2024. [Paper]
15. On the Diversity of Synthetic Data and its Impact on Training Large Language Models Hao Chen, Abdul Waheed, Xiang Li, et al. arXiv 2024. [Paper]
16. Towards Effective and Efficient Continual Pre-training of Large Language Models Jie Chen, Zhipeng Chen, Jiapeng Wang, et al. arXiv 2024. [Paper]
17. Effective Data Augmentation With Diffusion Models Brandon Trabucco, Kyle Doherty, Max Gurinas, et al. arXiv 2023. [Paper]
18. Mistral 7B Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, et al. arXiv 2023. [Paper]
19. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis Dustin Podell, Zion English, Kyle Lacey, et al. arXiv 2023. [Paper]
20. Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus Jesse Dodge, Maarten Sap, Ana Marasović, et al. EMNLP 2021. [Paper]
21. First Steps of an Approach to the ARC Challenge based on Descriptive Grid Models and the Minimum Description Length Principle Sébastien Ferré (Univ Rennes, CNRS, IRISA). arXiv 2021. [Paper]
22. TinyBERT: Distilling BERT for Natural Language Understanding Xiaoqi Jiao, Yichun Yin, Lifeng Shang, et al. Findings of EMNLP 2020. [Paper]
23. HellaSwag: Can a Machine Really Finish Your Sentence? Rowan Zellers, Ari Holtzman, Yonatan Bisk, et al. ACL 2019. [Paper]

SFT Data Augmentation

1. Augmenting Math Word Problems via Iterative Question Composing [Paper]
2. Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoning Yiming Huang, Xiao Liu, Yeyun Gong, et al. arXiv 2024. [Paper]

1.7 End-to-End Data Processing Pipelines

⬆️top

1.7.1 Typical data processing frameworks

1. Data-Juicer: A One-Stop Data Processing System for Large Language Models Daoyuan Chen, Yilun Huang, Zhijian Ma, et al. SIGMOD 2024. [Paper]
2. An Integrated Data Processing Framework for Pretraining Foundation Models Yiding Sun, Feng Wang, Yutao Zhu, et al. SIGIR 2024. [Paper]
3. Dataverse: Open-Source ETL (Extract, Transform, Load) Pipeline for Large Language Models Hyunbyung Park, Sukyung Lee, Gyoungjin Gim, et al. arXiv 2024. [Paper]

1.7.2 Typical data pipelines

1. Exploring the limits of transfer learning with a unified text-to-text transformer [Paper]
2. The RefinedWeb dataset for falcon LLM: outperforming curated corpora with web data only Guilherme Penedo, Quentin Malartic, Daniel Hesslow, et al. NeurIPS 2023. [Paper]
3. Scaling Language Models: Methods, Analysis & Insights from Training Gopher Jack W. Rae, Sebastian Borgeaud, Trevor Cai, et al. arXiv 2021. [Paper]
4. CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data Guillaume Wenzek, Marie - Anne Lachaux, Alexis Conneau, et al. LREC 2020. [Paper]
5. Bag of Tricks for Efficient Text Classification Armand Joulin, Edouard Grave, Piotr Bojanowski, Tomas Mikolov. arXiv 2016. [Paper]

1.7.3 Orchestration of data pipelines

1. Data-Juicer Sandbox: A Feedback-Driven Suite for Multimodal Data-Model Co-development Daoyuan Chen, Haibin Wang, Yilun Huang, et al. ICML 2025 (Spotlight). [Paper]

2 Data Storage for LLM

2.1 Data Formats

⬆️top

Training Data Format

1. TFRecord: A binary data storage format recommended by TensorFlow, suitable for efficient storage and reading of large-scale training data. [Source]
2. MindRecord: An efficient data storage format used by MindSpore, supporting multi-platform data management. [Source]
3. tf.data.Dataset: An abstract interface in TensorFlow representing collections of training data, enabling flexible data manipulation. [Source]
4. COCO JSON: COCO JSON format uses structured J