登录

data-prep-kit

data-prep-kitApache-2.0活跃维护

面向 GenAI 应用的开源数据准备工具包，提供去重、数据清洗与预处理流水线，帮助把原始数据整理成可用于微调大模型的高质量数据集。

Open source project for data preparation for GenAI applications

v1.1.8发布于 2026-06-22 16:30 UTC968star257fork同步于 2026-11-07 07:29 UTC
数据准备数据清洗去重预处理流水线数据筛选
对比 GitHub 官网code-qualitydatadata-prepdata-preparationdata-preprocessing

怎么上手

由 README 提炼，执行前请确认
原文 
pip install uv
uv pip install 'data-prep-toolkit-transforms[all]'
硬件要求按 README 说明

README 未说明具体显存或 GPU 型号要求，只说明可从普通笔记本扩展到数据中心规模；Ray 运行时用于横向扩展。

下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼
没人工点评？写下你的经验

面向 GenAI 的开源数据准备工具，把非结构化数据清洗、转换、增强成可训练或可 RAG 的数据

面向 GenAI 应用的开源数据准备工具包，提供去重、数据清洗与预处理流水线，帮助把原始数据整理成可用于微调大模型的高质量数据集。

核心能力

  • 提供去重、过滤、分词、PII 脱敏等数十种可组合 transform
  • 同时支持 Python 与 Ray 两种运行时，可从笔记本扩展到数据中心
  • 覆盖自然语言、代码、图像三种数据模态
  • 内置 Parquet、ZIP、NDJSON、JSONL 格式处理框架，可写自定义 transform
  • 支持在 Kubernetes 上以 Python 或 Ray 作业部署，并用 Tekton 串联多步流水线

容易踩的坑

需要 Python 3.10~3.13，版本不符会装不上；[all] 会安装全部 transform，依赖较重，可按需只装子集；部分 transform 需要额外下载模型或数据，README 提示可从 HuggingFace 取真实数据测试。

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

最近发版

5 个版本v1.1.8
全部 release
  1. v1.1.8Release 1.1.8最新正式版2026-06-22 · 2026-06-22 16:30 UTC

    What's Changed

    Full Changelog: https://github.com/data-prep-kit/data-prep-kit/compare/v1.1.7...v1.1.8

  2. v1.1.72026-02-11 · 2026-02-11 15:51 UTC
    查看更新说明

    What's Changed

    Full Changelog: https://github.com/data-prep-kit/data-prep-kit/compare/v1.1.6...v1.1.

  3. v1.1.6v.1.1.62025-11-14 · 2025-11-14 16:11 UTC
    查看更新说明

    What's Changed

  4. v1.1.52025-10-02 · 2025-10-02 15:20 UTC
    查看更新说明

    What's Changed

    New Contributors

    Full Changelog: https://github.com/data-prep-kit/data-prep-kit/compare/v1.1.4...v1.1.5

  5. v1.1.42025-09-16 · 2025-09-16 01:39 UTC
    查看更新说明

    What's Changed

    Full Changelog: https://github.com/data-prep-kit/data-prep-kit/compare/v1.1.3...v1.1.4

用户点评

登录用户的真实使用记录，可按版本和硬件筛选
共 0 条

    还没有人点评。第一个说出真实体验的人最有价值。

    README 原文

    未翻译
    在 GitHub 打开

    arXiv Docs PyPI version pre-commit Apache 2.0 GitHub Issues GitHub Pull Requests LF AI & Data OpenSSF Best Practices

    Data-Prep-Kit accelerates unstructured data preparation for LLM app developers. Developers can use Data-Prep-Kit to cleanse, transform, and enrich use case-specific unstructured data to pre-train LLMs, fine-tune LLMs, instruct-tune LLMs, or build Retrieval Augmented Generation (RAG) applications for LLMs

    Data-Prep-Kit can readily scale from a commodity laptop all the way to data center scale.

    Features

    • The kit provides a growing set of modules/transforms targeting laptop-scale to datacenter-scale processing.
    • The data modalities supported today are: Natural Language, Code, and Image.
    • The modules are built on common frameworks for Python and Ray runtimes for scaling up data processing.
    • The kit provides a framework for developing custom transforms for processing Parquet files as well as ZIP, NDJSON, and JSONL file formats.
    • The kit provides examples of how a single transform can be deployed on Kubernetes clusters as a Python or a Ray job. Additionally, when multiple transforms are deployed in a sequence, the kit uses Tekton pipelines.

    Installation

    The latest version of the Data-Prep-Kit is available on PyPi for Python 3.10, 3.11, 3.12, and 3.13. It can be installed using:

    pip install uv
uv pip install 'data-prep-toolkit-transforms[all]'

    This will install all available transforms.

    For guidance on creating the virtual environment for installing the data prep kit, click here.

    🚀 Getting Started

    Fastest way to experience Data-Prep-Kit

    With no setup necessary, let's use a Google Colab friendly notebook to try Data-Prep-Kit. This is a simple transform to extract content from PDF files: examples/notebooks/Run_your_first_transform_colab.ipynb | Open In Colab. (Here are some tips for running Data-Prep-Kit transforms on Google Colab. For this simple example, these tips are either already taken care of, or are not needed.) The same notebook can be downloaded and run on the local machine, without cloning the repo or any other setup.

    Examples

    Now that you have run a single transform, the next step is to explore how to put these transforms together to run a data prep pipeline for end to end real enterprise use cases like fine-tuning a model or building a RAG application.

    We have a complete set of data processing recipes for such use cases.

    We also have a developer tutorial for contributing a new transform to the kit.

    For advanced users, here is more information for adding your own transform, running transforms from the command line, scaling and automation and more. Also, repository structure and use are discussed here.

    Using HuggingFace data files

    All the transforms in the kit include small sample data files for testing, but advanced users who want to download real data files from HuggingFace and use them in testing, can refer to this.

    Supported data transforms

    The matrix below shows the the combination of modules and supported runtimes. All the modules can be accessed here and can be combined to form data processing pipelines, as shown in the examples folder.

    ModulesPython-onlyRay
    Data Ingestion
    Code (from zip) to Parquet:white_check_mark::white_check_mark:
    Docling to Parquet:white_check_mark::white_check_mark:
    HTML to Parquet:white_check_mark::white_check_mark:
    Web to Parquet:white_check_mark:
    Universal (Code & Language)
    Exact dedup filter:white_check_mark::white_check_mark:
    Fuzzy dedup filter:white_check_mark::white_check_mark:
    Unique ID annotation:white_check_mark::white_check_mark:
    Filter on annotations:white_check_mark::white_check_mark:
    Profiler:white_check_mark::white_check_mark:
    Resize:white_check_mark::white_check_mark:
    Hate, Abuse, Profanity (HAP):white_check_mark::white_check_mark:
    Tokenizer:white_check_mark::white_check_mark:
    Tokenization2Arrow:white_check_mark::white_check_mark:
    Repetition removal:white_check_mark::white_check_mark:
    Bloom filter:white_check_mark:
    Collapse(column concatenation):white_check_mark::white_check_mark:
    Blocklist:white_check_mark::white_check_mark:
    C4 annotator:white_check_mark::white_check_mark:
    Fineweb quality annotator:white_check_mark::white_check_mark:
    Gopher repetition annotator:white_check_mark::white_check_mark:
    Opensearch:white_check_mark::white_check_mark:
    Folder2Parquet:white_check_mark:
    Language-only
    Language identification:white_check_mark::white_check_mark:
    Document quality:white_check_mark::white_check_mark:
    Document chunking for RAG:white_check_mark::white_check_mark:
    Text encoder:white_check_mark::white_check_mark:
    PII Annotator/Redactor:white_check_mark::white_check_mark:
    Similarity:white_check_mark:
    GneissWeb classification:white_check_mark::white_check_mark:
    Readability scores:white_check_mark::white_check_mark:
    Extreme tokenized annotation:white_check_mark::white_check_mark:
    ML Filter:white_check_mark::white_check_mark:
    ML Enrichment(quality annotation):white_check_mark::white_check_mark:
    Code-only
    Programming language annotation:white_check_mark::white_check_mark:
    Code quality annotation:white_check_mark::white_check_mark:
    Malware annotation:white_check_mark::white_check_mark:
    Header cleanser:white_check_mark::white_check_mark:
    Semantic file ordering:white_check_mark:
    License Select Annotation:white_check_mark::white_check_mark:
    Code profiler:white_check_mark::white_check_mark:
    Images
    Faces:white_check_mark::white_check_mark:
    NSFW(Not Safe For Work):white_check_mark::white_check_mark:
    People:white_check_mark::white_check_mark:

    Logging configuration

    DPK uses a unified logger - dpk. It can be configured, by setting the following environment variables

    Variable nameDefault valueDescription
    DPK_LOG_LEVELINFOThe loggger level
    DPK_LOG_FILENoneThe path to the log file, if set the log message will be stored in the file
    DPK_LOG_JSON_HANDLER""If set to any value of "true", "1", "yes", or "on" (case insensitive) the console logs will be in JSON format
    DPK_LOG_PROPAGATION""If set to any value of "true", "1", "yes", or "on" (case insensitive), the logger will propagate all log messages to its parent logger

    Contributing

    Contributors are welcome to add new modules to expand to other data modalities as well as add runtime support for existing modules! Please read this for details.

    Get help and support

    Please feel free to connect with us using the discussion section.

    MAINTAINERS

    For a list of current maintainers, please see.

    CHANGELOG

    For the history of releases and changes, please see.

    Resources

    Papers, talks, presentations and tutorials

    Granite open source LLM models

    GneissWeb

    Citation

    If you use Data-Prep-Kit in your research, please cite our paper:

    @misc{wood2024dataprepkitgettingdataready,
      title={Data-Prep-Kit: getting your data ready for LLM application development}, 
      author={David Wood and Boris Lublinsky and Alexy Roytman and Shivdeep Singh 
      and Constantin Adam and Abdulhamid Adebayo and Sungeun An and Yuan Chi Chang 
      and Xuan-Hong Dang and Nirmit Desai and Michele Dolfi and Hajar Emami-Gohari 
      and Revital Eres and Takuya Goto and Dhiraj Joshi and Yan Koyfman 
      and Mohammad Nassar and Hima Patel and Paramesvaran Selvam and Yousaf Shah  
      and Saptha Surendran and Daiki Tsuzuku and Petros Zerfos and Shahrokh Daijavad},
      year={2024},
      eprint={2409.18164},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2409.18164}, 
}

    License

    All source files must include a Copyright and License header. If you would like to see the detailed LICENSE click here.

    LF AI & Data

    Data-Prep-Kit is hosted as a project in the LF AI & Data Foundation.

    IBM ❤️ Open Source AI

    The project was started by the Data for AI Models team at IBM Research.

    Copyright © Data-Prep-Kit Framework - a Series of LF Projects, LLC.

    相关项目

    同分类里 star 最高的项目
    对比全部 →
    ruff
    astral-sh·Rust

    用 Rust 写成的超快 Python 代码检查与格式化工具，能替代 Flake8、Black 等，大幅缩短 lint 和格式化耗时。

    Python 代码检查+4代码质量
    51146+3842.6k0.17.42026-11-10 04:17 UTC
    open-code-review
    alibaba·Go

    阿里开源的大规模代码审查工具，用确定性流水线加 LLM Agent 混合架构，精准定位行级问题，内置 NPE、线程安全、XSS、SQL 注入等多语言规则集。

    代码审查+4代码质量
    49738+1513.7kv1.13.62026-11-10 03:53 UTC
    SwiftLint
    realm·Swift

    Swift 代码风格检查工具，按预设规则自动扫描并报告不符合规范的写法，帮团队统一 Swift 代码风格。

    Swift+4代码质量
    19783+42.3k0.65.12026-11-05 17:57 UTC
    diff-so-fancy
    so-fancy·Perl

    把 Git diff 输出重新排版成更易读的格式，突出改动位置和代码块，方便快速定位缺陷、提升代码审查效率。

    代码审查+3代码质量
    18112+4350v1.4.142026-10-18 18:29 UTC