kedro

kedro-org活跃维护

Kedro 是一个面向生产环境的数据科学工具箱,用软件工程最佳实践帮你搭建可复现、易维护、模块化的数据工程与数据科学流水线。

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.

1.8.0发布于 2026-11-09 12:35 UTC11,051star +3 / 7 天1,097fork同步于 2026-11-10 23:30 UTC

怎么上手

由 README 提炼,执行前请确认
uv pip install kedro
硬件要求按 README 说明

README 未说明,纯 Python 框架,无 GPU 或显存要求

下载:源码 zip·GitHub Releases(安装包)本站不镜像安装包

项目说明

由 README 自动翻译提炼

用软件工程最佳实践搭建可复现、可维护、模块化的数据与数据科学流水线

Kedro 是一个面向生产环境的数据科学工具箱,用软件工程最佳实践帮你搭建可复现、易维护、模块化的数据工程与数据科学流水线。

核心能力

  • 提供基于 Cookiecutter Data Science 的项目模板
  • 用 Data Catalog 连接多种文件格式与文件系统并做版本管理
  • 自动解析纯 Python 函数间的依赖并可视化流水线
  • 内置 pytest、Sphinx、ruff 与标准 logging 的编码规范
  • 支持 Argo、Prefect、Kubeflow、AWS Batch、Databricks 部署

容易踩的坑

需 Python 3.10 至 3.14;流水线可视化要另装 Kedro-Viz;kedro-datasets 的 Python 版本支持比核心框架更早结束

内容由模型根据 README 生成,可能不完整或不准确,以官方文档为准。看原文 README

最近发版

5 个版本1.8.0
  1. 1.8.0最新正式版2026-11-09 · 2026-11-09 12:35 UTC

    Major features and improvements

    • Added support for passing keyword arguments such as use_cache or replace to OmegaConf.register_new_resolver() when registering custom resolvers with OmegaConfigLoader: a resolver in CONFIG_LOADER_ARGS["custom_resolvers"] can now be given as a (callable, kwargs) tuple. A resolver that is already registered is skipped unless replace=True is passed.
    • Added the option to allow data to be injected at run time when using the KedroServiceSession.
    • Added pre-loading of raw configuration in serving mode to improve request handling performance.
    • Added node groups to the project snapshot. Each PipelineSnapshot has a groups list of GroupSnapshot entries with the group's nodes, dependencies, inputs and outputs, also returned by GET /snapshot. get_project_snapshot takes a group_by argument to choose the grouping strategy, GET /snapshot takes it as a query parameter, and each PipelineSnapshot records it in group_by.
    • Added kedro.inspection.validate_deployment_grouping to check a node grouping before deployment. It reports datasets kept in memory that are passed between groups, cycles between groups, and datasets passed between groups on local disk.

    Bug fixes and other changes

    • Fixed the project snapshot (kedro inspect and GET /snapshot) listing parameter inputs such as params:model_options.test_size and parameters as datasets when a dataset factory pattern, for example a catch-all {default}, matches their names.
    • Fixed MemoryDataset losing its empty sentinel identity after pickle/unpickle, which could cause load() to return the sentinel instead of raising DatasetError.
    • Fixed AttributeError when naming a node built from a callable object, such as a class instance with __call__, and no explicit name. The node name now falls back to the class name.
    • Fixed typed parameter validation binding a params: input to the wrong function argument when a node declared with a dict of inputs leaves an earlier argument to its default. The typed requirement was taken from an argument the node never receives, so a run could fail on a type the user never asked for.
    • Fixed --load-versions (and DataCatalog.from_config(load_versions=...)) rejecting a dataset name that the catalog resolves through a user catch-all pattern, even though the name is in the catalog and returns a versioned dataset.
    • Fixed Pipeline.only_nodes_with_namespaces (and kedro run --namespaces) raising ValueError: Pipeline nodes must have unique names when a namespace is selected together with one it already covers, such as --namespaces a,a.b.
    • Stream outputs from generator nodes decorated with functools.wraps, and reject these nodes when asynchronous loading and saving is enabled. A decorator that consumes the generator, for example with list(), still has its result saved once.
    • Fixed the runner entry in the run_params passed to hooks: it now carries the runner name, instead of 'None' for a default run or the repr of the runner instance, which included a memory address. Applies to both KedroSession and KedroServiceSession.
    • Fixed az:// and abfs://container@account paths losing the container name.
    • Fixed OmegaConfigLoader ignoring config_loader["globals"] set after a ${globals:} interpolation was resolved.
    • Added @experimental tag to KedroServiceSession and the HTTP server (create_http_server) to clarify their experimental status.
    • Fixed CLI error messages being replaced by NoneType: None and the "Kedro project not found" hint being skipped with Click 8.5.

    Security hardening

    • Logging configuration now applies KEDRO_LOGGING_MODULE_ALLOWLIST to class, ext://, and QueueHandler listener references. Bare names, private objects, attribute chains, and cfg:// are no longer supported. A QueueHandler queue factory must be queue.Queue or queue.SimpleQueue, and a listener must be a QueueListener subclass. Use fully qualifie
  2. 1.7.02026-09-28 · 2026-09-28 16:13 UTC
    查看更新说明

    Major features and improvements

    • Added an optional params query parameter to GET /snapshot for resolving ${runtime_params:...} interpolation per request, using the same format as kedro run --params.
    • Graduated llm_context_node, LLMContextNode, LLMContext and tool from experimental to stable. They no longer emit a KedroExperimentalWarning.

    Breaking changes to the API

    • Security fix: Custom logging handler, filter or formatter classes referenced in conf/logging.yml are no longer imported unless their module is on the logging allowlist. Only the logging standard-library package (including logging.handlers) and kedro.logging are trusted by default. Projects that point conf/logging.yml at their own logging classes must now list those modules in the KEDRO_LOGGING_MODULE_ALLOWLIST environment variable (comma-separated) to keep them working.
  3. 1.6.02026-09-09 · 2026-09-09 09:14 UTC
    查看更新说明

    Major features and improvements

    • Added dataset validation: datasets can declare a validator in their catalog entry, and the DataCatalog enforces it on every load and save.
      • Failures raise DataValidationError (a DatasetError subclass), reporting every failed check at once.
      • Pandera is the built-in adapter, installed with the new kedro[pandera-pandas] or kedro[pandera-polars] extras; custom validator classes and functions also work.
      • The DATASET_VALIDATION setting and the KEDRO_DATASET_VALIDATION environment variable switch validation off.
      • validate_dataset and validate_catalog validate on demand, reporting structured ValidationResult outcomes instead of raising.
    • Added --runner-params to kedro run, allowing runner constructor keyword arguments such as max_workers to be passed from the CLI.
    • Added the node function name to project inspection snapshots as NodeSnapshot.func_name.
    • Added node source location metadata (NodeSnapshot.source) to inspection snapshots for displaying node code.
    • Added optional runtime_params to get_project_snapshot().

    Bug fixes and other changes

    • Fixed a thread-safety issue in _ProjectPipelines where concurrent access could trigger duplicate pipeline loading or operate on a cleared cache; all shared state mutations are now protected by a reentrant lock.
    • Fixed a thread-safety issue in the Kedro HTTP server by adding a serving_mode argument to KedroServiceSession.create() that preloads all pipelines upfront.
    • Fixed a RecursionError when initialising a session with dynaconf-backed settings by converting settings.SESSION_STORE_ARGS to a plain dict before deepcopying it.
    • Excluded kedro_benchmarks from the built wheel so benchmark tests are no longer shipped with the package.
    • Fixed get_close_matches returning duplicate suggestions when several inputs matched the same target, and being able to exhaust a one-shot iterable passed as targets.
    • Fixed docs generated with kedro new --tools=docs so the Sphinx HTML build runs sphinx-apidoc and creates API docs.
    • Deprecated --async flag for kedro run in favour of --runner-params=is_async=True.
    • Added a warning in ParallelRunner catalog validation that identifies cloud-backed datasets (e.g. S3, GCS, Azure Data Lake) failing to pickle and suggests using ThreadRunner or SequentialRunner instead.
    • Redacted credentials, signed-URL query parameters and fragments from dataset filepaths in AbstractDataset.__repr__, DatasetError messages, and HTTP server /snapshot and /run responses.

    Documentation changes

    • Added a documentation page for dataset validation.
    • Documented the HTTP server's restriction on runtime_params-resolved catalog type fields, in the templating guide and the HTTP server guide.
    • Added a new "Vibe coding with skills" page documenting the kedro-skills plugin.
    • Added mdformat as a Markdown autoformatter, run via make lint and as a pre-commit hook, and reformatted the existing Markdown. Fixed several step-by-step guides where numbered lists were rendering as repeated "1." instead of counting up, because their content wasn't indented under the right list item.
    • Fixed the DaskRunner example on the Dask deployment page.
    • Fixed the open_args_load examples in the data catalog documentation to use text mode so the encoding option applies.
    • Added a cheatsheet with Kedro concepts and CLI commands to the documentation.

    Breaking changes to the API

    • Security fix: A catalog type field resolved with runtime_params supplied through the HTTP server's POST /run no longer accepts an AbstractDataset class chosen by the request. runtime_params-driven type selection through other, trusted callers (for example, kedro run --params) is unaffected.
    • The settings.py entry RUNNER_MODULES_WHITELIST has been renamed to RUNNER_MODULE_ALLOWLIST, following the <COMPONENT>_ALLOWLIST naming convention for Kedro al
  4. 1.5.02026-06-29 · 2026-06-29 14:53 UTC
    查看更新说明

    Major features and improvements

    • Added a new HTTP Server layer which uses the KedroServiceSession to execute pipelines from HTTP requests. It offers the following endpoints:
      • /run to run pipelines with optional runtime parameters.
      • /health to check the server's health status.
      • /snapshot to retrieve the current project snapshot.
    • Added a new CLI command kedro server start to run the server.
    • Added selective pipeline loading: when --pipelines is specified, only the requested pipeline modules are imported.

    Bug fixes and other changes

    • Fixed Rich logging integration so node input/output brackets render correctly in console logs and dataset colour markup does not leak into plain log handlers.
    • Improved the AbstractDataset.from_config() error message for custom dataset classes that are still abstract, so it no longer suggests invalid constructor arguments when required dataset methods are missing.
    • Fixed kedro new accepting project names whose derived package name shadows a Python standard library module or is a Python keyword (e.g. email, json, import), which silently produced a broken, unimportable project. Such names are now rejected at creation time with a clear message.
    • Fixed kedro pipeline create accepting Python keywords (e.g. for, import, return) as pipeline names. Such names are now rejected at creation time with a clear error message.

    Documentation changes

    • Documented hooks limitation when using ParallelRunner.
    • Updated the Amazon EMR Serverless deployment guide with a strategy-first layout, custom container images, Spark catalog configuration, and troubleshooting for common deployment issues.
    • Updated the AWS Step Functions deployment guide with a strategy-first layout, pipeline-level namespace grouping, Lambda container deployment, and an end-to-end Spaceflights walkthrough.
    • Updated the AWS Batch deployment guide with a strategy-first layout, custom AWSBatchRunner usage, namespace-based job submission, and S3-backed catalog configuration.

    Community contributions

    Many thanks to the following Kedroids for contributing PRs to this release:

  5. 1.4.02026-05-22 · 2026-05-22 08:39 UTC
    查看更新说明

    Major features and improvements

    • Add official support for Python 3.14.
    • Added inspection API to get project snapshot.
    • Improved %load_node to include same-module helper dependencies via AST extraction, with explicit fallback warnings when extraction degrades to function-only source loading.
    • Scoped parameter type-hint validation to the pipelines being run. When kedro run --pipelines=<name> runs one or more named pipelines, only those pipelines' node signatures are inspected for type hints, avoiding unnecessary work and spurious "conflicting type" warnings from pipelines that aren't executing.

    Experimental features

    • Added the first iteration of the KedroServiceSession, a new session implementation that allows for multiple runs and data injection. NOTE: This session implementation is under active development and may occasionally contain bugs or breaking changes. We encourage users to try it out and share their feedback with us.

    Bug fixes and other changes

    • Fixed parameter validation for optional typed params so explicit None values are preserved and do not trigger validation errors.
    • Added review-kedro-pr agent skill (Cursor, GitHub Copilot) for Kedro-aware PR review with optional GitHub comment posting.
    • Removed outdated TRANSCODING_SEPARATOR alias from kedro.pipeline.pipeline.
    • Added an optional pydantic dependency extra, allowing users to enable Pydantic support with pip install "kedro[pydantic]".
    • Fixed parameter validation for namespaced pipelines.
    • Fixed runners iterating over iterable-but-non-generator node outputs (e.g. mne.Epochs) before saving them. Streaming behaviour is now restricted to generator-function nodes, so custom business objects are passed to catalog.save unchanged.

    Documentation changes

    • Added documentation for KedroServiceSession.
    • Updated the API documentation for kedro.framework.session to include the new KedroServiceSession and AbstractSession classes.
    • Added security model documentation covering trust boundaries, user responsibilities, and framework vulnerabilities.
    • Updated advanced_configuration.md with custom loader info.

    Community contributions

用户点评

登录用户的真实使用记录,可按版本和硬件筛选
共 0 条

    还没有人点评。第一个说出真实体验的人最有价值。

    README 原文

    未翻译

    Python version PyPI version Conda version License Slack Organisation Slack Archive GitHub Actions Workflow Status - Main GitHub Actions Workflow Status - Develop Documentation OpenSSF Best Practices Monthly downloads Total downloads

    Powered by Kedro

    What is Kedro?

    Kedro is a toolbox for production-ready data engineering and data science pipelines. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular. You can find out more at kedro.org.

    Kedro is an open-source Python framework hosted by the LF AI & Data Foundation.

    How do I install Kedro?

    To install Kedro from the Python Package Index (PyPI) run:

    uv pip install kedro
    

    It is also possible to install Kedro using conda:

    conda install -c conda-forge kedro
    

    Our Get Started guide contains full installation instructions, and includes how to set up Python virtual environments.

    Installation from source

    To access the latest Kedro version before its official release, install it from the main branch.

    uv pip install git+https://github.com/kedro-org/kedro@main
    

    What are the main features of Kedro?

    FeatureWhat is this?
    Project TemplateA standard, modifiable and easy-to-use project template based on Cookiecutter Data Science.
    Data CatalogA series of lightweight data connectors used to save and load data across many different file formats and file systems, including local and network file systems, cloud object stores, and HDFS. The Data Catalog also includes data and model versioning for file-based systems.
    Pipeline AbstractionAutomatic resolution of dependencies between pure Python functions and data pipeline visualisation using Kedro-Viz.
    Coding StandardsTest-driven development using pytest, produce well-documented code using Sphinx, create linted code with support for ruff and make use of the standard Python logging library.
    Flexible DeploymentDeployment strategies that include single or distributed-machine deployment as well as additional support for deploying on Argo, Prefect, Kubeflow, AWS Batch, and Databricks.

    How do I use Kedro?

    The Kedro documentation first explains how to install Kedro and then introduces key Kedro concepts.

    You can then review the spaceflights tutorial to build a Kedro project for hands-on experience.

    For new and intermediate Kedro users, there's a comprehensive section on how to visualise Kedro projects using Kedro-Viz.

    Additional documentation explains how to work with Kedro and Jupyter notebooks, and there are a set of advanced user guides for advanced for key Kedro features. We also recommend the API reference documentation for further information.

    Why does Kedro exist?

    Kedro is built upon our collective best-practice (and mistakes) trying to deliver real-world ML applications that have vast amounts of raw unvetted data. We developed Kedro to achieve the following:

    • To address the main shortcomings of Jupyter notebooks, one-off scripts, and glue-code because there is a focus on creating maintainable data engineering and data science code
    • To enhance team collaboration when different team members have varied exposure to software engineering concepts
    • To increase efficiency, because applied concepts like modularity and separation of concerns inspire the creation of reusable analytics code

    Find out more about how Kedro can answer your use cases from the product FAQs on the Kedro website.

    The humans behind Kedro

    The Kedro product team and a number of open source contributors from across the world maintain Kedro.

    Can I contribute?

    Yes! We welcome all kinds of contributions. Check out our guide to contributing to Kedro.

    Where can I learn more?

    There is a growing community around Kedro. We encourage you to ask and answer technical questions on Slack and bookmark the Linen archive of past discussions.

    We keep a list of technical FAQs in the Kedro documentation and you can find a growing list of blog posts, videos and projects that use Kedro over on the awesome-kedro GitHub repository. If you have created anything with Kedro we'd love to include it on the list. Just make a PR to add it!

    How can I cite Kedro?

    If you're an academic, Kedro can also help you, for example, as a tool to solve the problem of reproducible research. Use the "Cite this repository" button on our repository to generate a citation from the CITATION.cff file.

    Python version support policy

    • The core Kedro Framework supports all Python versions that are actively maintained by the CPython core team. When a Python version reaches end of life, support for that version is dropped from Kedro. This is not considered a breaking change.
    • The Kedro Datasets package follows the NEP 29 Python version support policy. This means that kedro-datasets generally drops Python version support before kedro. This is because kedro-datasets has a lot of dependencies that follow NEP 29 and the more conservative version support approach of the Kedro Framework makes it hard to manage those dependencies properly.

    ☕️ Kedro Coffee Chat 🔶

    We appreciate our community and want to stay connected. For that, we offer a public Coffee Chat format where we share updates and cool stuff around Kedro once every two weeks and give you time to ask your questions live.

    Check out the upcoming demo topics and dates at the Kedro Coffee Chat wiki page.

    Follow our Slack announcement channel to see Kedro Coffee Chat announcements and access demo recordings.