需 Python 3.10 至 3.14;流水线可视化要另装 Kedro-Viz;kedro-datasets 的 Python 版本支持比核心框架更早结束
kedro
kedro-org活跃维护Kedro 是一个面向生产环境的数据科学工具箱,用软件工程最佳实践帮你搭建可复现、易维护、模块化的数据工程与数据科学流水线。
Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
uv pip install kedroREADME 未说明,纯 Python 框架,无 GPU 或显存要求
用软件工程最佳实践搭建可复现、可维护、模块化的数据与数据科学流水线
Kedro 是一个面向生产环境的数据科学工具箱,用软件工程最佳实践帮你搭建可复现、易维护、模块化的数据工程与数据科学流水线。
核心能力
- 提供基于 Cookiecutter Data Science 的项目模板
- 用 Data Catalog 连接多种文件格式与文件系统并做版本管理
- 自动解析纯 Python 函数间的依赖并可视化流水线
- 内置 pytest、Sphinx、ruff 与标准 logging 的编码规范
- 支持 Argo、Prefect、Kubeflow、AWS Batch、Databricks 部署
容易踩的坑
- 1.8.0最新正式版2026-11-09 · 2026-11-09 12:35 UTC
Major features and improvements
- Added support for passing keyword arguments such as
use_cacheorreplacetoOmegaConf.register_new_resolver()when registering custom resolvers withOmegaConfigLoader: a resolver inCONFIG_LOADER_ARGS["custom_resolvers"]can now be given as a(callable, kwargs)tuple. A resolver that is already registered is skipped unlessreplace=Trueis passed. - Added the option to allow data to be injected at run time when using the
KedroServiceSession. - Added pre-loading of raw configuration in serving mode to improve request handling performance.
- Added node groups to the project snapshot. Each
PipelineSnapshothas agroupslist ofGroupSnapshotentries with the group's nodes, dependencies, inputs and outputs, also returned byGET /snapshot.get_project_snapshottakes agroup_byargument to choose the grouping strategy,GET /snapshottakes it as a query parameter, and eachPipelineSnapshotrecords it ingroup_by. - Added
kedro.inspection.validate_deployment_groupingto check a node grouping before deployment. It reports datasets kept in memory that are passed between groups, cycles between groups, and datasets passed between groups on local disk.
Bug fixes and other changes
- Fixed the project snapshot (
kedro inspectandGET /snapshot) listing parameter inputs such asparams:model_options.test_sizeandparametersas datasets when a dataset factory pattern, for example a catch-all{default}, matches their names. - Fixed MemoryDataset losing its empty sentinel identity after pickle/unpickle, which could cause load() to return the sentinel instead of raising DatasetError.
- Fixed
AttributeErrorwhen naming a node built from a callable object, such as a class instance with__call__, and no explicit name. The node name now falls back to the class name. - Fixed typed parameter validation binding a
params:input to the wrong function argument when a node declared with a dict of inputs leaves an earlier argument to its default. The typed requirement was taken from an argument the node never receives, so a run could fail on a type the user never asked for. - Fixed
--load-versions(andDataCatalog.from_config(load_versions=...)) rejecting a dataset name that the catalog resolves through a user catch-all pattern, even though the name is in the catalog and returns a versioned dataset. - Fixed
Pipeline.only_nodes_with_namespaces(andkedro run --namespaces) raisingValueError: Pipeline nodes must have unique nameswhen a namespace is selected together with one it already covers, such as--namespaces a,a.b. - Stream outputs from generator nodes decorated with
functools.wraps, and reject these nodes when asynchronous loading and saving is enabled. A decorator that consumes the generator, for example withlist(), still has its result saved once. - Fixed the
runnerentry in therun_paramspassed to hooks: it now carries the runner name, instead of'None'for a default run or thereprof the runner instance, which included a memory address. Applies to bothKedroSessionandKedroServiceSession. - Fixed
az://andabfs://container@accountpaths losing the container name. - Fixed
OmegaConfigLoaderignoringconfig_loader["globals"]set after a${globals:}interpolation was resolved. - Added
@experimentaltag toKedroServiceSessionand the HTTP server (create_http_server) to clarify their experimental status. - Fixed CLI error messages being replaced by
NoneType: Noneand the "Kedro project not found" hint being skipped with Click 8.5.
Security hardening
- Logging configuration now applies
KEDRO_LOGGING_MODULE_ALLOWLISTtoclass,ext://, andQueueHandlerlistenerreferences. Bare names, private objects, attribute chains, andcfg://are no longer supported. AQueueHandlerqueuefactory must bequeue.Queueorqueue.SimpleQueue, and alistenermust be aQueueListenersubclass. Use fully qualifie
- Added support for passing keyword arguments such as
- 1.7.02026-09-28 · 2026-09-28 16:13 UTC
查看更新说明
Major features and improvements
- Added an optional
paramsquery parameter toGET /snapshotfor resolving${runtime_params:...}interpolation per request, using the same format askedro run --params. - Graduated
llm_context_node,LLMContextNode,LLMContextandtoolfrom experimental to stable. They no longer emit aKedroExperimentalWarning.
Breaking changes to the API
- Security fix: Custom logging handler, filter or formatter classes referenced in
conf/logging.ymlare no longer imported unless their module is on the logging allowlist. Only theloggingstandard-library package (includinglogging.handlers) andkedro.loggingare trusted by default. Projects that pointconf/logging.ymlat their own logging classes must now list those modules in theKEDRO_LOGGING_MODULE_ALLOWLISTenvironment variable (comma-separated) to keep them working.
- Added an optional
- 1.6.02026-09-09 · 2026-09-09 09:14 UTC
查看更新说明
Major features and improvements
- Added dataset validation: datasets can declare a
validatorin their catalog entry, and theDataCatalogenforces it on everyloadandsave.- Failures raise
DataValidationError(aDatasetErrorsubclass), reporting every failed check at once. - Pandera is the built-in adapter, installed with the new
kedro[pandera-pandas]orkedro[pandera-polars]extras; custom validator classes and functions also work. - The
DATASET_VALIDATIONsetting and theKEDRO_DATASET_VALIDATIONenvironment variable switch validation off. validate_datasetandvalidate_catalogvalidate on demand, reporting structuredValidationResultoutcomes instead of raising.
- Failures raise
- Added
--runner-paramstokedro run, allowing runner constructor keyword arguments such asmax_workersto be passed from the CLI. - Added the node function name to project inspection snapshots as
NodeSnapshot.func_name. - Added node source location metadata (
NodeSnapshot.source) to inspection snapshots for displaying node code. - Added optional
runtime_paramstoget_project_snapshot().
Bug fixes and other changes
- Fixed a thread-safety issue in
_ProjectPipelineswhere concurrent access could trigger duplicate pipeline loading or operate on a cleared cache; all shared state mutations are now protected by a reentrant lock. - Fixed a thread-safety issue in the Kedro HTTP server by adding a
serving_modeargument toKedroServiceSession.create()that preloads all pipelines upfront. - Fixed a
RecursionErrorwhen initialising a session with dynaconf-backed settings by convertingsettings.SESSION_STORE_ARGSto a plaindictbefore deepcopying it. - Excluded
kedro_benchmarksfrom the built wheel so benchmark tests are no longer shipped with the package. - Fixed
get_close_matchesreturning duplicate suggestions when several inputs matched the same target, and being able to exhaust a one-shot iterable passed astargets. - Fixed docs generated with
kedro new --tools=docsso the Sphinx HTML build runssphinx-apidocand creates API docs. - Deprecated
--asyncflag forkedro runin favour of--runner-params=is_async=True. - Added a warning in
ParallelRunnercatalog validation that identifies cloud-backed datasets (e.g. S3, GCS, Azure Data Lake) failing to pickle and suggests usingThreadRunnerorSequentialRunnerinstead. - Redacted credentials, signed-URL query parameters and fragments from dataset filepaths in
AbstractDataset.__repr__,DatasetErrormessages, and HTTP server/snapshotand/runresponses.
Documentation changes
- Added a documentation page for dataset validation.
- Documented the HTTP server's restriction on
runtime_params-resolved catalogtypefields, in the templating guide and the HTTP server guide. - Added a new "Vibe coding with skills" page documenting the
kedro-skillsplugin. - Added
mdformatas a Markdown autoformatter, run viamake lintand as a pre-commit hook, and reformatted the existing Markdown. Fixed several step-by-step guides where numbered lists were rendering as repeated "1." instead of counting up, because their content wasn't indented under the right list item. - Fixed the
DaskRunnerexample on the Dask deployment page. - Fixed the
open_args_loadexamples in the data catalog documentation to use text mode so theencodingoption applies. - Added a cheatsheet with Kedro concepts and CLI commands to the documentation.
Breaking changes to the API
- Security fix: A catalog
typefield resolved withruntime_paramssupplied through the HTTP server'sPOST /runno longer accepts anAbstractDatasetclass chosen by the request.runtime_params-driventypeselection through other, trusted callers (for example,kedro run --params) is unaffected. - The
settings.pyentryRUNNER_MODULES_WHITELISThas been renamed toRUNNER_MODULE_ALLOWLIST, following the<COMPONENT>_ALLOWLISTnaming convention for Kedro al
- Added dataset validation: datasets can declare a
- 1.5.02026-06-29 · 2026-06-29 14:53 UTC
查看更新说明
Major features and improvements
- Added a new HTTP Server layer which uses the
KedroServiceSessionto execute pipelines from HTTP requests. It offers the following endpoints:/runto run pipelines with optional runtime parameters./healthto check the server's health status./snapshotto retrieve the current project snapshot.
- Added a new CLI command
kedro server startto run the server. - Added selective pipeline loading: when
--pipelinesis specified, only the requested pipeline modules are imported.
Bug fixes and other changes
- Fixed Rich logging integration so node input/output brackets render correctly in console logs and dataset colour markup does not leak into plain log handlers.
- Improved the
AbstractDataset.from_config()error message for custom dataset classes that are still abstract, so it no longer suggests invalid constructor arguments when required dataset methods are missing. - Fixed
kedro newaccepting project names whose derived package name shadows a Python standard library module or is a Python keyword (e.g.email,json,import), which silently produced a broken, unimportable project. Such names are now rejected at creation time with a clear message. - Fixed
kedro pipeline createaccepting Python keywords (e.g.for,import,return) as pipeline names. Such names are now rejected at creation time with a clear error message.
Documentation changes
- Documented hooks limitation when using
ParallelRunner. - Updated the Amazon EMR Serverless deployment guide with a strategy-first layout, custom container images, Spark catalog configuration, and troubleshooting for common deployment issues.
- Updated the AWS Step Functions deployment guide with a strategy-first layout, pipeline-level namespace grouping, Lambda container deployment, and an end-to-end Spaceflights walkthrough.
- Updated the AWS Batch deployment guide with a strategy-first layout, custom
AWSBatchRunnerusage, namespace-based job submission, and S3-backed catalog configuration.
Community contributions
Many thanks to the following Kedroids for contributing PRs to this release:
- Added a new HTTP Server layer which uses the
- 1.4.02026-05-22 · 2026-05-22 08:39 UTC
查看更新说明
Major features and improvements
- Add official support for Python 3.14.
- Added inspection API to get project snapshot.
- Improved
%load_nodeto include same-module helper dependencies via AST extraction, with explicit fallback warnings when extraction degrades to function-only source loading. - Scoped parameter type-hint validation to the pipelines being run. When
kedro run --pipelines=<name>runs one or more named pipelines, only those pipelines' node signatures are inspected for type hints, avoiding unnecessary work and spurious "conflicting type" warnings from pipelines that aren't executing.
Experimental features
- Added the first iteration of the
KedroServiceSession, a new session implementation that allows for multiple runs and data injection. NOTE: This session implementation is under active development and may occasionally contain bugs or breaking changes. We encourage users to try it out and share their feedback with us.
Bug fixes and other changes
- Fixed parameter validation for optional typed params so explicit
Nonevalues are preserved and do not trigger validation errors. - Added
review-kedro-pragent skill (Cursor, GitHub Copilot) for Kedro-aware PR review with optional GitHub comment posting. - Removed outdated
TRANSCODING_SEPARATORalias fromkedro.pipeline.pipeline. - Added an optional
pydanticdependency extra, allowing users to enable Pydantic support withpip install "kedro[pydantic]". - Fixed parameter validation for namespaced pipelines.
- Fixed runners iterating over iterable-but-non-generator node outputs (e.g.
mne.Epochs) before saving them. Streaming behaviour is now restricted to generator-function nodes, so custom business objects are passed tocatalog.saveunchanged.
Documentation changes
- Added documentation for
KedroServiceSession. - Updated the API documentation for
kedro.framework.sessionto include the newKedroServiceSessionandAbstractSessionclasses. - Added security model documentation covering trust boundaries, user responsibilities, and framework vulnerabilities.
- Updated
advanced_configuration.mdwith custom loader info.
Community contributions
用户点评
登录用户的真实使用记录,可按版本和硬件筛选还没有人点评。第一个说出真实体验的人最有价值。
What is Kedro?
Kedro is a toolbox for production-ready data engineering and data science pipelines. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular. You can find out more at kedro.org.
Kedro is an open-source Python framework hosted by the LF AI & Data Foundation.
How do I install Kedro?
To install Kedro from the Python Package Index (PyPI) run:
uv pip install kedro
It is also possible to install Kedro using conda:
conda install -c conda-forge kedro
Our Get Started guide contains full installation instructions, and includes how to set up Python virtual environments.
Installation from source
To access the latest Kedro version before its official release, install it from the main branch.
uv pip install git+https://github.com/kedro-org/kedro@main
What are the main features of Kedro?
| Feature | What is this? |
|---|---|
| Project Template | A standard, modifiable and easy-to-use project template based on Cookiecutter Data Science. |
| Data Catalog | A series of lightweight data connectors used to save and load data across many different file formats and file systems, including local and network file systems, cloud object stores, and HDFS. The Data Catalog also includes data and model versioning for file-based systems. |
| Pipeline Abstraction | Automatic resolution of dependencies between pure Python functions and data pipeline visualisation using Kedro-Viz. |
| Coding Standards | Test-driven development using pytest, produce well-documented code using Sphinx, create linted code with support for ruff and make use of the standard Python logging library. |
| Flexible Deployment | Deployment strategies that include single or distributed-machine deployment as well as additional support for deploying on Argo, Prefect, Kubeflow, AWS Batch, and Databricks. |
How do I use Kedro?
The Kedro documentation first explains how to install Kedro and then introduces key Kedro concepts.
You can then review the spaceflights tutorial to build a Kedro project for hands-on experience.
For new and intermediate Kedro users, there's a comprehensive section on how to visualise Kedro projects using Kedro-Viz.
Additional documentation explains how to work with Kedro and Jupyter notebooks, and there are a set of advanced user guides for advanced for key Kedro features. We also recommend the API reference documentation for further information.
Why does Kedro exist?
Kedro is built upon our collective best-practice (and mistakes) trying to deliver real-world ML applications that have vast amounts of raw unvetted data. We developed Kedro to achieve the following:
- To address the main shortcomings of Jupyter notebooks, one-off scripts, and glue-code because there is a focus on creating maintainable data engineering and data science code
- To enhance team collaboration when different team members have varied exposure to software engineering concepts
- To increase efficiency, because applied concepts like modularity and separation of concerns inspire the creation of reusable analytics code
Find out more about how Kedro can answer your use cases from the product FAQs on the Kedro website.
The humans behind Kedro
The Kedro product team and a number of open source contributors from across the world maintain Kedro.
Can I contribute?
Yes! We welcome all kinds of contributions. Check out our guide to contributing to Kedro.
Where can I learn more?
There is a growing community around Kedro. We encourage you to ask and answer technical questions on Slack and bookmark the Linen archive of past discussions.
We keep a list of technical FAQs in the Kedro documentation and you can find a growing list of blog posts, videos and projects that use Kedro over on the awesome-kedro GitHub repository. If you have created anything with Kedro we'd love to include it on the list. Just make a PR to add it!
How can I cite Kedro?
If you're an academic, Kedro can also help you, for example, as a tool to solve the problem of reproducible research. Use the "Cite this repository" button on our repository to generate a citation from the CITATION.cff file.
Python version support policy
- The core Kedro Framework supports all Python versions that are actively maintained by the CPython core team. When a Python version reaches end of life, support for that version is dropped from Kedro. This is not considered a breaking change.
- The Kedro Datasets package follows the NEP 29 Python version support policy. This means that
kedro-datasetsgenerally drops Python version support beforekedro. This is becausekedro-datasetshas a lot of dependencies that follow NEP 29 and the more conservative version support approach of the Kedro Framework makes it hard to manage those dependencies properly.
☕️ Kedro Coffee Chat 🔶
We appreciate our community and want to stay connected. For that, we offer a public Coffee Chat format where we share updates and cool stuff around Kedro once every two weeks and give you time to ask your questions live.
Check out the upcoming demo topics and dates at the Kedro Coffee Chat wiki page.
Follow our Slack announcement channel to see Kedro Coffee Chat announcements and access demo recordings.