pipelines

kubeflowApache-2.0活跃维护

在 Kubernetes 上编排和运行机器学习流水线的平台,把数据准备、训练、部署等步骤串成可复现、可调度的 DAG 工作流。

Machine Learning Pipelines for Kubeflow

2.18.0发布于 2026-10-28 01:31 UTC4,262star2,186fork同步于 2026-11-10 17:39 UTC

怎么上手

由 README 提炼,执行前请确认
pip install kfp
硬件要求按 README 说明

README 未说明;部署依赖 Kubernetes 集群,需配套 Argo Workflows v4.0/v4.1 与 MySQL v8。

下载:源码 zip·GitHub Releases(安装包)本站不镜像安装包

项目说明

由 README 自动翻译提炼

在 Kubernetes 上编排可复用端到端机器学习流水线的平台

在 Kubernetes 上编排和运行机器学习流水线的平台,把数据准备、训练、部署等步骤串成可复现、可调度的 DAG 工作流。

核心能力

  • 用 Python SDK 定义并复用端到端 ML 流水线
  • 底层由 Argo Workflows 编排 Kubernetes 资源
  • 支持实验管理与多次试验对比
  • 组件与流水线可复用,免重复搭建
  • 可随 Kubeflow 平台或独立部署

容易踩的坑

KFP 3.0 分支只支持 Argo Workflows 4.x,升级前须先把控制器升到受支持的 4.x 版本;Argo 3.x 在该分支不受支持;README 未给出独立部署的具体命令,需参照官方安装文档。

内容由模型根据 README 生成,可能不完整或不准确,以官方文档为准。看原文 README

最近发版

5 个版本2.18.0
  1. 2.18.0Version 2.18.0最新正式版2026-10-28 · 2026-10-28 01:31 UTC

    Kubeflow Pipelines 2.18 makes pipelines easier to inspect and author, adds data-transfer and retention tools, and strengthens multi-user security and operational resilience.

    Highlights

    • Automatic artifact previews. Supported Python components can generate Dataset and Model previews in run details, helping you inspect outputs without downloading the full artifact. #15087
    • Clearer pipeline visualization. Nested sub-DAGs expand inline in the UI. Standalone HTML/Mermaid previews with local watch mode let you explore pipelines without deploying a server. #14864, #15064
    • More authoring options. Components gain Kubernetes Dynamic Resource Allocation, init containers, Pydantic BaseModel inputs and outputs, and custom task cache keys. DRA requires a compatible backend, Kubernetes configuration, and resource driver. #14243, #13651, #13995, #13762
    • Pipeline data management. Export/import transfers experiments, pipeline versions and tags, completed run history with MLMD lineage, and schedule definitions between compatible installations. Optional retention policies archive and delete expired completed runs; retention is disabled by default. #14659, #13532

    Security improvements

    • Stronger authorization and namespace isolation. Runs and recurring schedules check permissions for workflow service accounts. Pipeline references, shared uploads, and streamed logs receive additional authorization checks; pipeline lookups and legacy cache reuse enforce namespace boundaries more consistently. #14655, #14568, #14359
    • Safer artifact and viewer access. Artifact, archived-log, and TensorBoard handling gain stronger authorization, endpoint/path validation, and redirect protections. #14349, #14706, #15143
    • Additional hardening. Archive and request-size limits reduce resource-exhaustion risks. API-server TLS minimum versions and cipher suites are configurable, and dependencies and runtime images receive security updates. #14171, #14655, #13609

    Availability and resilience

    • Recovery across restarts and replicas. Shared UI signing keys preserve TensorBoard URLs across subsequent restarts and replicas. Retention workers use leader election and graceful draining; database-coordinated index preparation recovers interrupted managed builds. These improvements do not establish HA for every component or guarantee zero-downtime upgrades. #14655, #14877
    • More reliable scheduling and execution. Managed upgrades automatically adopt eligible recurring schedules. Driver retries avoid duplicate executions, and workflow-status reporting recovers more reliably from interrupted work. Scheduling mutations can briefly wait for the upgrade handoff instead of immediately failing. #14741, #14412, #14776, [#15049](https://github.com/kubeflow/pipelin
  2. sdk-2.18.0KFP SDK v2.18.02026-10-28 · 2026-10-28 01:30 UTC
    查看更新说明

    Kubeflow Pipelines SDK 2.18 adds local pipeline visualization, richer component authoring, and better control over task retries, caching, and namespaces. Python 3.9+ remains supported.

    Highlights

    • Preview pipelines locally. Export standalone HTML/Mermaid visualizations and use local watch mode to inspect pipeline structure without deploying a KFP server. #15064
    • Preview component outputs. Supported Python components can automatically generate Dataset and Model previews for the 2.18 run-details UI. Privacy controls include opt-outs and schema-only previews. #15087
    • Author richer components. Python components support Pydantic BaseModel inputs and outputs. The Kubernetes extension adds init containers and Dynamic Resource Allocation (DRA). #13995, #13651, #14243
    • Control retries and caching. set_retry(policy=...) selects which failures trigger retries with Argo Workflows. set_caching_options(cache_key=...) allows custom task cache keys. #14655, #13762
    • Use namespaces explicitly. Client methods support explicit namespaces for immediate and recurring runs and private-pipeline lookup. Run listing gains a server-side pipeline filter, and runs can be created from a pipeline ID without specifying a version ID. #14812, #14611, #14660, #14711

    Bug fixes

    • Correct pipeline-ID-only run creation and private-pipeline namespace lookup. #14711, #14611
    • Preserve embedded artifacts reached through symlinks and expose the underlying cause of Kubernetes configuration errors. #13554, #13871
    • Fix ParallelFor task naming and component-description parsing edge cases. #13707, #13619

    Installation

    Install the matching 2.18.0 packages on Python 3.9 or later:

    python -m pip install kfp==2.18.0 kfp-pipeline-spec==2.18.0 kfp-server-api==2.18.0
    

    For Kubernetes-specific pipeline features, also install:

    python -m pip install kfp-kubernetes==2.18.0
    

    The four distributions remain separate. Upgrading these packages does not upgrade your cluster or rebuild existing component images. Generic container components do not acquire a Python requirement merely because KFP executes them.

    Compatibility and upgrade guidance

    • Backend and Kubernetes support: use a matching 2.18 backend for the new runtime features. DRA additionally requires Kubernetes 1.31+ with DynamicResourceAllocation enabled and an installed DRA driver; the feature is GA and enabled by default in Kubernetes 1.34. Older KFP backends reject the new DRA platform field.
    • Retry policies and custom cache keys: retry policy selection is Argo-specific. An omitted policy uses the deployment's Argo configuration; bundled 2.18 manifests use OnError. Custom cache keys replace automatic input, image, command, argument, and PVC-name contributions, so callers must account for changes that should invalidate cached results.
    • Preview privacy: previews may contain sample data or model configuration. Review privacy settings before enabling them for sensitive outputs. Existing artifacts are not backfilled.
    • Private package mirrors: generated component installatio
  3. 2.17.3Version 2.17.32026-10-22 · 2026-10-22 15:03 UTC
    查看更新说明

    What's changed

    This maintenance release updates Go, Python, frontend dependencies and runtime images while retaining SDK support for Python 3.9.

    • Backport the Go trust-store fix from #15025, including Go 1.27.3 and updated builders, with additional security dependency updates in #15026.
    • Fix the proxy and cache-deployer runtime images (#15036).
    • Remove frontend server development dependencies from the runtime image (#15040).
    • Pin CI pipeline SDK installation to the tested commit (#15041), quote release manifest sources (#15030), and preserve release image publication with read-only workflow defaults (#15027).

    Security and compatibility

    Qualification of the released commit rebuilt and scanned all 24 image/platform combinations with zero scanner-reported fixable image findings. The four SDK packages passed installation and compilation checks on Python 3.9–3.13.

    Security: Release-image scans found no vulnerabilities with available fixes at qualification time. Python 3.9 remains supported with documented dependency exceptions; the tested Python 3.10–3.13 environments had no findings. See the security qualification details and remaining findings.

    The latest image tags now point to 2.17.3. Use the explicit 2.17.3 tag to pin deployments to this release. See the SDK release for Python packages.

    Full changelog: https://github.com/kubeflow/pipelines/compare/2.17.2...2.17.3

  4. sdk-2.17.3KFP SDK v2.17.32026-10-22 · 2026-10-22 15:03 UTC
    查看更新说明

    Release of:

    • KFP SDK
    • KFP Kubernetes
    • KFP Server API
    • KFP Pipeline Spec

    To install the KFP SDK:

    pip install kfp-pipeline-spec==2.17.3
    pip install kfp-server-api==2.17.3
    pip install kfp==2.17.3
    pip install kfp-kubernetes==2.17.3
    

    For changelog, see https://github.com/kubeflow/pipelines/blob/sdk-2.17.3/sdk/RELEASE.md.

  5. 2.17.2Version 2.17.22026-09-04 · 2026-09-04 21:46 UTC
    查看更新说明

用户点评

登录用户的真实使用记录,可按版本和硬件筛选
共 0 条

    还没有人点评。第一个说出真实体验的人最有价值。

    README 原文

    未翻译

    Kubeflow Pipelines

    SDK Documentation Status SDK Package version SDK Supported Python versions OpenSSF Best Practices Ask DeepWiki

    Overview of the Kubeflow pipelines service

    Kubeflow is a machine learning (ML) toolkit that is dedicated to making deployments of ML workflows on Kubernetes simple, portable, and scalable.

    Kubeflow pipelines are reusable end-to-end ML workflows built using the Kubeflow Pipelines SDK.

    The Kubeflow pipelines service has the following goals:

    • End to end orchestration: enabling and simplifying the orchestration of end to end machine learning pipelines
    • Easy experimentation: making it easy for you to try numerous ideas and techniques, and manage your various trials/experiments.
    • Easy re-use: enabling you to re-use components and pipelines to quickly cobble together end to end solutions, without having to re-build each time.

    Installation

    Dependencies Compatibility Matrix

    DependencyVersions
    Argo Workflowsv4.0, v4.1
    MySQLv8

    KFP 3.0 compatibility notice: This branch supports Argo Workflows 4.0.x and 4.1.x, tested at 4.0.13 and 4.1.6 respectively. Argo 3.x is not supported on this branch; KFP 2.x release branches retain their documented compatibility coverage. Before upgrading to KFP 3.0, operators must upgrade their controller to a supported 4.x release. Follow the KFP 3.0 compatibility tracking issue for the removal and migration work.

    Documentation

    See Releases for what changed, upgrade guidance, compatibility, and the maintainer release runbook.

    Get started with your first pipeline and read further information in the Kubeflow Pipelines overview.

    See the various ways you can use the Kubeflow Pipelines SDK.

    See the Kubeflow Pipelines API doc for API specification.

    Consult the Python SDK reference docs when writing pipelines using the Python SDK.

    Deep Wiki

    Check out our AI Powered repo documentation on DeepWiki.

    :warning: Please note, this is AI generated and may not have completely accurate information.

    Contributing to Kubeflow Pipelines

    Before you start contributing to Kubeflow Pipelines, read the guidelines in How to Contribute. To learn how to build and deploy Kubeflow Pipelines from source code, read the developer guide.

    Optional just command runner

    For local developer convenience, this repository includes an optional just command runner at the repo root. It provides short aliases for existing make targets and does not replace any CI or release workflows.

    To use it, install just and run, for example:

    just           # list available recipes
    just backend-test
    just backend-images
    

    Notes:

    • All just recipes are thin wrappers around existing make targets (for example, make -C backend/src/v2 test).
    • There is intentionally no generic just build or just test recipe; heavy or Docker-building flows are exposed only via explicitly named recipes such as backend-images.

    Kubeflow Pipelines Community

    Community Meeting

    The Kubeflow Pipelines Community Meeting occurs every other Wed 10-11AM (PST).

    Calendar Invite

    Direct Meeting Link

    Meeting notes

    Slack

    We also have a slack channel (#kubeflow-pipelines) on the Cloud Native Computing Foundation Slack workspace. You can find more details at https://www.kubeflow.org/docs/about/community/#kubeflow-slack-channels

    Architecture

    Details about the KFP Architecture can be found at Architecture.md

    Blog posts

    Acknowledgments

    Kubeflow pipelines uses Argo Workflows by default under the hood to orchestrate Kubernetes resources. The Argo community has been very supportive and we are very grateful.