登录

cleanrl

vwxyzjn更新放缓 有专家点评

单文件实现的深度强化学习算法集合，涵盖 PPO、DQN、SAC 等主流算法，代码简洁易读，适合做研究和快速实验。

High-quality single file implementation of Deep Reinforcement Learning algorithms with research-friendly features (PPO, DQN, C51, DDPG, TD3, SAC, PPG)

v1.0.0发布于 2022-11-14 04:06 UTC10,672star +20 / 7 天1,195fork同步于 2026-11-11 23:28 UTC
深度强化学习单文件实现PPO算法实现研究实验
对比 GitHub 官网a2cactor-criticadvantage-actor-criticaleatari

专家点评

上手难度：进阶更新于 2026-11-11

单文件 DRL 算法参考实现，用来读源码和跑基线，不是拿来当依赖库的

适用场景

  • 复现论文基线：每个算法一个文件、种子固定、超参写死在文件顶部，改一行就能跑对照实验，比在模块化框架里翻三层继承找超参快得多
  • 读算法细节：想搞清楚 PPO 的 GAE 和 clip 具体怎么落地、PPG 的辅助阶段怎么切，直接读 300 多行的单文件比读一个框架的抽象层有效
  • 小规模原型验证：CartPole、经典控制这类 CPU 能跑的任务，clone 完装完就能出 Tensorboard 曲线，不需要先搭训练基础设施
  • 教学或团队内部分享：单文件天然适合逐行讲，换成一个 import 式的框架反而要花时间解释抽象

踩坑

  • 想把它当库 import 进自己的训练代码时会撞墙：README 明说设计上不供复用，算法逻辑和训练循环、日志、评估耦合在同一个脚本里，抽出来等于重写
  • 环境依赖是分层的：Atari、Procgen、envpool 各自要装可选依赖，envpool 加速只在 Linux 可用，在 macOS 上跑 Atari 会退回慢路径
  • Python 版本被卡在 >=3.7.1,<3.11 且要求较新的 uv，环境里已有 3.11+ 的项目要单独建 venv；issue 里出现过装完 poetry 后 No module named 'tomli' 这类依赖冲突，说明和已有工具链共存需要留意
  • 缺模型保存/加载和设计评估的模块，issue 里这条被反复讨论，想接自己的评测流水线要自己补

同类对比

要模块化、可组合、能当依赖用的选 Stable-Baselines3；要大规模并行采样吞吐的看 EnvPool 配套方案或 LeanRL 这类走 torch.compile/CUDA Graphs 的路线。CleanRL 的分流点是：你要的是可读的参考实现和可复现基线，不是可扩展的框架。

怎么上手

由 README 提炼，执行前请确认
原文 
git clone https://github.com/vwxyzjn/cleanrl.git && cd cleanrl
uv pip install .
uv run python cleanrl/ppo.py --seed 1 --env-id CartPole-v0 --total-timesteps 50000
硬件要求按 README 说明

README 未说明显存要求；经典控制任务可 CPU 运行，Atari 等任务建议用 GPU，envpool 加速仅支持 Linux

多卡 / 张量并行
下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼

单文件实现的深度强化学习算法库，适合读源码理解算法细节或做原型验证

单文件实现的深度强化学习算法集合，涵盖 PPO、DQN、SAC 等主流算法，代码简洁易读，适合做研究和快速实验。

核心能力

  • 每个算法变体写成单个独立文件，如 ppo_atari.py 仅 340 行
  • 覆盖 PPO、DQN、C51、SAC、DDPG、TD3、PPG 等算法及变体
  • 内置 Tensorboard 日志与随机种子本地复现
  • 支持 Weights and Biases 实验管理与 Docker/AWS 云端批量跑实验
  • 提供 7+ 算法、34+ 游戏的对标基准结果

容易踩的坑

要求 Python >=3.7.1,<3.11 并使用 uv 0.7.9+；Atari、Procgen、envpool 等环境需额外安装对应可选依赖（如 uv pip install ".[atari]"）；它不是模块化库，设计上不供 import 复用

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

最近发版

5 个版本v1.0.0
全部 release
  1. v1.0.0v1.0.0 CleanRL Release 🎉最新正式版2022-11-14 · 2022-11-14 04:06 UTC

    🎉 We are thrilled to announce the v1.0.0 CleanRL Release. Along with our CleanRL paper's recent publication in Journal of Machine Learning Research, our v1.0.0 release includes reworked documentation, new algorithm variants, support for google's new ML framework JAX, hyperparameter tuning utilities, and more. CleanRL has come a long way making high-quality deep reinforcement learning implementations easy to understand and reproducible. This release is a major milestone for the project and we are excited to share it with you. Over 90 PRs were merged to make this release possible. We would like to thank all the contributors who made this release possible.

    Reworked documentation

    One of the biggest change of the v1 release is the added documentation at docs.cleanrl.dev. Having great documentation is important for building a reliable and reproducible project. We have reworked the documentation to make it easier to understand and use. For each implemented algorithm, we have documented as much as we can to promote transparency:

    Here is a list of the algorithm variants and their documentation:

    AlgorithmVariants Implemented
    ✅ Proximal Policy Gradient (PPO)ppo.py, docs
    ppo_atari.py, docs
    ppo_continuous_action.py, docs
    ppo_atari_lstm.py, docs
    ppo_atari_envpool.py, docs
    ppo_atari_envpool_xla_jax.py, docs
    ppo_procgen.py, docs
    ppo_atari_multigpu.py, docs
    ppo_pettingzoo_ma_atari.py, docs
    ppo_continuous_action_isaacgym.py, docs
    ✅ Deep Q-Learning (DQN)dqn.py, docs
    [dqn_atari.py](https://git
  2. v1.0.0b2v1.0.0b2 JAX Support and Hyperparameter Tuning2022-10-03 · 2022-10-03 19:41 UTC
    查看更新说明

    🎉 I am thrilled to announce the v1.0.0b2 CleanRL Beta Release. This new release comes with exciting new features. First, we now support JAX-based learning algorithms, which are usually faster than the torch equivalent! Here are the docs of the new JAX-based DQN, TD3, and DDPG implementations:

    Also, we now have preliminary support for hyperparameter tuning via optuna (see docs), which is designed to help researchers to find a single set of hyperparameters that work well with a kind of games. The current API looks like below:

    import optuna
from cleanrl_utils.tuner import Tuner
tuner = Tuner(
    script="cleanrl/ppo.py",
    metric="charts/episodic_return",
    metric_last_n_average_window=50,
    direction="maximize",
    aggregation_type="average",
    target_scores={
        "CartPole-v1": [0, 500],
        "Acrobot-v1": [-500, 0],
    },
    params_fn=lambda trial: {
        "learning-rate": trial.suggest_loguniform("learning-rate", 0.0003, 0.003),
        "num-minibatches": trial.suggest_categorical("num-minibatches", [1, 2, 4]),
        "update-epochs": trial.suggest_categorical("update-epochs", [1, 2, 4, 8]),
        "num-steps": trial.suggest_categorical("num-steps", [5, 16, 32, 64, 128]),
        "vf-coef": trial.suggest_uniform("vf-coef", 0, 5),
        "max-grad-norm": trial.suggest_uniform("max-grad-norm", 0, 5),
        "total-timesteps": 100000,
        "num-envs": 16,
    },
    pruner=optuna.pruners.MedianPruner(n_startup_trials=5),
    sampler=optuna.samplers.TPESampler(),
)
tuner.tune(
    num_trials=100,
    num_seeds=3,
)

    Besides, we added support for new algorithms/environments, which are

    I would like to cordially thank the core dev members @dosssman @yooceii @Dipamc @kinalmehta for their efforts in helping maintain the CleanRL repository. I would also like to give a shout-out to our new contributors @cool-RR, @Howuhh, @jseppanen, @joaogui1, @kinalmehta, and @ALPH2H.

    New CleanRL Supported Publications

    Jiayi Weng, Min Lin, Shengyi Huang, Bo Liu, Denys Makoviichuk, Viktor Makoviychuk, Zichen Liu, Yufan Song, Ting Luo, Yukun Jiang, Zhongwen Xu, & Shuicheng YAN (2022). EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Engine. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. https://openreview.net/forum?id=BubxnHpuMbG

    New Features PR

  3. v1.0.0b1v1.0.0b1 CleanRL Beta Release 🎉2022-06-07 · 2022-06-07 00:18 UTC
    查看更新说明

    🎉 I am thrilled to announce the v1.0.0b1 CleanRL Beta Release. CleanRL has come a long way making high-quality deep reinforcement learning implementations easy to understand. In this release, we have put a huge effort into revamping our documentation site, making our implementation friendly to use for new users.

    I would like to cordially thank the core dev members @dosssman @yooceii @Dipamc77 @bragajj for their efforts in helping maintain the CleanRL repository. I would also like to give a shout-out to our new contributors @ElliotMunro200 and @Dipamc77.

    New CleanRL supported publications

    New algorithm variants

    Refactoring changes

    Documentation changes

    A significant amount of documentation changes (tracked by https://github.com/vwxyzjn/cleanrl/issues/121).

    See the overview documentation page here: https://docs.cleanrl.dev/rl-algorithms/overview/

  4. v0.6.0v0.6.0 Major Refactoring2022-03-16 · 2022-03-16 15:07 UTC
    查看更新说明

    What's Changed

    Full Changelog: https://github.com/vwxyzjn/cleanrl/compare/v0.5.0...v0.6.0

  5. v0.5.02021-11-12 · 2021-11-12 16:01 UTC
    查看更新说明

    What's Changed

    New Contributors

    Full Changelog: https://github.com/vwxyzjn/cleanrl/compare/v0.4.8...v0.5.0

用户点评

登录用户的真实使用记录，可按版本和硬件筛选
共 0 条

    还没有人点评。第一个说出真实体验的人最有价值。

    README 原文

    未翻译
    在 GitHub 打开

    CleanRL (Clean Implementation of RL Algorithms)

    tests docs Code style: black Imports: isort Open In Colab

    CleanRL is a Deep Reinforcement Learning library that provides high-quality single-file implementation with research-friendly features. The implementation is clean and simple, yet we can scale it to run thousands of experiments using AWS Batch. The highlight features of CleanRL are:

    • 📜 Single-file implementation
      • Every detail about an algorithm variant is put into a single standalone file.
      • For example, our ppo_atari.py only has 340 lines of code but contains all implementation details on how PPO works with Atari games, so it is a great reference implementation to read for folks who do not wish to read an entire modular library.
    • 📊 Benchmarked Implementation (7+ algorithms and 34+ games at https://benchmark.cleanrl.dev)
    • 📈 Tensorboard Logging
    • 🪛 Local Reproducibility via Seeding
    • 🎮 Videos of Gameplay Capturing
    • 🧫 Experiment Management with Weights and Biases
    • 💸 Cloud Integration with docker and AWS

    You can read more about CleanRL in our JMLR paper and documentation.

    Notable CleanRL-related projects:

    • corl-team/CORL: Offline RL algorithm implemented in CleanRL style
    • pytorch-labs/LeanRL: Fast optimized PyTorch implementation of CleanRL RL algorithms using CUDAGraphs.

    ℹ️ Support for Gymnasium: Farama-Foundation/Gymnasium is the next generation of openai/gym that will continue to be maintained and introduce new features. Please see their announcement for further detail. We are migrating to gymnasium and the progress can be tracked in vwxyzjn/cleanrl#277.

    ⚠️ NOTE: CleanRL is not a modular library and therefore it is not meant to be imported. At the cost of duplicate code, we make all implementation details of a DRL algorithm variant easy to understand, so CleanRL comes with its own pros and cons. You should consider using CleanRL if you want to 1) understand all implementation details of an algorithm's variant or 2) prototype advanced features that other modular DRL libraries do not support (CleanRL has minimal lines of code so it gives you great debugging experience and you don't have do a lot of subclassing like sometimes in modular DRL libraries).

    Get started

    Prerequisites:

    To run experiments locally, give the following a try:

    git clone https://github.com/vwxyzjn/cleanrl.git && cd cleanrl
uv pip install .

# alternatively, you could use `uv venv` and do
# `python run cleanrl/ppo.py`
uv run python cleanrl/ppo.py \
    --seed 1 \
    --env-id CartPole-v0 \
    --total-timesteps 50000

# open another terminal and enter `cd cleanrl/cleanrl`
tensorboard --logdir runs

    To use experiment tracking with wandb, run

    wandb login # only required for the first time
uv run python cleanrl/ppo.py \
    --seed 1 \
    --env-id CartPole-v0 \
    --total-timesteps 50000 \
    --track \
    --wandb-project-name cleanrltest

    If you are not using uv, you can install CleanRL with requirements.txt:

    # core dependencies
pip install -r requirements/requirements.txt

# optional dependencies
pip install -r requirements/requirements-atari.txt
pip install -r requirements/requirements-mujoco.txt
pip install -r requirements/requirements-mujoco_py.txt
pip install -r requirements/requirements-procgen.txt
pip install -r requirements/requirements-envpool.txt
pip install -r requirements/requirements-pettingzoo.txt
pip install -r requirements/requirements-jax.txt
pip install -r requirements/requirements-docs.txt
pip install -r requirements/requirements-cloud.txt
pip install -r requirements/requirements-memory_gym.txt

    To run training scripts in other games:

    uv venv

# classic control
python cleanrl/dqn.py --env-id CartPole-v1
python cleanrl/ppo.py --env-id CartPole-v1
python cleanrl/c51.py --env-id CartPole-v1

# atari
uv pip install ".[atari]"
python cleanrl/dqn_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/c51_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/ppo_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/sac_atari.py --env-id BreakoutNoFrameskip-v4

# NEW: 3-4x side-effects free speed up with envpool's atari (only available to linux)
uv pip install ".[envpool]"
python cleanrl/ppo_atari_envpool.py --env-id BreakoutNoFrameskip-v4
# Learn Pong-v5 in ~5-10 mins
# Side effects such as lower sample efficiency might occur
uv run python ppo_atari_envpool.py --clip-coef=0.2 --num-envs=16 --num-minibatches=8 --num-steps=128 --update-epochs=3

# procgen
uv pip install ".[procgen]"
python cleanrl/ppo_procgen.py --env-id starpilot
python cleanrl/ppg_procgen.py --env-id starpilot

# ppo + lstm
uv pip install ".[atari]"
python cleanrl/ppo_atari_lstm.py --env-id BreakoutNoFrameskip-v4

    You may also use a prebuilt development environment hosted in Gitpod:

    Open in Gitpod

    Algorithms Implemented

    AlgorithmVariants Implemented
    ✅ Proximal Policy Optimization (PPO)ppo.py, docs
    ppo_atari.py, docs
    ppo_continuous_action.py, docs
    ppo_atari_lstm.py, docs
    ppo_atari_envpool.py, docs
    ppo_atari_envpool_xla_jax.py, docs
    ppo_atari_envpool_xla_jax_scan.py, docs)
    ppo_procgen.py, docs
    ppo_atari_multigpu.py, docs
    ppo_pettingzoo_ma_atari.py, docs
    ppo_continuous_action_isaacgym.py, docs
    ppo_trxl.py, docs
    ✅ Deep Q-Learning (DQN)dqn.py, docs
    dqn_atari.py, docs
    dqn_jax.py, docs
    dqn_atari_jax.py, docs
    ✅ Categorical DQN (C51)c51.py, docs
    c51_atari.py, docs
    c51_jax.py, docs
    c51_atari_jax.py, docs
    ✅ Soft Actor-Critic (SAC)sac_continuous_action.py, docs
    sac_atari.py, docs
    ✅ Deep Deterministic Policy Gradient (DDPG)ddpg_continuous_action.py, docs
    ddpg_continuous_action_jax.py, docs
    ✅ Twin Delayed Deep Deterministic Policy Gradient (TD3)td3_continuous_action.py, docs
    td3_continuous_action_jax.py, docs
    ✅ Phasic Policy Gradient (PPG)ppg_procgen.py, docs
    ✅ Random Network Distillation (RND)ppo_rnd_envpool.py, docs
    ✅ Qdaggerqdagger_dqn_atari_impalacnn.py, docs
    qdagger_dqn_atari_jax_impalacnn.py, docs

    Open RL Benchmark

    To make our experimental data transparent, CleanRL participates in a related project called Open RL Benchmark, which contains tracked experiments from popular DRL libraries such as ours, Stable-baselines3, openai/baselines, jaxrl, and others.

    Check out https://benchmark.cleanrl.dev/ for a collection of Weights and Biases reports showcasing tracked DRL experiments. The reports are interactive, and researchers can easily query information such as GPU utilization and videos of an agent's gameplay that are normally hard to acquire in other RL benchmarks. In the future, Open RL Benchmark will likely provide an dataset API for researchers to easily access the data (see repo).

    Support and get involved

    We have a Discord Community for support. Feel free to ask questions. Posting in Github Issues and PRs are also welcome. Also our past video recordings are available at YouTube

    Citing CleanRL

    If you use CleanRL in your work, please cite our technical paper:

    @article{huang2022cleanrl,
  author  = {Shengyi Huang and Rousslan Fernand Julien Dossa and Chang Ye and Jeff Braga and Dipam Chakraborty and Kinal Mehta and João G.M. Araújo},
  title   = {CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms},
  journal = {Journal of Machine Learning Research},
  year    = {2022},
  volume  = {23},
  number  = {274},
  pages   = {1--18},
  url     = {http://jmlr.org/papers/v23/21-1342.html}
}

    Acknowledgement

    CleanRL is a community-powered by project and our contributors run experiments on a variety of hardware.

    • We thank many contributors for using their own computers to run experiments
    • We thank Google's TPU research cloud for providing TPU resources.
    • We thank Hugging Face's cluster for providing GPU resources.

    相关项目

    同分类里 star 最高的项目
    对比全部 →
    transformers
    huggingface·Python

    统一加载和运行文本、视觉、音频、多模态的预训练模型，覆盖推理与训练，省去为每个模型重写代码的麻烦。

    预训练模型+4训练框架进阶
    168878+26635kv5.22.22026-11-11
    LlamaFactory
    hiyouga·Python

    一套统一接口搞定 100+ 大语言模型和多模态模型的微调，支持 LoRA 等高效训练方式，省去为每个模型单独写训练脚本的麻烦。

    微调框架+4训练框架进阶
    76004+819.4kv0.9.52026-11-11
    keras
    keras-team·Python

    面向人类友好的深度学习 API，用 Python 编写，可跑在 TensorFlow、JAX、PyTorch 上，帮开发者快速搭建和训练神经网络。

    深度学习+4训练框架进阶
    64422+620kv3.15.12026-11-11
    langextract
    google·Python

    用 LLM 从非结构化文本中抽取结构化信息，并给出精确的原文出处定位和交互式可视化，方便核对抽取结果。

    信息抽取+4训练框架进阶
    39303+292.8kv1.7.22026-11-10