stable-baselines3

DLR-RMMIT活跃维护

基于 PyTorch 的强化学习算法库,提供 PPO、SAC、TD3 等常用算法的可靠实现,开箱即用,适合机器人控制和仿真环境训练。

PyTorch version of Stable Baselines, reliable implementations of reinforcement learning algorithms.

v2.9.2发布于 2026-10-26 09:35 UTC13,999star +12 / 7 天2,183fork同步于 2026-11-10 11:27 UTC

怎么上手

由 README 提炼,执行前请确认
pip install 'stable-baselines3[extra]'
硬件要求按 README 说明

README 未说明具体显存或 GPU 型号要求;算法基于 PyTorch 实现,可运行于 CPU 或 GPU。

下载:源码 zip·GitHub Releases(安装包)本站不镜像安装包

项目说明

由 README 自动翻译提炼

PyTorch 版强化学习算法可靠实现库,提供稳定基线便于复现与对比

基于 PyTorch 的强化学习算法库,提供 PPO、SAC、TD3 等常用算法的可靠实现,开箱即用,适合机器人控制和仿真环境训练。

核心能力

  • 提供经过性能测试的多种 SOTA 强化学习算法实现
  • 统一接口支持自定义环境与自定义策略
  • 支持 Dict 观测空间与 Tensorboard 日志记录
  • 提供自定义回调机制并兼容 Notebook 交互使用
  • 代码符合 PEP8 规范并带类型提示与高测试覆盖率

容易踩的坑

需 Python 3.10+ 且 PyTorch >= 2.8;Windows 安装需参考官方文档的额外说明;实验性算法(如 Recurrent PPO、TQC)需另装 SB3-Contrib。

内容由模型根据 README 生成,可能不完整或不准确,以官方文档为准。看原文 README

最近发版

5 个版本v2.9.2
  1. v2.9.2v2.9.2: Bug fixes and doc improvements最新正式版2026-10-26 · 2026-10-26 09:35 UTC

    [!WARNING] Stable-Baselines3 (SB3) v2.9.2 will be the last one supporting Python 3.10 (end of life in October 2026) We highly recommended you to upgrade to Python >= 3.11.

    SB3 Contrib (more algorithms): https://github.com/Stable-Baselines-Team/stable-baselines3-contrib RL Zoo3 (training framework): https://github.com/DLR-RM/rl-baselines3-zoo Stable-Baselines Jax (SBX): https://github.com/araffin/sbx

    To upgrade:

    pip install stable_baselines3 sb3_contrib rl_zoo3 --upgrade
    

    Bug Fixes:

    • Fixed file descriptor leak in SubprocVecEnv.close() by closing parent-side pipes to prevent "too many open files" errors in long-running loops
    • Fixed Pandas deprecation warnings on inplace operations

    SB3-Contrib

    • Upgraded to Stable-Baselines3 >= 2.9.2

    RL Zoo

    • Set learning_starts to 500 for parking-v0 in TQC hyperparameters to ensure the first episode is finished
    • Fixed "too many open files" issue when doing hyperparameter optimization: the trial environments are now closed in a finally block and gc_after_trial=True is passed to Optuna to free them between trials (also needs latest SB3 master version to fully work)
    • Set use_expln=True for pybullet envs when using gSDE to avoid NaNs
    • Add a note regarding pybullet source installation error on macOS due to legacy code
    • Migrate packaging setup from setup.py to pyproject.toml and integrate requirements.txt as a new extra

    SBX (SB3 + Jax)

    • Updated Jax dependency version range

    Deprecations:

    Others:

    • Fixed moviepy test compatibility with moviepy >= 2.0 (PyTorch doesn't support it yet)
    • Added filter for numpy newshape deprecation warning from torch tensorboard
    • Fixed mypy error in test_vec_envs.py by wrapping itertools.product with list()
    • Improved tests coverage for RMSpropTFLike optimizer (invalid params, centered/momentum/weight_decay branches, pickle roundtrip)
    • Improved test coverage for save_util.py: added tests for BadZipFile error handling in load_from_zip_file and IsADirectoryError/FileNotFoundError handling in open_path
    • Fixed Docker build by adding the missing --system flag to uv pip uninstall opencv-python (required by recent uv)
    • Added a warning when n_steps > 1 is passed together with a custom replay_buffer_class, as n_steps is only applied when the replay buffer class is selected automatically (@Koustav-github)
    • Updated tests dependencies

    Documentation:

    • Clarified in the n_steps docstring that it only applies when replay_buffer_class is None, and that a custom replay buffer must be configured with replay_buffer_kwargs={"n_steps": ..., "gamma": ...} (@Koustav-github)
    • Updated algorithm table (added the ones from SBX and sorted the table)
    • Fixed typos, grammar, and Gymnasium API inconsistencies across documentation

    New Contributors

    Full Changelog: https://github.com/DLR-RM/stable-baselines3/compare/v2.9.0...v2.9.2

  2. v2.9.0v2.9.0: Updated dependencies (pandas is now optional, gymnasium 1.3.0 support, torch>=2.8)2026-06-15 · 2026-06-15 16:59 UTC
    查看更新说明

    Breaking Changes:

    • Relaxed Gymnasium version range (from "gymnasium>=0.29.1,<1.3.0" to "gymnasium>=0.29.1,<2.0")
    • pandas and matplotlib are no longer core dependencies; they are now optional and only required for loading results and plotting (moved to stable-baselines3[extra]).
    • Moved read_json and read_csv helper functions to test files
    • Raised torch minimum version from 2.3 to 2.8 to mitigate https://github.com/advisories/GHSA-887c-mr87-cxwp

    Bug Fixes:

    • Fixed deprecated error Taxi-v3 from gymnasium v1.3.0 in tests

    SB3-Contrib

    • Optimized tests (faster to run)
    • Fixed dead link for RecurrentPPO.

    RL Zoo

    SBX (SB3 + Jax)

    • Added support for rollout_buffer_class and rollout_buffer_kwargs arguments in PPO and OnPolicyAlgorithmJax constructors, as in Stable Baselines3. (@Trenza1ore)
    • Updated Jax dependency

    Others:

    • Optimized tests (faster to run)
    • Add workflow to automatically publish to PyPi when a new tag is created

    Documentation:

    • Added example for using torch.compile
    • Fixed many broken links and updated links to https whenever possible

    Full Changelog: https://github.com/DLR-RM/stable-baselines3/compare/v2.8.0...v2.9.0

  3. v2.8.0v2.8.0: Dropped Python 3.9, added Python 3.13 support, MaskablePPO bug fix, default hyperparams for unlisted env in the RL Zoo, Markdown doc2026-04-01 · 2026-04-01 11:32 UTC
    查看更新说明

    Breaking Changes:

    • Removed support for Python 3.9, please upgrade to Python >= 3.10
    • Set strict=True for every call to zip(...)
    • Switched to pygame-ce when installing extras

    New Features:

    • Added official support for Python 3.13

    Bug Fixes:

    • Fixed saving and loading of Torch compiled models (using th.compile()) by updating get_parameters()
    • Added a warning to env-checker if a multidiscrete space has multi-dimensional array (@unexploredtest)
    • Fixed pandas.concat futurewarnings occuring when dataframes are empty by removing empty frames from the list before concatenating

    SB3-Contrib

    • Set strict=True for every call to zip(...)
    • Fixed MaskablePPO and RecurrentPPO inaccurate n_updates counting when target_kl early exits the training loop
    • Fixed RecurrentPPO and MaskablePPO forward and predict not reshaping the action before clipping it (@immortal-boy)
    • Do not call forward() method directly in RecurrentPPO (@immortal-boy)
    • Fixed MaskableCategorical.apply_masking() crashing with ValueError: Simplex when cached probs deviate from sum=1 in float32 with large action spaces (torch 2.9+) (@kirann-05)
    • Switched to Markdown documentation (using MyST parser)

    RL Zoo

    • Set strict=True for every call to zip(...)
    • Allow to specify env_kwargs in the hyperparam config
    • Switched to Markdown documentation (using MyST parser)
    • Allow to use default hyperparameters for any environment
    • Save training command in Weights & Biases (Wandb)
    • Save training command and default hyperparameters as study attributes

    SBX (SB3 + Jax)

    • Increased Jax version range and use tf-nightly

    Deprecations:

    • zip_strict() is not needed anymore since Python 3.10, please use zip(..., strict=True) instead

    Others:

    • Updated to Python 3.10+ annotations
    • Removed some unused variables (@unexploredtest)
    • Improved type hints for distributions
    • Simplified zip file loading by removing Python 3.6 workaround and enabling weights_only=True (PyTorch 2.x)
    • Sped up saving/loading tests
    • Updated black from v25 to v26
    • Updated monitor test to check handling of empty monitor files

    Documentation:

    • Added a note on MultiDiscrete spaces with multi-dimensional arrays and a wrapper to fix the issue (@unexploredtest)
    • Added an example of manual export of SBX (SB3 + Jax) model to ONNX (@m-abr)
    • Switched to Markdown documentation (using MyST parser)

    New Contributors

    Full Changelog: https://github.com/DLR-RM/stable-baselines3/compare/v2.7.1...v2.8.0

  4. v2.7.1v2.7.1: Bug fixes and documentation improvements2025-12-05 · 2025-12-05 11:30 UTC
    查看更新说明

    [!WARNING] Stable-Baselines3 (SB3) v2.7.1 will be the last one supporting Python 3.9 (end of life in October 2025) We highly recommended you to upgrade to Python >= 3.10.

    SB3 Contrib (more algorithms): https://github.com/Stable-Baselines-Team/stable-baselines3-contrib RL Zoo3 (training framework): https://github.com/DLR-RM/rl-baselines3-zoo Stable-Baselines Jax (SBX): https://github.com/araffin/sbx

    To upgrade:

    pip install stable_baselines3 sb3_contrib rl_zoo3 --upgrade
    

    New Features:

    • RolloutBuffer and DictRolloutBuffer now uses the actual observation / action space dtype (instead of float32), this should save memory (@Trenza1ore)

    Bug Fixes:

    • Fixed env checker to properly handle Sequence observation spaces when nested inside composite spaces (Dict, Tuple, OneOf) (@copilot)
    • Update env checker to warn users when using Graph space (@dhruvmalik007).
    • Fixed memory leak in VecVideoRecorder where recorded_frames stayed in memory due to reference in the moviepy clip (@copilot)
    • Remove double space in StopTrainingOnRewardThreshold callback message (@sea-bass)

    SB3-Contrib

    • Fixed tensorboard log name for MaskablePPO

    RL Zoo

    SBX (SB3 + Jax)

    • Added CnnPolicy to PPO

    Documentation:

    • Added plotting documentation and examples
    • Added documentation clarifying gSDE (Generalized State-Dependent Exploration) inference behavior for PPO, SAC, and A2C algorithms
    • Documented Atari wrapper reset behavior where env.reset() may perform a no-op step instead of truly resetting when terminal_on_life_loss=True (default), and how to avoid this behavior by setting terminal_on_life_loss=False
    • Clarified comment in _sample_action() method to better explain action scaling behavior for off-policy algorithms (@copilot)
    • Added sb3-plus to projects page
    • Added example usage of ONNX JS
    • Updated link to paper of community project DeepNetSlice (@AlexPasqua)
    • Added example usage of Tensorflow JS
    • Included exact versions in ONNX JS and example project
    • Made step 2 (pip install) of CONTRIBUTING.md more robust

    New Contributors

    Full Changelog: https://github.com/DLR-RM/stable-baselines3/compare/v2.7.0...v2.7.1

  5. v2.7.0v2.7.0: n-step returns for all off-policy algorithms via the `n_steps` argument2025-07-25 · 2025-07-25 09:55 UTC
    查看更新说明

    SB3 Contrib (more algorithms): https://github.com/Stable-Baselines-Team/stable-baselines3-contrib RL Zoo3 (training framework): https://github.com/DLR-RM/rl-baselines3-zoo Stable-Baselines Jax (SBX): https://github.com/araffin/sbx

    To upgrade:

    pip install stable_baselines3 sb3_contrib rl_zoo3 --upgrade
    

    New Features:

    • Added support for n-step returns for off-policy algorithms via the n_steps parameter
    from stable_baselines3 import SAC
    
    # SAC with n-step returns
    model = SAC("MlpPolicy", "Pendulum-v1", n_steps=3, verbose=1)
    model.learn(10_000)
    
    • Added NStepReplayBuffer that allows to compute n-step returns without additional memory requirement (and without for loops)
    • Added Gymnasium v1.2 support

    Bug Fixes:

    • Fixed docker GPU image (PyTorch GPU was not installed)
    • Fixed segmentation faults caused by non-portable schedules during model loading (@akanto)

    SB3-Contrib

    • Added support for n-step returns for off-policy algorithms via the n_steps parameter
    • Use the FloatSchedule and LinearSchedule classes instead of lambdas in the ARS, PPO, and QRDQN implementations to improve model portability across different operating systems

    RL Zoo

    • linear_schedule now returns a SimpleLinearSchedule object for better portability
    • Renamed LunarLander-v2 to LunarLander-v3 in hyperparameters
    • Renamed CarRacing-v2 to CarRacing-v3 in hyperparameters
    • Docker GPU images are now working again
    • Use ConstantSchedule, and SimpleLinearSchedule instead of constant_fn and linear_schedule
    • Fixed CarRacing-v3 hyperparameters for newer Gymnasium version

    SBX (SB3 + Jax)

    • Added support for n-step returns for off-policy algorithms via the n_steps parameter
    • Added KL Adaptive LR for PPO and LR schedule for SAC/TQC

    Deprecations:

    • get_schedule_fn(), get_linear_fn(), constant_fn() are deprecated, please use FloatSchedule(), LinearSchedule(), ConstantSchedule() instead

    Documentation:

    • Clarify evaluate_policy documentation
    • Added doc about training exceeding the total_timesteps parameter
    • Updated LunarLander and LunarLanderContinuous environment versions to v3 (@j0m0k0)
    • Added sb3-extra-buffers to the project page (@Trenza1ore)

    New Contributors

    Full Changelog: https://github.com/DLR-RM/stable-baselines3/compare/v2.6.0...v2.7.0

用户点评

登录用户的真实使用记录,可按版本和硬件筛选
共 0 条

    还没有人点评。第一个说出真实体验的人最有价值。

    README 原文

    未翻译

    CI Documentation Status coverage report codestyle

    Stable Baselines3

    Stable Baselines3 (SB3) is a set of reliable implementations of reinforcement learning algorithms in PyTorch. It is the next major version of Stable Baselines.

    You can read a detailed presentation of Stable Baselines3 in the v1.0 blog post or our JMLR paper.

    These algorithms will make it easier for the research community and industry to replicate, refine, and identify new ideas, and will create good baselines to build projects on top of. We expect these tools will be used as a base around which new ideas can be added, and as a tool for comparing a new approach against existing ones. We also hope that the simplicity of these tools will allow beginners to experiment with a more advanced toolset, without being buried in implementation details.

    Note: Despite its simplicity of use, Stable Baselines3 (SB3) assumes you have some knowledge about Reinforcement Learning (RL). You should not utilize this library without some practice. To that extent, we provide good resources in the documentation to get started with RL.

    Main Features

    The performance of each algorithm was tested (see Results section in their respective page). You can take a look at the issues #48 and #49 for more details.

    We also provide detailed logs and reports on the OpenRL Benchmark platform.

    FeaturesStable-Baselines3
    State of the art RL methods:heavy_check_mark:
    Documentation:heavy_check_mark:
    Custom environments:heavy_check_mark:
    Custom policies:heavy_check_mark:
    Common interface:heavy_check_mark:
    Dict observation space support:heavy_check_mark:
    Ipython / Notebook friendly:heavy_check_mark:
    Tensorboard support:heavy_check_mark:
    PEP8 code style:heavy_check_mark:
    Custom callback:heavy_check_mark:
    High code coverage:heavy_check_mark:
    Type hints:heavy_check_mark:

    Planned features

    Since most of the features from the original roadmap have been implemented, there are no major changes planned for SB3; it is now stable. If you want to contribute, you can search in the issues for the ones where help is welcomed and the other proposed enhancements.

    While SB3 development is now focused on bug fixes and maintenance (doc update, user experience, ...), there is more active development going on in the associated repositories:

    • newer algorithms are regularly added to the SB3 Contrib repository
    • faster variants are developed in the SBX (SB3 + Jax) repository
    • the training framework for SB3, the RL Zoo, has an active roadmap

    Migration guide: from Stable-Baselines (SB2) to Stable-Baselines3 (SB3)

    A migration guide from SB2 to SB3 can be found in the documentation.

    Documentation

    Documentation is available online: https://stable-baselines3.readthedocs.io/

    Integrations

    Stable-Baselines3 has some integration with other libraries/services like Weights & Biases for experiment tracking or Hugging Face for storing/sharing trained models. You can find out more in the dedicated section of the documentation.

    RL Baselines3 Zoo: A Training Framework for Stable Baselines3 Reinforcement Learning Agents

    RL Baselines3 Zoo is a training framework for Reinforcement Learning (RL).

    It provides scripts for training, evaluating agents, tuning hyperparameters, plotting results and recording videos.

    In addition, it includes a collection of tuned hyperparameters for common environments and RL algorithms, and agents trained with those settings.

    Goals of this repository:

    1. Provide a simple interface to train and enjoy RL agents
    2. Benchmark the different Reinforcement Learning algorithms
    3. Provide tuned hyperparameters for each environment and RL algorithm
    4. Have fun with the trained agents!

    Github repo: https://github.com/DLR-RM/rl-baselines3-zoo

    Documentation: https://rl-baselines3-zoo.readthedocs.io/en/master/

    SB3-Contrib: Experimental RL Features

    We implement experimental features in a separate contrib repository: SB3-Contrib

    This allows SB3 to maintain a stable and compact core, while still providing the latest features, like Recurrent PPO (PPO LSTM), CrossQ, Truncated Quantile Critics (TQC), Quantile Regression DQN (QR-DQN) or PPO with invalid action masking (Maskable PPO).

    Documentation is available online: https://sb3-contrib.readthedocs.io/

    Stable-Baselines Jax (SBX)

    Stable Baselines Jax (SBX) is a proof of concept version of Stable-Baselines3 in Jax, with recent algorithms like DroQ or CrossQ.

    It provides a minimal number of features compared to SB3 but can be much faster (up to 20x times!): https://twitter.com/araffin2/status/1590714558628253698

    Installation

    Note: Stable-Baselines3 supports PyTorch >= 2.8

    Prerequisites

    Stable Baselines3 requires Python 3.10+.

    Windows

    To install stable-baselines on Windows, please look at the documentation.

    Install using pip

    Install the Stable Baselines3 package:

    pip install 'stable-baselines3[extra]'
    

    This includes optional dependencies like Tensorboard, OpenCV, ale-py to train on atari games, as well as pandas and matplotlib for plotting and analyzing results. If you do not need those, you can use:

    pip install stable-baselines3
    

    Please read the documentation for more details and alternatives (from source, using docker).

    Example

    Most of the code in the library tries to follow a sklearn-like syntax for the Reinforcement Learning algorithms.

    Here is a quick example of how to train and run PPO on a cartpole environment:

    import gymnasium as gym
    
    from stable_baselines3 import PPO
    
    env = gym.make("CartPole-v1", render_mode="human")
    
    model = PPO("MlpPolicy", env, verbose=1)
    model.learn(total_timesteps=10_000)
    
    vec_env = model.get_env()
    obs = vec_env.reset()
    for i in range(1000):
        actions, _states = model.predict(obs, deterministic=True)
        obs, rewards, dones, info = vec_env.step(actions)
        vec_env.render()
        # VecEnv resets automatically
        # if done:
        #   obs = env.reset()
    
    env.close()
    

    Or just train a model with a one liner if the environment is registered in Gymnasium and if the policy is registered:

    from stable_baselines3 import PPO
    
    model = PPO("MlpPolicy", "CartPole-v1").learn(10_000)
    

    Please read the documentation for more examples.

    Try it online with Colab Notebooks!

    All the following examples can be executed online using Google Colab notebooks:

    Implemented Algorithms

    NameRecurrentBoxDiscreteMultiDiscreteMultiBinaryMulti Processing
    A2C:x::heavy_check_mark::heavy_check_mark::heavy_check_mark::heavy_check_mark::heavy_check_mark:
    ARS1:x::heavy_check_mark::heavy_check_mark::x::x::heavy_check_mark:
    CrossQ12:x::heavy_check_mark::x::x::x::heavy_check_mark:
    DDPG:x::heavy_check_mark::x::x::x::heavy_check_mark:
    DQN:x::x::heavy_check_mark::x::x::heavy_check_mark:
    DroQ2:x::heavy_check_mark::x::x::x::heavy_check_mark:
    HER:x::heavy_check_mark::heavy_check_mark::x::x::heavy_check_mark:
    Maskable PPO1:x::x::heavy_check_mark::heavy_check_mark::heavy_check_mark::heavy_check_mark:
    PPO:x::heavy_check_mark::heavy_check_mark::heavy_check_mark::heavy_check_mark::heavy_check_mark:
    QR-DQN1:x::x::heavy_check_mark::x::x::heavy_check_mark:
    RecurrentPPO1:heavy_check_mark::heavy_check_mark::heavy_check_mark::heavy_check_mark::heavy_check_mark::heavy_check_mark:
    SAC:x::heavy_check_mark::x::x::x::heavy_check_mark:
    SAC-N2:x::heavy_check_mark::x::x::x::heavy_check_mark:
    SimBa2:x::heavy_check_mark::x::x::x::heavy_check_mark:
    TD3:x::heavy_check_mark::x::x::x::heavy_check_mark:
    TQC1:x::heavy_check_mark::x::x::x::heavy_check_mark:
    TRPO1:x::heavy_check_mark::heavy_check_mark::heavy_check_mark::heavy_check_mark::heavy_check_mark:

    1: Implemented in SB3 Contrib GitHub repository.

    2: Implemented in SBX (SB3 + Jax)

    Actions gymnasium.spaces:

    • Box: A N-dimensional box that contains every point in the action space.
    • Discrete: A list of possible actions, where only one action can be used per timestep.
    • MultiDiscrete: A list of possible actions, where each timestep only one action of each discrete set can be used.
    • MultiBinary: A list of possible actions, where each timestep any of the actions can be used in any combination.

    Testing the installation

    Install dependencies

    pip install -e '.[docs,tests,extra]'
    

    Run tests

    All unit tests in stable baselines3 can be run using pytest runner:

    make pytest
    

    To run a single test file:

    python3 -m pytest -v tests/test_env_checker.py
    

    To run a single test:

    python3 -m pytest -v -k 'test_check_env_dict_action'
    

    You can also do a static type check using mypy:

    pip install mypy
    make type
    

    Codestyle check with ruff:

    pip install ruff
    make lint
    

    Projects Using Stable-Baselines3

    We try to maintain a list of projects using stable-baselines3 in the documentation, please tell us if you want your project to appear on this page ;)

    Citing the Project

    To cite this repository in publications:

    @article{stable-baselines3,
      author  = {Antonin Raffin and Ashley Hill and Adam Gleave and Anssi Kanervisto and Maximilian Ernestus and Noah Dormann},
      title   = {Stable-Baselines3: Reliable Reinforcement Learning Implementations},
      journal = {Journal of Machine Learning Research},
      year    = {2021},
      volume  = {22},
      number  = {268},
      pages   = {1-8},
      url     = {https://jmlr.org/papers/v22/20-1364.html}
    }
    

    Note: If you need to refer to a specific version of SB3, you can also use the Zenodo DOI.

    Maintainers

    Stable-Baselines3 is currently maintained by Ashley Hill (aka @hill-a), Antonin Raffin (aka @araffin), Maximilian Ernestus (aka @ernestum), Adam Gleave (@AdamGleave), Anssi Kanervisto (@Miffyli) and Quentin Gallouédec (@qgallouedec).

    Important Note: We do not provide technical support, or consulting and do not answer personal questions via email. Please post your question on the RL Discord, Reddit, or Stack Overflow in that case.

    How To Contribute

    For anyone interested in making the baselines better, there is still some documentation that needs to be done. If you want to contribute, please read CONTRIBUTING.md guide first.

    Acknowledgments

    The initial work to develop Stable Baselines3 was partially funded by the project Reduced Complexity Models from the Helmholtz-Gemeinschaft Deutscher Forschungszentren, and by the EU Horizon 2020 Research and Innovation Programme under grant number 951992 (VeriDream).

    The original version, Stable Baselines, was created in the robotics lab U2IS (INRIA Flowers team) at ENSTA ParisTech.

    Logo credits: L.M. Tenkes