mini-verl

verl for a single consumer GPU.

Run PPO, GRPO and on-policy distillation on one NVIDIA GPU. Bring your verl configs, train locally, pick up interrupted runs and export your models.

PyPI v0.16.0 is stable; main is development.

Your first local experiment

Install the matching CUDA-enabled build using the PyTorch installer, then:

python -m pip install "miniverl[train,hydra]"
miniverl data sample --reward-profile target-length --rows 8 --out data/rl-prompts.parquet
miniverl run --example hydra-ppo --bind reward.provider=target_length --dry-run
miniverl run --example hydra-ppo --bind reward.provider=target_length --run-id local-ppo

This Qwen3-0.6B example trains with a simple length reward. Inspect it in rewards.jsonl, or follow the five-minute workflow to use GRPO, resume training and export an adapter.

Already have a verl experiment? Bring its config tree and Hydra overrides:

miniverl run --verl-config-path /path/to/verl/trainer/config \
  --verl-config-name ppo_trainer --dry-run \
  algorithm.adv_estimator=grpo actor_rollout_ref.rollout.n=8 trainer.n_gpus_per_node=8

The direct-config guide walks through binding your data and reward function. The single-GPU guide helps you choose a model and memory settings. Install extras add training, Hydra or quantization dependencies; choose the CUDA PyTorch build separately.

Choose your path

You want to…	Start here	Take away
Train with rewards	PPO / GRPO walkthrough	A trained adapter, reward logs and resumable checkpoints
Distill a teacher	OPD quickstart	A student trained on its own rollouts with teacher feedback
Move to a larger setup	Export and hand off	PEFT, Parquet and config files, plus a report of remaining setup

SFT, DPO and offline KD are also available through native recipes. Hardware planning helps fit a workload to your GPU.

What a run gives you

• Familiar configs. Reuse verl fields and overrides; inspect how each maps to local execution.
• Visible training. Follow rollouts, rewards, losses and teacher targets.
• Exact recovery. Resume model, optimizer, data position and random state from a checkpoint.
• Reusable outputs. Export adapters, datasets and the configuration and source identities behind the run.

How it works

Actor, critic, reference, reward and teacher roles share the GPU in phases. PPO trains an actor and a separate critic; GRPO uses grouped rewards; OPD learns from teacher targets on the student's current rollouts.

Measured results

On an RTX 4080, the Qwen3-0.6B/1.7B OPD workload completed eight updates over 32 distinct prompts at 3.1914 GiB peak reserved VRAM. An interruption after update four reproduced byte-identical trajectories, adapter and optimizer tensors on resume. The SmolLM2-360M/1.7B workload used 1.4961 GiB. Workload and measurements · 24-cell rollout-backend study.

For task outcomes, read the calculator study, RecoveryBench, Alignment Lab and External Alignment Gate. These include negative and mixed results, with their original data and scope.

About and compatibility

miniVERL is an independent Apache-2.0 project for single-GPU NVIDIA CUDA training. It supports documented verl configuration profiles; model fit depends on GPU memory and workload. See the compatibility matrix for supported upstream versions and semantics, and limitations for hardware coverage, scientific scope and the distributed-execution boundary.

Development

git clone https://github.com/DaoyuanLi2816/mini-verl.git
cd mini-verl
python -m pip install -e ".[dev,train]"
pytest -q -m "not gpu and not network"

Contributing · Changelog · Citation · Security · License