DiffusionART

News

• 📰 [2026.10.30] Diffusion-ART is reported by 机器之心.

• ⚡ [2026.10.25] Diffusion-ART-Flash is supported. Training only high-noise timesteps achieves comparable performance to all-timesteps training with substantially less training time cost.

• 🎉 [2026.10.22] MiniMax-H3 is now supported for text-to-audio/video (omni) training, with DiffusionART post-training recipes.

• 🥇 [2026.10.12] DiffusionART ranked #1 on Hugging Face Daily Papers!

• 2026.10.17: ART now splits each prompt's trajectories evenly between its two update windows, retaining all prompts in each update to reduce gradient variance. Guidance still uses the full K-sample group.

• 2026.10.14: Fixed Hunyuan recipes that defaulted to rollout/update microbatches of 1 instead of 8 (ReFL remains 1/1). Microbatch size can strongly affect GPU utilization and training throughput. Use commit 0fb23a7 or a later version containing this fix. Smaller microbatches remain supported for memory-constrained devices; see batch-size guidance.

---

DiffusionART is an on-policy diffusion RL algorithm via Aggregation-First Reward Tilting with Adaptive Restoration. It combines the advantages of forward-process and reverse-process reinforcement learning while avoiding their limitations. DiffusionART is highly efficient, stable and simple, requires minimal hyperparameter tuning and no differentiable reward model, naturally supports pairwise feedback, and is robust to small rollout groups for scalable diffusion reinforcement learning.

This repository provides a comprehensive framework for fine-tuning visual generation models, supporting widely used open-source base models, reward models and state-of-the-art algorithms across text-to-image, text-to-video, image-to-video, and text-to-audio/video generation.

Method

Motivations and Insights

1. NFT-style forward-process RL contains a hidden forward-reverse consistency force in its objective, slowing and destabilizing training.
2. Off-policy strategies stabilize NFT-style diffusion RL through a strong restoration force. Even a small output-space restoration MSE can produce a large aggregated parameter gradient.

Aggregation-First Guidance and Adaptive Restoration

DiffusionART is simple: roll out K samples per prompt with an ODE solver, compute the reward-endpoint covariance as guidance, apply it to the visited states and actions, then perform two disjoint minibatch updates. We call this reward-improving direction guidance, bringing a classifier-free guidance (CFG)-like idea into diffusion RL: the model learns to internalize reward guidance, much like distilling CFG. For a reward-tilted endpoint distribution with tilt strength eta,

Here c is a prompt, x₀ is a clean latent endpoint, and R is its reward. DiffusionART systematically removes the self-consistency force: it aggregates reward evidence before applying the normalized guidance to reverse-process rollout states, without relying on an ideal forward-process velocity field. The second update retains guidance while adaptively restoring velocity drift toward the rollout policy, without an EMA off-policy strategy.

DiffusionART-Pairwise

DiffusionART-Pairwise is a variant in which the reward model gives preference scores for two visual outputs: 1 for the winner, 0 for the loser, and 0.5 for a tie. It accommodates pairwise feedback by simply replacing scalar rewards with these labels when computing the shared guidance. See the preference-label and pairwise-guidance implementation.

DiffusionART-Flash

DiffusionART-Flash trains on high-noise timesteps. For video tasks, Our recommended recipe is to randomly select 2 timestep transitions for noise level between 0.95 to 1 (very high-noise region) at training stage. For our Hunyuan-Video experiment, we randomly selects two of the first four timesteps for each rollout. It processes far fewer sample–timestep pairs during policy updates while achieving comparable performance to training across the full denoising schedule. Rollout stage still follows the complete trajectory to guarantee the sample quality. Results:

Enable with training.timestep_selection: high_noise. See the Flash guide and example configs.

Note: The idea of high-noise selection is similar to Flow-GRPO-Fast vs Flow-GRPO. However, we found that DiffusionART-Flash is not sensitive to the timestep selection, while Flow-GRPO-Fast does, in video tasks. This is because DiffusionART significantly mitigates the credit-assignment issue, compared with conventional reverse-process RL. To the extreme case, only training the first timestep (t=1) in DiffusionART-Flash is already sufficient in terms of the convergence speed and stability.

Key Advantages

• ✅ ODE solver: deterministic rollouts, with no reverse-transition likelihood.
• ✅ Rollout efficiency: reuse rollout states and cached predictions, without adding noise again or recomputing predictions on newly noised states.
• ✅ Sample efficiency: fewer samples per prompt with minimal performance loss and lower GPU-hour cost.
• ✅ Training efficiency: remove forward-process RL's unstable self-consistency force and address reverse-process RL's inefficient credit assignment.
• ✅ Simple and scalable: minimal hyperparameter tuning, no differentiable rewards required, and natural support for pairwise reward models.

Visualization Results

Released Models

Task	Model	Release
Text-to-image	SD3.5-M · Multi-Reward	Hugging Face
Text-to-video	HunyuanVideo-13B	Hugging Face

Each model card includes its inference setup, base-model requirements and examples.

Quick Feature Summary

Feature Category	Supported Capabilities
Tasks	✅ Text-to-image✅ Text-to-video✅ Image-to-video✅ Text-to-audio/video
Algorithms	✅ DiffusionART / DiffusionART-Flash✅ DiffusionNFT (EMA / on-policy)✅ Flow-GRPO / Flow-GRPO-Fast✅ Flow-DPPO✅ DanceGRPO✅ ReFL (video)
Base Models	✅ SD3.5-M✅ FLUX.1-dev✅ HunyuanVideo-13B✅ Wan2.1-I2V-14B (CFG/step-distilled)✅ MiniMax-H3 (audio/video)
Reward Models	✅ GenEval / PickScore / OCR✅ CLIPScore / Aesthetic✅ VideoReward (TA / VQ / MQ)✅ VideoAlign / HPSv3 / AudioBox / CLAP / DeSync✅ ImageBind + AV post-training✅ Configurable reward weights
Feedback	✅ Scalar rewards✅ Pairwise preferences (Hunyuan ART)✅ Configurable rollout groups: K = 2, 4, 8
Training	✅ LoRA / full-parameter tuning✅ DDP / FSDP✅ Checkpoint resume / local evaluation artifacts

Quick Start

1. Install

Use Python 3.10+ and uv. Pretrained training requires Linux and CUDA.

git clone https://github.com/vanzll/DiffusionART.git
cd DiffusionART
uv sync --locked --python 3.10 --extra diffusers

No GPU, model downloads or W&B account needed:

uv run --locked visual-rl --config configs/examples/cpu_smoke.yaml --validate
uv run --locked visual-rl --config configs/examples/cpu_smoke.yaml

outputs/cpu-smoke/status.json should report completed and four optimizer steps. To repeat, pass --output-dir outputs/cpu-smoke-2.

2. Configure Access

Accept the model license on Hugging Face, then configure HF_TOKEN and WANDB_API_KEY, or use existing Hub/W&B logins. Paper recipes enable W&B by default. Set VISUAL_RL_ASSET_ROOT to choose a writable asset cache. Videos use the pinned video environment.

3. Train

Run the complete SD3 PickScore experiment on eight GPUs:

uv run --locked --extra diffusers visual-rl \
  --config configs/paper/image/sd3_pickscore_art.yaml --nproc-per-node 8

The command prepares pinned models and prompt data before starting 1000 training updates. Missing models are downloaded from Hugging Face automatically; existing assets are reused. No manual model download is required for this recipe. GenEval and OCR also download benchmark prompts from our HF dataset. First complete the GenEval scorer setup or OCR scorer setup, then use the full training command in that guide. See all image recipes for other algorithms.

Resume the latest complete checkpoint, optionally extending the total budget:

uv run --locked --extra diffusers visual-rl \
  --config configs/paper/image/sd3_pickscore_art.yaml --nproc-per-node 8 \
  --resume latest --max-updates 1500

Video recipes use the same entrypoint inside the video environment. For shared multi-node preparation, generate a fresh launch ID once:

python -c 'import uuid; print(uuid.uuid4().hex)'

Export that same value as PREPARATION_RUN_ID on every node. This lets peers reuse one full asset validation within this launch; use a new ID on restart. T2V (400 updates):

visual-rl --config configs/paper/video/hunyuan_art_721.yaml \
  --nproc-per-node 8 --nnodes 4 --node-rank "$NODE_RANK" \
  --master-addr "$MASTER_ADDR" --preparation-run-id "$PREPARATION_RUN_ID" \
  --rendezvous-timeout-seconds 7200

I2V (400 updates, automatically generated evaluation references):

visual-rl --config configs/examples/wan_art_online_eval.yaml \
  --nproc-per-node 8 --nnodes 4 --node-rank "$NODE_RANK" \
  --master-addr "$MASTER_ADDR" --preparation-run-id "$PREPARATION_RUN_ID" \
  --rendezvous-timeout-seconds 7200

Run on each node with its rank and a shared asset/output root. Models and prompt data are prepared from HF automatically. I2V generates FLUX evaluation images on first use and caches them for later evaluations and resume; no pre-existing image manifest is needed. See the I2V guide for baseline recipes and shared reference caches. Add --prepare-only to prepare without training, or --validate for network-free checks. See distributed execution for multi-node launches and checkpoint/resume for recovery and evaluation.

MiniMax-H3

We provide a recipe for fine-tuning MiniMax-H3 to support future research, using six rewards for audio/video post-training at 480P, ~5 seconds, 10 denoising steps. Although the official model release recommends 50 denoising steps, we have found 10 steps sufficient for training. However, resolutions below 480P significantly degrade rollout quality and training performance. With sufficient computational resources, we recommend the official 768P resolution.

We use DiffusionART-Flash to accelerate training by randomly selecting 2 of the first 4 video steps (noise level σ ≈ 0.966–1) and 2 of the first 3 audio steps (σ ≈ 0.923–1) for training in each rollout.

Each reward is centered and divided by its own prompt-group standard deviation, then routed:

• Video: VideoAlign + 1.5 HPSv3 + DeSync + 0.5 ImageBind.
• Audio: 0.5 AudioBox + CLAP + DeSync + 0.5 ImageBind.

The curves above use Full64 (CP2). Full32 uses a smaller update microbatch.

Note: Open-source reward models cannot fully capture human preferences for visual outputs. As a result, reward scores can improve even when the generated videos remain visually unsatisfactory. For research purposes, we suggest using reward scores as the primary evaluation metric; for industrial applications, we recommend using higher-quality reward models.

See setup and launch instructions:

GPUs	Training	Rollout / update microbatch	Recipe
32	Full parameters	8 / 4	Full32
64	Full parameters, CP2	8 / 8	Full64
8	LoRA (rank 32)	1 / 1	LoRA8

Configure Your Experiment

Extend an example or paper recipe. Save this as configs/my-pickscore.yaml:

extends: examples/sd3_pickscore.yaml
name: my-pickscore
training:
  max_updates: 1000
runtime:
  output_dir: outputs/my-pickscore
  checkpoint_keep: 2

uv run --locked --extra diffusers visual-rl --config configs/my-pickscore.yaml

Select components through model.name, algorithm.name and reward.name; compose objectives with reward.weights. Sampling, microbatches, evaluation and checkpoint cadence live in the same file. Required asset paths use ${ENV_VAR}.

Metrics are saved to metrics.jsonl; evaluation images, videos and scores stay on disk. Set runtime.log: wandb for scalar logging.

Implementation Notes and Discussion

Rollout diversity and sampler compatibility. ART's frozen-target regression also applies to SDE transition means. For an affine mean a·vθ + b, where a is nonzero and a and b are independent of θ, matching cached states and corresponding frozen targets gives the same per-transition gradient direction, with magnitude scaled by a². Dividing each mean-space MSE by a² recovers the velocity-space gradient when timestep weights, restoration and masks match. Without normalization, timestep-dependent scaling can also change the aggregate gradient direction. SDE sampling changes rollout diversity and distribution; current public ART recipes use ODE rollouts.

Covariance normalization. Normalize the aggregated covariance before constructing the regression target. This controls the guidance scale independently of the raw covariance magnitude. For multiple rewards, combine their weighted contributions before normalization; normalizing each contribution separately changes their relative influence.

Prompt diversity and batch size. When compute allows, we recommend covering more distinct prompts per rollout. In our experience, broader prompt coverage often reduces gradient variance and improves training stability. Increase sampling.prompts_per_rollout to adjust this budget; keep the intended number of samples per prompt and optimizer updates per rollout explicit.

ART computes guidance from each complete group of K trajectories before splitting the training data. Both disjoint update batches contain K/2 trajectories from every prompt: for example, two batches of 32 prompts with K/2 trajectories each, rather than 16 prompts with K trajectories each. Two-update ART requires even K. Each trajectory keeps all its selected training timesteps; zero-guidance and masked rows remain in the original loss denominator. Images use a random split within each prompt; videos retain trajectory order. This changes older image splits and multi-prompt-per-rank video splits, not the historical video split with one prompt per rank. It is not a claim that this split is empirically optimal.

With limited memory, reduce microbatch size, not the logical update batch. Lower sampling.microbatch to generate the same rollout samples in more sequential chunks. Lower training.microbatch to accumulate gradients over more chunks within the same update window; the runner handles this automatically without changing the loss denominator or adding optimizer steps. Use the largest microbatches that fit comfortably for better GPU utilization and throughput. Shrinking the minibatch/update window instead creates more optimizer updates per rollout and additional opportunities for off-policy restoration, changing the algorithm's training dynamics rather than merely saving memory.

Reuse rollout predictions. ART trains on cached ODE trajectory states and reuses their old-policy predictions, avoiding the additional old-policy forward pass on re-noised states required by our NFT implementation. Current-policy forward and backward passes are still required. This reduces per-rollout compute; the overall speedup also depends on the number of updates needed to reach a target reward.

Reward quality matters. As with ReFL and other reward-based fine-tuning methods, DiffusionART depends on how well the reward model captures the desired behavior. Our experiments use open-source reward models. For practical applications, we recommend higher-quality reward models better aligned with the intended use case: DiffusionART improves rewards quickly, which can also expose weaknesses in the reward model and lead to reward hacking sooner.

In our experiments, when different algorithms reach a comparable score on an optimized dimension, they tend to exhibit similar levels of degradation in other dimensions. This suggests a shared limitation of the reward model rather than a failure unique to DiffusionART.

Learn More

FAQ: models, black-box rewards, pairwise feedback and on-policy training

Documentation  /  Development  /  Model interfaces  /  Reward interfaces

Acknowledgements

Thanks to the authors of Flow-GRPO, DiffusionNFT, DanceGRPO and UniRL for open-sourcing their work.

License

The code is released under the Apache License 2.0. Third-party code, model weights and datasets retain their respective licenses; see Third-Party Notices.

Citation

If you find DiffusionART inspiring or useful for your projects, please consider starring this repository and citing our paper:

@misc{wan2026diffusionartefficientstablescalable,
  title={DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models},
  author={Zhenglin Wan and Jiayu Chen and Zhengfeiyang Zhang and Pengfei Zhou and Wangbo Zhao and Fangyuan Kong and Pengfei Wan and Kun Gai and Yang You},
  year={2026},
  eprint={2610.12572},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2610.12572},
}

Star History