R1-VL

Jingyi Zhang1, Jiaxing Huang1, Huanjin Yao2, Shunyu Liu1, Xikun Zhang1, Shijian Lu1, Dacheng Tao1

1Nanyang Technological University, 2Tsinghua University

News

• June 25, 2025. R1-VL is accepted by ICCV 2025! 🎉
• May 22, 2025. Our new work R1-ShareVL is out! It achieves SOTA on six benchmarks by exploring and sharing diverse reasoning trajectories over an expanded question space, effectively mitigating sparse reward and advantage vanishing issues. Check out our paper.
• April 30, 2025. We release the training code for RL stage.
• April 16, 2025. We release the training details for SFT warm-up stage.
• March 22, 2025. We release our models R1-VL-7B and R1-VL-2B.
• March 17, 2025. We release our paper on arxiv.

Abstract

Recent studies generally enhance MLLMs' reasoning capabilities via supervised fine-tuning on high-quality chain-of-thought reasoning data, which often leads models to merely imitate successful reasoning paths without understanding what the wrong reasoning paths are. In this work, we aim to enhance the MLLMs’ reasoning ability beyond passively imitating positive reasoning paths. To this end, we design Step-wise Group Relative Policy Optimization (StepGRPO), a new online reinforcement learning framework that enables MLLMs to self-improve reasoning ability via simple, effective and dense step-wise rewarding. Specifically, StepGRPO introduces two novel rule-based reasoning rewards: Step-wise Reasoning Accuracy Reward (StepRAR) and Step-wise Reasoning Validity Reward (StepRVR). StepRAR rewards the reasoning paths that contain necessary intermediate reasoning steps via a soft key-step matching technique, while StepRAR rewards reasoning paths that follow a well-structured and logically consistent reasoning process through a reasoning completeness and logic evaluation strategy. With our proposed StepGRPO, we introduce R1-VL, a series of MLLMs with outstanding capabilities in step-by-step reasoning.

Evaluation

We use VLMEvalKit to evaluate our models on different benchmarks. Here, we provide the evaluation instructions.

First, install VLMEvalKit according to the official instructions.

Then, replace the necessary files with those in this folder.

Finally, perform evaluation via following command:

python run.py --data MathVista_MINI --model R1-VL-7B --verbose

For more evaluation options, please refer to VLMEvalKit.

Training

SFT warm-up

Data Preparation

Download Mulberry-260K and remove the reflective reasoning data.

Training

We use LLaMA-Factory to perform SFT warm-up:

First, install LLaMA-Factory according to the official_instruction.

Then, refer to here and update the following customized dataset into dataset_info.json in LLaMA-Factory.

"r1_vl_sft": {
    "file_name": "./mulberry_sft_wo_reflective_data.json",
    "formatting": "sharegpt",
    "columns": {
      "messages": "messages",
      "images": "images"
    },
    "tags": {
      "role_tag": "role",
      "content_tag": "content",
      "user_tag": "user",
      "assistant_tag": "assistant"
    }
  },

Finally, use the following command to train the models.

llamafactory-cli train examples/train_full/qwen2vl_2b_full_sft.yaml

Please refer to this folder for the training parameters used in the SFT warm-up stage.

RL with StepGRPO

Data Preparation

You may download the dataset from link, or use the provided prompt to extract keysteps from your own data.

Training

Our StepGRPO implementation is based on R1-V:

First, setup the environment.

bash setup.sh

Then, modify the SFT model path and dataset path in the following files: run_grpo_2b_vllm.sh and run_grpo_7b.sh.

Finally, perform RL using the following commands.

#qwen2-vl 2b
bash src/r1-vl/run_grpo_2b_vllm.sh
#qwen2-vl 7b
bash src/r1-vl/run_grpo_7b.sh

Main results

We conduct experiments with two powerful baseline models, including Qwen2-VL-2B and Qwen2-VL-7B. The Main Results comparing the R1-VL models with other state-of-the-art models across several widely-adopted benchmarks are shown in the figure below. All the experiments are conducted on 4 H100-80GB GPUs.

Quantitative Results

Acknowledgment

Our work is built on the following codebases, and we are deeply grateful for their contributions.

• R1-V
• LLaMA-Factory
• VLMEvalKit

Citation

We appreciate your citations if you find our paper related and useful to your research!

@InProceedings{Zhang_2025_ICCV,
    author    = {Zhang, Jingyi and Huang, Jiaxing and Yao, Huanjin and Liu, Shunyu and Zhang, Xikun and Lu, Shijian and Tao, Dacheng},
    title     = {R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2025},
    pages     = {1859-1869}
}