OpenS2V-Nexus

Open-Sora Plan: Open-Source Large Video Generation Model Bin Lin, Yunyang Ge and Xinhua Cheng etc.

Helios: Real Real-Time Long Video Generation Model Shenghai Yuan, Jinfa Huang and Xianyi He etc.

ConsisID: Identity-Preserving Text-to-Video Generation by Frequency Decomposition Shenghai Yuan, Jinfa Huang and Xianyi He etc.

MagicTime: Time-lapse Video Generation Models as Metamorphic Simulators Shenghai Yuan, Jinfa Huang and Yujun Shi etc.

ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generation Shenghai Yuan, Jinfa Huang and Yongqi Xu etc.

📣 News

• ⏳⏳⏳ Evaluating more models and updating the . PRs are welcome!
• [2026.04.03] 🏃‍♂️ We add the evaluation results for RefAlign.
• [2026.03.08] 👋 We introduce Helios, a breakthrough video generation model that achieves minute-scale, high-quality video synthesis at 19.5 FPS on a single H100 GPU — without relying on conventional long video anti-drifting strategies or standard video acceleration techniques. Welcome to check Technical Report!
• [2026.01.08] 🏃‍♂️ We add the evaluation results for VINO; click here for details.
• [2026.01.07] 🔥 We release the high-quality subset. It’s not just 0.3M samples — we applied filtering across the entire 5M data. You can click here for more details, and click here to download.
• [2026.01.06] 🚀 We will release the high-quality subset very soon! Stay tuned!
• [2025.12.05] 🏃‍♂️ We add the evaluation results for Saber; click here for details.
• [2025.11.04] 🚀 Great work on BindWeave! The team has open-sourced the entire codebase, including the pipeline for training with OpenS2V-5M and evaluation on OpenS2V-Eval. You can refer to this repository as a practical example of how to use OpenS2V-Nexus.
• [2025.10.21] 🏃‍♂️ We add the evaluation results for Stand-In and BindWeave; click here and here for details.
• [2025.09.19] ✨ Our paper is accepted by NeurIPS 2025 D&B!
• [2025.08.30] 🚀 Thanks for the excellent work DanceGRPO on transferring ConsisID data for I2V RL training, please refer to here for more details. Similarly, you can also try using OpenS2V-5M for RL training.
• [2025.08.05] 🔥 We provide a dev version that increases the max number of subject images in OpenS2V-Eval to 5, and have uploaded results for Vidu-Q1, Vidu-2.0, and Kling1.6. Also, OpenS2V-Eval-Leaderboard v1.1 is out, which introduces Motion Smoothness score for improved motion quality measurement.
• [2025.07.01] 🎉 Thanks to our amazing community — the OpenS2V-5M dataset has reached ~40,000 downloads on Hugging Face in just one month!
• [2025.06.21] 🏃‍♂️ We add the evaluation results for MAGREF-480P; click here and here for details.
• [2025.06.19] 🔥 The preprocessed Cross-Frame Pairs is now available on Hugging Face, eliminating the need for online processing with this code during training. We also provide a demo dataloader here demonstrating how to use OpenS2V-5M during the training phase.
• [2025.05.31] 🏃‍♂️ We add the evaluation results for Concat-ID-Wan-AdaLN; click here and here for details.
• [2025.05.28] 🏃‍♂️ We add the evaluation results for Phantom-14B; click here and here for details.
• [2025.05.27] 🔥 Our arXiv paper on OpenS2V-Nexus is now available; click here for details.
• [2025.05.26] 🔥 All codes & datasets are out! We also release the testing prompts, reference images and videos generated by different models in OpenS2V-Eval, and you can click here to see more details.

💡 Community Works

If your research or project builds upon OpenS2V-5M or OpenS2V-Eval, and you would like more people to see it, please inform us.

• OmniShow, the first all-in-one model for human-object interaction video generation, unifying text, reference images, audio, and pose conditions into one model for multimodal controllable generative modeling. It supports R2V, A2V, RA2V, RP2V, and RAP2V tasks within one coherent framework, and also exhibits remarkable versatility in broader applications, such as audio-driven avatars, object swapping, and video remixing.
• RefAlign, a training-time alignment framework for improving reference fidelity, identity consistency, and text controllability in reference-to-video generation.
• VINO, a unified framework for image and video generation and editing. It uses a shared diffusion backbone and interleaved conditioning to enable multi-reference grounding, instruction following, and consistent identity preservation across modalities within a single model.
• Saber, a zero-shot reference-to-video framework that generates identity-consistent videos from text and reference images without explicit R2V training data, using masked video frames as self-supervision and attention-guided learning to support flexible multi-subject customization.
• ID-Composer, a reinforcement learning-based multi-subject video generation framework that synthesizes high-quality, identity-consistent videos from text prompts and reference images through hierarchical identity-preserving attention and vision-language model semantic guidance.
• BindWeave, a unified framework that uses multimodal large language model reasoning to disentangle complex prompt semantics and generate high-fidelity, subject-consistent videos across diverse single- and multi-subject scenarios.
• Humo, a unified, human-centric video generation framework designed to produce high-quality, fine-grained, and controllable human videos from multimodal inputs—including text, images, and audio. It supports strong text prompt following, consistent subject preservation, synchronized audio-driven motion.
• Stand-In, a lightweight, plug-and-play framework for identity-preserving video generation. By training only 1% additional parameters compared to the base video generation model, they achieve state-of-the-art results in both Face Similarity and Naturalness, outperforming various full-parameter training methods.
• MagRef, a unified video generation framework that uses masked guidance to achieve high-quality, multi-subject consistent video synthesis from multiple reference images and text prompts.

✨ Highlights

1. New S2V Benchmark.
   • We introduce OpenS2V-Eval for comprehensive evaluation of S2V models and propose three new automatic metrics aligned with human perception.
2. New Insights for S2V Model Selection.
   • Our evaluations using OpenS2V-Eval provide crucial insights into the strengths and weaknesses of various subject-to-video generation models.
3. Million-Scale S2V Dataset.
   • We create OpenS2V-5M, a dataset with 5.1M high-quality regular data and 0.35M Nexus Data, the latter is expected to address the three core challenges of subject-to-video.

Resources

• OpenS2V-Eval: including 180 open-domain subject-text pairs, of which 80 are real and 100 are synthetic samples.
• OpenS2V-5M: including 5M open-domain subject-text-video triples, which not only include Regular Data but also incorporate Nexus Data constructed using GPT-Image-1 and cross-video associations.
• ConsisID-Bench: including 150 human-domain subject images and 90 text prompts, respectively.
• ConsisID-Preview-Data: including 32K human-domain high-quality subject-text-video triples.

😍 In-house Model Gallery

This model (Ours‡) was trained on a subset of OpenS2V-5M, using about 0.3M high-quality data.

⚙️ Requirements and Installation

We recommend the requirements as follows.

Base Environment

# 0. Clone the repo
git clone --depth=1 https://github.com/PKU-YuanGroup/OpenS2V-Nexus.git
cd OpenS2V-Nexus

# 1. Create conda environment
conda create -n opens2v python=3.12.0
conda activate opens2v

# 3. Install PyTorch and other dependencies
# CUDA 11.8
pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu118
pip install flashinfer-python==0.2.2.post1 -i https://flashinfer.ai/whl/cu118/torch2.6
# CUDA 12.4
pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu124
pip install flashinfer-python==0.2.2.post1 -i https://flashinfer.ai/whl/cu124/torch2.6

# 4. Install main dependencies
pip install -r requirements.txt
pip install flash-attn --no-build-isolation

Base Checkpoints

cd OpenS2V-Nexus

huggingface-cli download --repo-type model \
BestWishYsh/OpenS2V-Weight \
--local-dir ckpts

Once ready, the weights will be organized in this format:

📦 OpenS2V-Nexus/
├── 📂 LaMa
├── 📂 face_extractor
├── 📄 aesthetic-model.pth
├── 📄 glint360k_curricular_face_r101_backbone.bin
├── 📄 groundingdino_swint_ogc.pth
├── 📄 sam2.1_hiera_large.pt
├── 📄 yolo_world_v2_l_image_prompt_adapter-719a7afb.pth

🗝️ Benchmark

OpenS2V-Eval Results

We visualize the evaluation results of various Subject-to-Video generation models across Open-Domain, Human-Domain and Single-Object.

Get Videos Generated by Different S2V models

To facilitate future research and to ensure full transparency, we release all the videos we sampled and used for OpenS2V-Eval evaluation. You can download them on Hugging Face. We also provide detailed explanations of the sampled videos and detailed setting for the models under evaluation here.

Leaderboard

See numeric values at our Leaderboard :1st_place_medal::2nd_place_medal::3rd_place_medal:

or you can run it locally:

cd leaderboard
python app.py

Evaluate Your Own Models

Please refer to this guide for how to evaluate customized models.

🤗 Dataset

Subject-Text-Video Triples in OpenS2V-5M

Get the Data

We release the subset of the OpenS2V-5M. The dataset is available at HuggingFace, or you can download it with the following command. Some samples can be found on our Project Page.

huggingface-cli download --repo-type dataset \
BestWishYsh/OpenS2V-5M \
--local-dir BestWishYsh/OpenS2V-5M

Usage of OpenS2V-5M

Please refer to this guide for how to use OpenS2V-5M dataset.

Process Your Own Videos

Please refer to this guide for how to process customized videos.

👍 Acknowledgement

• This project wouldn't be possible without the following open-sourced repositories: Open-Sora Plan, Video-Dataset-Scripts, YOLO-World, Grounded-SAM-2, improved-aesthetic-predictor, Qwen2.5-VL, vllm, VBench, ChronoMagic-Bench, Phantom, VACE, SkyReels-A2, HunyuanCustom, ConsisID, Concat-ID, Fantasy-ID, EchoVideo.

🔒 License

• The majority of this project is released under the Apache 2.0 license as found in the LICENSE file.
• The service is a research preview. Please contact us if you find any potential violations. ([email protected])

✏️ Citation

If you find our paper and code useful in your research, please consider giving a star :star: and citation :pencil:.

@article{yuan2025opens2v,
  title={OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation},
  author={Yuan, Shenghai and He, Xianyi and Deng, Yufan and Ye, Yang and Huang, Jinfa and Lin, Bin and Luo, Jiebo and Yuan, Li},
  journal={arXiv preprint arXiv:2505.20292},
  year={2025}
}

🤝 Contributors