OpenWorldLib

Welcome to join us open-source world model project !

---

English | 中文

Extension repo：[3D generation] | [VLA] | [simulator]

We define a world model as: A model or framework centered on perception, equipped with interaction and long-term memory capabilities, for understanding and predicting the complex world. Accordingly, 🎓 Multimodal Understanding, 🤖 Visual Action Prediction, and 🖼️ Visual Generation are all sub-tasks that a world model needs to accomplish.

We warmly welcome researchers to share their views on this framework or thoughts on world models in the Issues section. We also hope that you can submit valuable world-model-related methods to our framework via Pull Requests, or document and submit them to [awesome_world_model]. Feel free to give our repo a star 🌟 to follow the latest progress of OpenWorldLib!

Important Docs

The following four documents are essential to this project (click to navigate):

• docs/planning.md: This document tracks the short-term optimization goals and future development plans for OpenWorldLib.
• docs/awesome_world_model.md: This document records cutting-edge research, related surveys, and open-source projects on world models.
• docs/installation.md: This document provides installation instructions for different methods in OpenWorldLib.
• Development Guide: This document provides a framework template for OpenWorldLib, intended to serve as a reference for code integration for developers in the world model community.

Table of Contents

• Features
  • Project Goals
  • Supported Tasks
• Getting Started
  • Installation
  • Quickstart
• Structure
• Planning
• For Developers
• Acknowledgment
• Citation

Features

Project Goals

The main goals of OpenWorldLib include:

• Establishing a unified and standardized world model framework to make the invocation of existing world-model-related code more consistent and well-structured;
• Integrating open-source world model research outcomes and systematically curating related papers for researchers' reference and use.

Supported Tasks

OpenWorldLib covers the following research directions related to World Models, We sincerely thank all the excellent methods included in this framework for their significant contributions to world model:

Task Category	Sub-task	Representative Methods/Models
Video Generation	Navigation Generation	lingbot, matrix-game, hunyuan-worldplay, genie3, etc.
	Long Video Generation	sora-2, veo-3, wan, etc.
	Egocentric Interaction Generation	Hand2World, etc.
3D Scene Generation	3D Scene Generation	flash-world, vggt, etc.
Reasoning	VQA (Visual Question Answering)	spatialVLM, omnivinci and other VLMs with world understanding
	VLA (Vision-Language-Action)	pi-0, pi-0.5, giga-brain, etc.

Commonly used inference frameworks include: diffusers, DiffSynth, LightX2V

Getting Started

Installation

First, create a conda environment:

conda create -n "openworldlib" python=3.10 -y
conda activate "openworldlib"

Then install using the provided script:

cd OpenWorldLib
bash scripts/setup/default_install.sh

Some methods have special installation requirements. All installation scripts are located in ./scripts/setup.

📖 For the full installation guide, please refer to docs/installation.md

Quickstart

After installing the base environment, you can test matrix-game-2 generation and multi-turn interaction with the following commands:

cd OpenWorldLib
bash scripts/test_inference/test_nav_video_gen.sh matrix-game-2
bash scripts/test_stream/test_nav_video_gen.sh matrix-game-2

Scripts for other methods can be found under scripts/test_inference and scripts/test_stream. Currently, we are primarily using GPUs with 80GB and 141GB of VRAM for testing. In the future, we will test more models and provide updates in the ./docs/installation.md file.

Structure

To help developers and users better understand OpenWorldLib, we provide details about our codebase. The framework structure is as follows:

OpenWorldLib
├─ assets
├─ data                                # Test data
│  ├─ benchmarks
│  ├─ test_case
│  └─ ...
├─ docs                                # Documentation
├─ examples                            # Benchmark evaluation code
├─ scripts                             # All key test scripts
├─ src
│  └─ openworldlib                     # Main source path
│     ├─ base_models                   # Base models, provide basic modules for other parts
│     │  ├─ diffusion_model
│     │  ├─ llm_mllm_core
│     │  ├─ perception_core
│     │  └─ three_dimensions
│     ├─ memories                      # Memory module
│     │  ├─ reasoning
│     │  └─ visual_synthesis
│     ├─ operators                     # Input & interaction signal processing
│     ├─ pipelines                     # All runtime pipelines
│     ├─ reasoning                     # Reasoning module
│     │  ├─ audio_reasoning
│     │  ├─ general_reasoning
│     │  └─ spatial_reasoning
│     ├─ representations               # Representation module
│     │  ├─ point_clouds_generation
│     │  └─ simulation_environment
│     └─ synthesis                     # Generation module
│        ├─ audio_generation
│        ├─ visual_generation
│        └─ vla_generation
├─ submodules                          # Auxiliary installs (e.g., diff-gaussian-raster)
├─ test                                # All test code
├─ test_stream                         # All interactive test code
└─ tools                               # Utilities
   └─ vibe_code

When using OpenWorldLib, users typically call the pipeline class directly, which handles weight loading, environment initialization, and other tasks. Users interact with the operator class, and leverage the synthesis, reasoning, and representation classes for generation. In multi-turn interactions, the memory class is used to maintain the running context.

User Input
    │
    ▼
┌─────────────┐
│  Pipeline   │──── from_pretrained(): load models & initialize modules
│             │
│  __call__() │──┬── process() ──► Operator (validate & preprocess)
│      │      │  │
│      │      │  ├── ► Synthesis.predict()           → multimodel outputs
│      │      │  ├── ► Reasoning.inference()         → text outputs
│      │      │  └── ► Representation.get_repr..()   → 3D outputs
│      │      │
│  stream()   │──┬── memory.select()  → retrieve context
│      │      │  ├── __call__()       → generate current turn
│      │      │  └── memory.record()  → store results
└─────────────┘

Planning

• We document the latest cutting-edge world model research in docs/awesome_world_model.md, and welcome contributions of valuable research.
• We document our upcoming training and optimization plans in docs/planning.md.

For Developers

We welcome all developers to contribute and help improve OpenWorldLib as a unified world model repository. We recommend using Vibe Coding for quick code contributions — related prompts can be found under tools/vibe_code/prompts. You are also encouraged to add high-quality world model works to docs/planning.md and docs/awesome_world_model.md. We look forward to your contributions!

Related documents: [Development Guide]

Acknowledgment

This project is an extension of DataFlow and DataFlow-MM for world model tasks. We are also actively collaborating with RayOrch, Paper2Any, and other projects.

Citation

If OpenWorldLib has been helpful to you, please consider giving our repo a star 🌟 and citing the related papers:

@article{team2026openworldlib,
  title={OpenWorldLib: A Unified Codebase and Definition of Advanced World Models},
  author={Team, DataFlow and Zeng, Bohan and Hua, Daili and Zhu, Kaixin and Dai, Yifan and Li, Bozhou and Wang, Yuran and Tong, Chengzhuo and Yang, Yifan and Chang, Mingkun and others},
  journal={arXiv preprint arXiv:2604.04707},
  year={2026}
}

@article{zeng2026research,
  title={Research on World Models Is Not Merely Injecting World Knowledge into Specific Tasks},
  author={Zeng, Bohan and Zhu, Kaixin and Hua, Daili and Li, Bozhou and Tong, Chengzhuo and Wang, Yuran and Huang, Xinyi and Dai, Yifan and Zhang, Zixiang and Yang, Yifan and others},
  journal={arXiv preprint arXiv:2602.01630},
  year={2026}
}

To further elaborate on our framework's design philosophy and our understanding of world models, we will release a technical report for OpenWorldLib. We hope our work is helpful to you!