WorldGen

WorldGen: Generate Any 3D Scene in Seconds

Author 👨‍💻: Ziyang Xie Contact Email 📧: [email protected]
Feel free to contact me for any questions or collaborations!

🌟 Introduction

🌏 WorldGen can generate 3D scenes in seconds from text prompts and images. It is a powerful tool for creating 3D environments and scenes for games, simulations, robotics, and virtual reality applications.

• Instant 3D Generation ⚡️ : Create full 3D scenes from input data in seconds
• 360° Free Exploration 🧭 : WorldGen supports free 360° consistent exploration of the generated 3D scene with loop closure.
• Diverse Scenes Support 🌈 : WorldGen supports both indoor and outdoor scenes, both realistic and unrealistic scenes in any style.
• Flexible Rendering 📸 : WorldGen supports rendering at any resolution with any camera setting and trajectory in real-time.

Two lines of code to generate a 3D scene in seconds

# Use our API to generate a 3D scene
worldgen = WorldGen()
worldgen.generate_world("<TEXT PROMPT to describe the scene>")

Text-to-Scene Generation

Image-to-Scene Generation

---

News and TODOs

• 04.12.2026 🔧 [Improved Sharp] Rework ml-sharp pipeline: use cubemap depth to align Sharp's per-face gaussians for better global consistency. Reduced from 8+ views to 6 cubemap faces.
• 04.11.2026 🔄 [Updated Depth] Replace UniK3D with DA-2 for better 360° depth estimation.
• 03.17.2026 🔧 [Improved Quality] Improve the GS quality by fixing the project scale issue. Make ml-sharp dependency optional.
• 01.10.2026 🔥 [New feature] Add support for ml-sharp (modified to work on 360 images) for better GS generation (Currently in experimental mode)
• 05.10.2025 🤖 Add support for low-vram generation (Only use ~10GB VRAM for generation).
• 04.26.2025 📄 New Relase a project page for WorldGen
• 04.22.2025 🏡 Add support for mesh scene generation (Should give better results than splat)
• 04.21.2025 🎉 Opensource the WorldGen codebase
• 04.19.2025 🖼️ Add support for image-to-scene generation
• 04.17.2025 📝 Add support for text-to-scene generation
• Release technical report and video
• Support better background inpainting (Invisible region inpainting)

📦 Installation

Getting started with WorldGen is simple!

# Clone the repository 
git clone --recursive https://github.com/ZiYang-xie/WorldGen.git 
cd WorldGen

# Create a new conda environment
conda create -n worldgen python=3.11
conda activate worldgen

# Install torch and torchvision (with GPU support)
pip3 install torch torchvision

# Install worldgen
pip install .

# Install DA-2 (360 depth estimation) -- use --no-deps to avoid version conflicts
pip install git+https://github.com/EnVision-Research/DA-2.git#subdirectory=src --no-deps

# Install pytorch3d dependencies
pip install git+https://github.com/facebookresearch/pytorch3d.git --no-build-isolation

# 🔥 [New feature]: If you want to use the ml-sharp experimental feature, you need to install the ml-sharp dependencies
pip install -e submodules/ml-sharp

# You should also accept the license of the gated model (FLUX.1-dev).
# https://huggingface.co/black-forest-labs/FLUX.1-dev
# Login to Hugging Face and accept the license.
# huggingface-cli login

🕹️ Quick Start / Demos

We provide a demo script to help you quickly get started and visualize the 3D scene in a web browser. The script is powered by Viser.

# Generate a 3D scene from a text prompt
python demo.py -p "A beautiful landscape with a river and mountains"
# Indoor scene example
python demo.py -p "A well-designed cozy bedroom"

# 🔥 New feature: Generate a 3D scene using the ml-sharp experimental feature (It may produce better results than the default mode)
python demo.py -p "<TEXT PROMPT to describe the scene>" --use_sharp

# Generate a 3D scene from an image
python demo.py -i "path/to/your/image.jpg" -p "<Optional: TEXT PROMPT to describe the scene>" --use_sharp

# Generate a 3D scene in mesh mode
# Make sure you installed my customized viser to correctly visualize the mesh without backface culling
# pip install git+https://github.com/ZiYang-xie/viser.git

python demo.py -p "A beautiful landscape with a river and mountains" --return_mesh

After running the demo script, A local viser server will be launched at http://localhost:8080, where you can explore the generated 3D scene in real-time.

🎮 Advanced Usage

WorldGen API

Quick start with WorldGen (mode in t2s or i2s) and generate your first 3D scene in seconds:

• 📝 Text to Scene: Generate a 3D scene from a text prompt

# Example using the Python API
from worldgen import WorldGen
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

worldgen = WorldGen(mode="t2s", device=device, low_vram=False) # Set low_vram to True if your GPU VRAM is less than 24GB.
splat = worldgen.generate_world("<TEXT PROMPT to describe the scene>")
splat.save("path/to/your/output.ply") # Save splat file as a .ply file, which can be loaded and visualized using a standard gaussian splatting viewer

• 🖼️ Image to Scene: Generate a 3D scene from an image

worldgen = WorldGen(mode="i2s", device=device, low_vram=False) # Set low_vram to True if your GPU VRAM is less than 24GB.
image = Image.open("path/to/your/image.jpg")
splat = worldgen.generate_world(
    image=image,
    prompt="<Optional: TEXT PROMPT to describe the image and the scene>",
)

• 🏡 Generate a 3D scene in mesh mode

mesh = worldgen.generate_world("<TEXT PROMPT to describe the scene>", return_mesh=True)
o3d.io.write_triangle_mesh("path/to/your/output.ply", mesh) # Save mesh as a .ply file

[!Tip] We also support background inpainting for better scene generation, but it's currently an experimental feature, which may not work for all scenes.
It can be enabled by setting WorldGen(inpaint_bg=True).

# If want to use background inpainting feature, install iopaint
pip install iopaint --no-dependencies

Free-viewpoint Exploration in 3D Scene

---

[!Note] WorldGen internally support generating a 3D scene from a 360° panorama image 📸, which related to how WorldGen works: You can try it out if you happen to have a 360° panorama (equirectangular) image. Aspect ratio of the panorama image should be 2:1.

 pano_image = Image.open("path/to/your/pano_image.jpg")
 splat = worldgen._generate_world(pano_image=pano_image)

⭐️ Star History

Give a star to WorldGen if you like it!

📚 Citation

If you find this project useful, please consider citing it as follows:

@misc{worldgen2025ziyangxie,
  author = {Ziyang Xie},
  title = {WorldGen: Generate Any 3D Scene in Seconds},
  year = {2025},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/ZiYang-xie/WorldGen}},
}

---

🤝 Acknowledgements

This project is built on top of the follows, please consider citing them if you find them useful:

• DA-2
• UniK3D
• Layerpano3D
• Viser
• FLUX.1
• OneFormer
• LaMa
• ml-sharp

Some of the core methods and ideas in this project are inspired by the following projects, special thanks to them:

• WonderWorld [RGBD to GS conversion]
• WorldSheet [Mesh Generation]