nano-world-model

A minimalist repository for training video world models based on diffusion-forcing.

News and Updates

• 🚩 2026-05: We release 📕 Technical Report on arxiv, check it out!

• 🚩 2026-05: We release all model checkpoints on 🤗 huggingface.

• 🚩 2026-05: We release code for training, inference and evaluation.

Key Features

• 🚀 Instant Start — Minimal dependencies, easy data loading. From clone to first rollout in minutes.
• 🛠️ Unified Pipeline — Training, Validation, Evaluation; All managed with clean hydra-based configuration systems.
• 🔬 Scientific Transparency — Clean codebase with head-to-head ablations across prediction target, action injection, and model scale; Fully open-source, including model checkpoints.
• 🤖 Diverse Applications — Long-horizon rollouts, rollout to 3d point clouds, planning (MPC) out of the box.

🚀 Quick Start

Try the Colab quickstart notebook to download the official Point Maze checkpoint and preview predictions beside recorded validation frames. Select a GPU runtime, then run all cells; the notebook uses the same lock below.

Install uv first. The supported Linux x86_64 environment uses one committed lock for Python, PyTorch, CUDA wheels, and all dependencies. See environment setup and verification.

git clone https://github.com/simchowitzlabpublic/nano-world-model.git
cd nano-world-model
./nanowm sync
./nanowm doctor --require-cuda

For the locked RT-1/LeRobot dependencies, run ./nanowm sync --extra rt1. Do not install lerobot-datasets; it is a dataset format version, not a PyPI package.

Set data + results paths (or use the gitignored src/configs/local/paths.yaml template — see docs/config_system.md):

export DATASET_DIR=/path/to/dino_wm_data       # DINO-WM envs (point_maze, pusht, ...)
export CSGO_DATA_DIR=/path/to/csgo             # CSGO HDF5 files
export RT1_DATA_ROOT=/path/to/rt1_fractal      # RT-1 LeRobot mirror (optional)
export RESULTS_DIR=/path/to/results            # checkpoints + logs land here

Download the i3d torchscript used by FVD evaluation (FID uses InceptionV3, which downloads itself on first use):

mkdir -p pretrained_models/i3d && curl -L \
    "https://www.dropbox.com/scl/fi/c5nfs6c422nlpj880jbmh/i3d_torchscript.pt?rlkey=x5xcjsrz0818i4qxyoglp5bb8&dl=1" \
    -o pretrained_models/i3d/i3d_torchscript.pt

For dataset downloads (DINO-WM, RT-1, CSGO), see docs/datasets/README.md.

🥷 Train your first model

DINO-WM PushT, NanoWM-B/2, default settings (pred-v · additive injection · cosine + ZTSNR):

./nanowm python src/main.py experiment=dino_wm_pusht dataset=dino_wm/pusht model=nanowm_b2

CSGO with the L/2 model:

./nanowm python src/main.py experiment=csgo dataset=game/csgo model=nanowm_l2_csgo

RT-1 (fractal) main run:

./nanowm python src/main.py experiment=rt1 dataset=rt1/rt1 model=nanowm_b2

For reproducibility, we provide example scripts in src/scripts/. See docs/training.md for the full training guide, design choices, and ablation tables.

📦 Pretrained Checkpoints

Best-config runs (pred-v · additive · cosine + ZTSNR · NanoWM-B/2 unless noted):

Domain	Checkpoint	Steps
DINO-WM Point Maze	🤗 nanowm-b2-dino-wm-point-maze-30k	30k
DINO-WM Wall	🤗 nanowm-b2-dino-wm-wall-15k	15k
DINO-WM Rope	🤗 nanowm-b2-dino-wm-rope-15k	15k
DINO-WM Granular	🤗 nanowm-b2-dino-wm-granular-15k	15k
DINO-WM PushT	🤗 nanowm-b2-dino-wm-pusht-100k	100k
RT-1 (fractal)	🤗 nanowm-b2-rt1-300k	300k
CSGO	🤗 nanowm-l2-csgo-100k (NanoWM-L/2)	100k

We also provide RT-1 ablation tables with HF checkpoint paths. See docs/training.md#design-choices for the full table and ablation numbers.

🎬 Sample Predictions

CSGO 50-frame auto-regressive long-rollouts (NanoWM-L/2, 100k):

Quantitative Metrics Evaluated on 256 fixed samples (seed=42), 250 DDIM steps, sequential scheduling (frame-by-frame autoregressive denoising).

Dataset	Steps	PSNR ↑	SSIM ↑	LPIPS ↓	FID ↓
Point Maze	30k	36.74	0.984	0.019	9.66
Wall	15k	34.05	0.994	0.010	2.64
PushT	100k	33.19	0.982	0.016	13.63
Rope	15k	31.63	0.953	0.056	35.20
Granular	15k	26.08	0.917	0.073	40.05
RT-1	300k	24.36	0.787	0.180	35.08

Full per-domain numbers and methodology in docs/evaluation.md.

🧭 Applications

NanoWM rollouts can be used directly for downstream applications, including long-horizon generation, video-to-3D reconstruction, and MPC-style planning.

• Long-horizon rollout — autoregressive rollout from trained checkpoints
• Video → 3D map — Depth Anything 3 point cloud reconstruction from rollout videos
• MPC-style planning — CEM planning over world model rollouts

📚 Documentation

• docs/config_system.md — Hydra config layout, overrides, environment variables
• docs/training.md — training workflow, design choices, ablation tables, all checkpoints
• docs/evaluation.md — evaluation workflow, metric definitions, full result tables
• docs/datasets/README.md — DINO-WM / RT-1 / CSGO formats, downloads, splits
• docs/applications/planning.md — MPC + CEM model-predictive control
• docs/applications/long_rollout.md — long-horizon autoregressive rollout
• docs/applications/video_to_3d.md — Depth Anything 3 point cloud pipeline

🙏 Acknowledgements

We build upon a number of existing codebases: Latte, Vid2World, DFoT, and DINO-WM. More broadly, this repository draws inspirations and design principles from NanoGPT, NanoChat, and Boyuan Chen's Research Template. We sincerely thank the codebases above for open-sourcing their works.

📝 Citation

If you find this repository useful in your research, please consider citing:

@misc{huang2026nanoworldmodels,
  title={Nano World Models: A Minimalist Implementation of Future Video Prediction},
  author={Siqiao Huang and Partha Kaushik and Michael Chen and Hengkai Pan and Omar Chehab and Fernando Moreno-Pino and Max Simchowitz},
  year={2026},
  eprint={2605.23993},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2605.23993},
}

✨ Star History