Awesome-Video-World-Models-with-AR-Diffusion

📹 Awesome Video World Models with AR Diffusion

Overview

This repository focuses on Video World Models with Autoregressive (AR) Diffusion, a promising paradigm for scalable, consistent and interactive world modeling (e.g., Genie 3). It aims to serve as a comprehensive and structured resource for researchers, practitioners, and enthusiasts interested in AR diffusion-based video world modeling. To stay at the forefront of the field, this repository is updated weekly.

🔥NEW: minWM: Full-Stack Open-Source Video World Model Data & Traing & Distillation & Inference Framework

Try our minWM, the first full-stack open-source framework that walks you end-to-end through turning a bidirectional T2V foundation model into an action-conditioned video world model — with example data, runnable scripts, Claude skills capturing our hands-on experience, and onboarding knowledge for newcomers!

🌟 Key Features

• Structured Taxonomy: We organize the evolving ecosystem from three complementary perspectives: Algorithmic Foundations, Real-world Applications, and Infrastructure-level Acceleration. Together, these dimensions reflect the full stack of AR diffusion—from modeling design to real-time interactive deployment.
• One-Stop Citation Collection: 📚 We provide a consolidated BibTeX file containing all papers listed in this repository. You can easily import it into your LaTeX or Zotero projects with one click!

📬 Contact

This repository is curated and maintained by:

• Min Zhao ([email protected])
• Hongzhou Zhu ([email protected])
• Wenqiang Sun ([email protected])
• Bokai Yan ([email protected])

For any questions or suggestions, please feel free to reach out to us.

• 🎯 We have not yet compiled an exhaustive list of all related work. We apologize for any omissions and welcome pull requests to merge them in.
• 💡 We also welcome high-level categorization, synthesis, and perspective contributions to improve the organization and clarity of this repository.

Table of Contents

• 1. Algorithm
  • 1.1 AR Diffusion (native pretraining)
  • 1.2 AR Diffusion Distillation for Real-time Generation (post training)
  • 1.3 Long Video Generation
• 2. Application
  • 2.1 Open-source AR Video Foundation Models
  • 2.2 Interactive Video Action World Model
  • 2.3 Real-time Interactive Avtar & Motion & Physical & Audio Control
  • 2.4 Egocentric Interaction
  • 2.5 Embodied AI / Autonomous Driving
• 3 Infrastructure
  • 3.1 Sparse Attention
  • 3.2 Caching
  • 3.3 Quantized Attention
  • 3.4 Speculative Decoding

1. Algorithm

1.1 AR Diffusion (native pretraining)

These methods focus on basic AR Diffusion (where each chunk/frame is generated via diffusion and the frames are AR).

• Diffusion Forcing: "Next-token Prediction Meets Full-Sequence Diffusion".
• Pyramid Flow: "Pyramidal Flow Matching for Efficient Video Generative Modeling".
• DFoT, "History-Guided Video Diffusion".
• AR-Diffusion, "AR-Diffusion: Asynchronous Video Generation with Auto-Regressive Diffusion".
• PFVG, "Pack and force your memory: Long-form and consistent video generation".
• BAgger, "BAgger: Backwards Aggregation for Mitigating Drift in Autoregressive Video Diffusion Models".
• Resampling Forcing, "End-to-End Training for Autoregressive Video Diffusion via Self-Resampling".
• Helios, "Helios: Real Real-Time Long Video Generation Model".

1.2 🔥 AR Diffusion Distillation for Real-time Generation (post training)

This category of algorithms focuses on distilling multi-step bidirectional diffusion models into few-step AR models, specifically tailored for real-time streaming generation.

• From Multi-step Bidirectional Diffusion to Few-step Autoregressive Generators:
  • [⭐] CausVid, "From Slow Bidirectional to Fast Autoregressive Video Diffusion Models".
  • [⭐] Self Forcing, "Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion".
  • [⭐] Causal Forcing, "Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation".
  • [⭐] Causal Forcing++, "Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generation".
  • [⭐] (Another line of AR distillation via GAN) APT2, "Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generation".

• Further Improvements:
  • (Adversarial distillation) Seaweed APT2, "Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generation".
  • (One-step distillation) ASD, "Towards One-Step Causal Video Generation via Adversarial Self-Distillation".
  • (Two-steps distillation) Diagonal Distillation, "Streaming Autoregressive Video Generation via Diagonal Distillation".
  • (Reinforcement learning) Reward Forcing, "Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillation".
  • (Reinforcement learning) WorldCompass, "WorldCompass: Reinforcement Learning for Long-Horizon World Models".
  • (Reinforcement learning) AR-CoPO,"AR-CoPO: Align Autoregressive Video Generation with Contrastive Policy Optimization".
  • (Reinforcement learning) Astrolabe, "Astrolabe: Steering Forward-Process Reinforcement Learning for Distilled Autoregressive Video Models".
  • (Distribution matching) Salt, "Salt: Self-Consistent Distribution Matching with Cache-Aware Training for Fast Video Generation".
  • (Reinforcement learning) RAVEN, "RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-model GRPO".
  • (Distillation) Delta Forcing, "Delta Forcing: Trust Region Steering for Interactive AR Video Generation".

1.3 Long Video Generation

• From Short-video Generator to Long-video Generator:

  • FramePack, "Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Models".

  • LongLive, "LongLive: Real-time Interactive Long Video Generation".

  • Rolling Forcing, "Rolling Forcing: Autoregressive Long Video Diffusion in Real Time".

  • Self Forcing++, "Self-Forcing++: Towards Minute-Scale High-Quality Video Generation".

  • Infinite Forcing,

  • Infinity-RoPE, "Infinity-RoPE: Action-Controllable Infinite Video Generation Emerges From Autoregressive Self-Rollout".

  • Deep Forcing, "Deep Forcing: Training-Free Long Video Generation with Deep Sink and Participative Compression".

  • LoL, "LoL: Longer than Longer, Scaling Video Generation to Hour".

  • FLEX, "Train Short, Inference Long: Training-free Horizon Extension for Autoregressive Video Generation".

  • LIVE, "LIVE: Long-horizon Interactive Video World Modeling".

  • Rolling Sink, "Rolling Sink: Bridging Limited-Horizon Training and Open-Ended Testing in Autoregressive Video Diffusion".

  • MMM, "Mode Seeking meets Mean Seeking for Fast Long Video Generation".

  • MemRoPE, "MemRoPE: Training-Free Infinite Video Generation via Evolving Memory Tokens".

  • Anchor Forcing, "Anchor Forcing: Anchor Memory and Tri-Region RoPE for Interactive Streaming Video Diffusion".

  • ShotStream, "ShotStream: Streaming Multi-Shot Video Generation for Interactive Storytelling".

  • DCARL, "DCARL: A Divide-and-Conquer Framework for Autoregressive Long-Trajectory Video Generation".

  • PackForcing, "PackForcing: Short Video Training Suffices for Long Video Sampling and Long Context Inference".

  • Grounded Forcing, "Grounded Forcing: Bridging Time-Independent Semantics and Proximal Dynamics in Autoregressive Video Synthesis".

  • TempoMaster, "TempoMaster: Efficient Long Video Generation via Next-Frame-Rate Prediction",

  • Hybrid Forcing, "Long-Horizon Streaming Video Generation via Hybrid Attention with Decoupled Distillation",

  • Pyramid Forcing, "Pyramid Forcing: Head-Aware Pyramid KV Cache Policy for High-Quality Long Video Generation",

  • LongTake, "LongTake: Learning to Sustain Dynamics in Long-Horizon Video Generation".

• Multi-Shot Long Video:
  • CausalCine, "CausalCine: Real-Time Autoregressive Generation for Multi-Shot Video Narratives".

• Long-term Memory:
  • WORLDMEM, "WORLDMEM: Long-term Consistent World Simulation with Memory".
  • VRAG, "Learning World Models for Interactive Video Generation".
  • Context as Memory, "Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrieval".
• Echo-Memory, "Echo-Memory: A Controlled Study of Memory in Action World Models".
  • Memory Forcing, "Memory Forcing: Spatio-Temporal Memory for Consistent Scene Generation on Minecraft".
  • MemFlow, "MemFlow: Flowing Adaptive Memory for Consistent and Efficient Long Video Narratives".
  • StableWorld, "StableWorld: Towards Stable and Consistent Long Interactive Video Generation".
  • Infinite-World, "Infinite-World: Scaling Interactive World Models to 1000-Frame Horizons via Pose-Free Hierarchical Memory".
  • Context Forcing, "Context Forcing: Consistent Autoregressive Video Generation with Long Context".
  • ViewRope, "Geometry-Aware Rotary Position Embedding for Consistent Video World Model".
  • UCM, "UCM: Unifying Camera Control and Memory with Time-Aware Positional Encoding Warping for World Models".
  • PERSIST, "Beyond Pixel Histories: World Models with Persistent 3D State".
  • MosaicMem, "MosaicMem: Hybrid Spatial Memory for Controllable Video World Models".
  • HyDRA, "Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Models".
  • StreamHOI, "StreamHOI: Interaction-aware Temporal Memory Adaptation for Streaming HOI Video Generation".

2. Application

Up to 26.03 now. Welcome Pull Requests!

2.1 Open-source AR Video Foundation Models

• Pyramid Flow: "Pyramidal Flow Matching for Efficient Video Generative Modeling".
• SkyReels, "SkyReels-V2: Infinite-length Film Generative Model".
• MAGI-1, "MAGI-1: Autoregressive Video Generation at Scale".
• Helios, "Helios: Real Real-Time Long Video Generation Model".

2.2 Interactive Video Action World Model

• Genie3.

• Yan, "Yan: Foundational Interactive Video Generation".

• Matrix-Game 3.0, "Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory".

• Matrix-game 2.0, "Matrix-game 2.0: An open-source real-time and streaming interactive world model".

• PAN, "PAN: A World Model for General, Interactable, and Long-Horizon World Simulation".

• RELIC, "RELIC: Interactive Video World Model with Long-Horizon Memory".

• Astra, "Astra: General Interactive World Model with Autoregressive Denoising".

• HY-WorldPlay, "WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling".

• Yume 1.5, "Yume-1.5: A Text-Controlled Interactive World Generation Model".

• LingBot-World, "Advancing Open-source World Models".

• Olaf-World, "Olaf-World: Orienting Latent Actions for Video World Modeling".

• Solaris, "Solaris: Building a Multiplayer Video World Model in Minecraft".

• MagicWorld, "MagicWorld: Towards Long-Horizon Stability for Interactive Video World Exploration".

• OmniRoam, "OmniRoam: World Wandering via Long-Horizon Panoramic Video Generation".

• INSPATIO-WORLD, "INSPATIO-WORLD: A Real-Time 4D World Simulator via Spatiotemporal Autoregressive Modeling".

• ActionParty, "ActionParty: Multi-Subject Action Binding in Generative Video Games".

• MultiWorld, "MultiWorld: Scalable Multi-Agent Multi-View Video World Models".

• ABot-World-0, "ABot-World-0: Infinite Interactive World Rollout on a Single Desktop GPU".

2.3 Real-time Interactive Avtar & Motion & Physical & Audio Control

• MotionStream, "MotionStream: Real-Time Video Generation with Interactive Motion Controls".
• RealVideo,
• LiveAvatar, "Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Length".
• • Knot Forcing, "Knot Forcing: Taming Autoregressive Video Diffusion Models for Real-time Infinite Interactive Portrait Animation".
• SoulX-FlashTalk, "SoulX-FlashTalk: Real-Time Infinite Streaming of Audio-Driven Avatars via Self-Correcting Bidirectional Distillation".
• LiveTalk, "LiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillation".
• Avatar Forcing, "Avatar Forcing: Real-Time Interactive Head Avatar Generation for Natural Conversation".

• Geometry-as-context, "Geometry-as-context: Modulating Explicit 3D in Scene-consistent Video Generation to Geometry Context".

• DragStream, "Streaming Drag-Oriented Interactive Video Manipulation: Drag Anything, Anytime!".

• RealWonder, "RealWonder: Real-Time Physical Action-Conditioned Video Generation".

• WMReward, "Inference-time Physics Alignment of Video Generative Models with Latent World Models".

• SoulX-LiveAct, "SoulX-LiveAct: Towards Hour-Scale Real-Time Human Animation with Neighbor Forcing and ConvKV Memory".

• OmniForcing, "OmniForcing: Unleashing Real-time Joint Audio-Visual Generation".

• PRISM:, "PRISM: Streaming Human Motion Generation with Per-Joint Latent Decomposition".

• DiT as Real-Time Rerenderer, "DiT as Real-Time Rerenderer: Streaming Video Stylization with Autoregressive Diffusion Transformer".

• PortraitDirector,"PortraitDirector: A Hierarchical Disentanglement Framework for Controllable and Real-time Facial Reenactment".

2.4 Egocentric Interaction

This category focuses on first-person (egocentric) video generation, emphasizing hand-object interaction for VR.

• Hand2World, "Hand2World: Autoregressive Egocentric Interaction Generation via Free-Space Hand Gestures".

• Generated Reality, "Generated Reality: Human-centric World Simulation using Interactive Video Generation with Hand and Camera Control".

2.5 Embodied AI / Autonomous Driving

• DriveVA, "DriveVA: Video Action Models are Zero-Shot Drivers" (ECCV 2026).

• Vidarc, "Vidarc: Embodied Video Diffusion Model for Closed-loop Control".

• DreamDojo, "DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos".

• DreamZero, "World Action Models are Zero-shot Policies".

• FAR-Drive, "FAR-Drive: Fram