Awesome-Unified-Multimodal-Models

Figure 1: Timeline of Publicly Available and Unavailable Unified Multimodal Models. The models are categorized by their release years, from 2023 to 2025. Models underlined in the diagram represent any-to-any multimodal models, capable of handling inputs or outputs beyond text and image, such as audio, video, and speech. The timeline highlights the rapid growth in this field.

🔥 We are hiring!

We are looking for both interns and full-time researchers to join our team, focusing on multimodal understanding, generation, reasoning, AI agents, and unified multimodal models. If you are interested in exploring these exciting areas, please reach out to us at [email protected].

👉 What is This Repo for?

This repository provides a comprehensive collection of resources related to unified multimodal models, featuring:

• A survey of advances, challenges, and timelines for unified models
• Categorized lists of diffusion-based, autoregressive (MLLM), and hybrid architectures for unified image–text understanding and generation
• Benchmarks for evaluating multimodal comprehension, image generation, and interleaved image–text tasks
• Representative datasets covering multimodal understanding, text-to-image synthesis, image editing, and interleaved interactions

Designed to help researchers and practitioners explore, compare, and build state-of-the-art unified multimodal systems.

Awesome Papers & Datasets

• Text-and-Image Unified Models
  • Diffusion
  • MLLM AR
  • MLLM AR-Diffusion
• Any-to-Any Multimodal models
• Benchmark for Evaluation
  • Benchmarks on Understanding Tasks
  • Benchmarks on Image Generation Tasks
  • Benchmarks on Interleaved Tasks
• Dataset
  • Multimodal Understanding
  • Text-to-Image
  • Image Editing
  • Interleaved Image-Text
  • Other Text+Image-to-Image
• Applications and Opportunities

Text-and-Image Unified Models

Figure 2: Classification of Unified Multimodal Understanding and Generation Models. The models are divided into three main categories based on their backbone architecture: Diffusion, MLLM (AR), and MLLM (AR + Diffusion). Each category is further subdivided according to the encoding strategy employed, including Pixel Encoding, Semantic Encoding, Learnable Query Encoding, and Hybrid Encoding. We illustrate the architectural variations within these categories and their corresponding encoder-decoder configurations.

Diffusion

Name	Title	Venue	Date	Code	Demo
UniModel	UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generation	arXiv	2025/11/21	-	-
Lavida-O	Lavida-O: Elastic Large Masked Diffusion Models for Unified Multimodal Understanding and Generation	arXiv	2025/09/24	Github	-
Muddit	Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Model	arXiv	2025/05/23	Github	-
FUDOKI	FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocities	arXiv	2025/05/20	-	-
MMaDA	MMaDA: Multimodal Large Diffusion Language Models	arXiv	2025/05/21	Github	Demo
UniDisc	Unified Multimodal Discrete Diffusion	arXiv	2025/03/20	Github	-
Dual Diffusion	Dual Diffusion for Unified Image Generation and Understanding	arXiv	2024/12/31	Github	-

MLLM AR

b-1: Pixel Encoding

Name	Title	Venue	Date	Code	Demo
Emu3.5	Emu3.5: Native Multimodal Models are World Learners	arXiv	2025/10/30	Github	-
Uni-X	Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal Models	ICLR	2025/09/29	Github	-
OneCat	OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generation	arXiv	2025/09/03	Github	-
Selftok	Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoning	arXiv	2025/05/12	Github	-
TokLIP	TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generation	arXiv	2025/05/08	Github	-
Harmon	Harmonizing Visual Representations for Unified Multimodal Understanding and Generation	arXiv	2025/03/27	Github	Demo
UGen	UGen: Unified Autoregressive Multimodal Model with Progressive Vocabulary Learning	arXiv	2025/03/27	-	-
SynerGen-VL	SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Folding	arXiv	2024/12/12	-	-
Liquid	Liquid: Language Models are Scalable and Unified Multi-modal Generators	arXiv	2024/12/05	Github	Demo
Orthus	Orthus: Autoregressive Interleaved Image-Text Generation with Modality-Specific Heads	arXiv	2024/11/28	Github	-
MMAR	MMAR: Towards Lossless Multi-Modal Auto-Regressive Probabilistic Modeling	arXiv	2024/10/14	-	-
Emu3	Emu3: Next-Token Prediction is All You Need	arXiv	2024/09/27	Github	Demo
ANOLE	ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generation	arXiv	2024/07/08	Github	-
Chameleon	Chameleon: Mixed-Modal Early-Fusion Foundation Models	arXiv	2024/05/16	Github	-
LWM	World Model on Million-Length Video And Language With Blockwise RingAttention	ICLR	2024/02/13	Github	-

b-2: Semantic Encoding

Name	Title	Venue	Date	Code	Demo
MammothModa2	MammothModa2: A Unified AR-Diffusion Framework for Multimodal Understanding and Generation	arXiv	2025/11/23	GitHub	-
Ming-UniVision	Ming-UniVision: Joint Image Understanding and Generation with a Unified Continuous Tokenizer	arXiv	2025/10/08	GitHub	-
Bifrost-1	Bifrost-1: Bridging Multimodal LLMs and Diffusion Models with Patch-level CLIP Latents	arXiv	2025/08/08	GitHub	-
Qwen-Image	Qwen-Image Technical Report	arXiv	2025/08/04	GitHub	Demo
X-Omni	X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Again	arXiv	2025/07/29	GitHub	Demo
Ovis-U1	Ovis-U1 Technical Report	arXiv	2025/06/28	GitHub	Demo
UniCode$^2$	UniCode$^2$: Cascaded Large-scale Codebooks for Unified Multimodal Understanding and Generation	arXiv	2025/06/20	-	-
OmniGen2	OmniGen2: Exploration to Advanced Multimodal Generation	arXiv	2025/06/18	Github	Demo
Tar	Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations	arXiv	2025/06/18	Github	Demo
UniFork	UniFork: Exploring Modality Alignment for Unified Multimodal Understanding and Generation	arXiv	2025/06/17	Github	-
UniWorld	UniWorld: High-Resolution Semantic Encoders for Unified Visual Understanding and Generation	arXiv	2025/06/03	Github	-
Pisces	Pisces: An Auto-regressive Foundation Model for Image Understanding and Generation	arXiv	2025/06/10	Github	-
DualToken	DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabularies	arXiv	2025/03/18	Github	-
UniTok	UniTok: A Unified Tokenizer for Visual Generation and Understanding	arXiv	2025/02/27	Github	Demo
QLIP	QLIP: Text-Aligned Visual Tokenization Unifies Auto-Regressive Multimodal Understanding and Generation	arXiv	2025/02/05	Github	-
MetaMorph	MetaMorph: Multimodal Understanding and Generation via Instruction Tuning	arXiv	2024/12/18	Github	-
ILLUME	ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhance	arXiv	2024/12/09	-	-
PUMA	PUMA: Empowering Unified MLLM with Multi-granular Visual Generation	arXiv	2024/10/17	Github	-
VILA-U	VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation	ICLR	2024/09/06	Github	Demo
Mini-Gemini	Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models	arXiv	2024/03/27	Github	Demo
MM-Interleaved	MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer	arXiv	2024/01/18	Github	-
VL-GPT	VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generation	arXiv	2023/12/14	Github	-
Emu2	Generative Multimodal Models are In-Context Learners	CVPR	2023/12/10	Github	Demo
DreamLLM	DreamLLM: Synergistic Multimodal Comprehension and Creation	ICLR	2023/09/20	Github	-
LaVIT	Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization	ICLR	2023/09/09	Github	-
Emu	Emu: Generative Pretraining in Multimodality	ICLR	2023/07/11	Github	Demo

b-3: Learnable Query Encoding

Name	Title	Venue	Date	Code	Demo
UniPic 2.0	Skywork UniPic 2.0: Building Kontext Model with Online RL for Unified Multimodal Model	arXiv	2025/09/04	Github	-
TBAC-UniImage	TBAC-UniImage: Unified Understanding and Generation by Ladder-Side Diffusion Tuning	arXiv	2025/08/11	Github	-
UniLIP	UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editing	arXiv	2025/07/31	-	-
OpenUni	OpenUni: A Simple Baseline for Unified Multimodal Understanding and Generation	arXiv	2025/05/23	Github	-
BLIP3-o	BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset	arXiv	2025/05/14	Github	-
Ming-Lite-Uni	Ming-Lite-Uni: Advancements in Unified Architecture for Natural Multimodal Interaction	arXiv	2025/05/05	Github	-
Nexus-Gen	Nexus-Gen: A Unified Model for Image Understanding, Generation, and Editing	arXiv	2025/04/30	Github	Demo
MetaQueries	Transfer between Modalities with MetaQueries	arXiv	2025/04/08	-	-
SEED-X	SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation	arXiv	2024/04/22	Github	Demo
SEED-LLaMA	Making LLaMA SEE and Draw with SEED Tokenizer	ICLR	2023/10/02	Github	Demo
SEED	Planting a SEED of Vision in Large Language Model	arXiv	2023/07/16	Github	Demo

b-4: Hybrid Encoding (Pseduo)

Name	Title	Venue	Date	Code	Demo
Skywork UniPic	Skywork UniPic: Unified Autoregressive Modeling for Visual Understanding and Generation	arXiv	2025/08/05	Github	Demo
MindOmni	MindOmni: Unleashing Reasoning Generation in Vision Language Models with RGPO	arXiv	2025/05/13	Github	Demo
UniFluid	Unified Autoregressive Visual Generation and Understanding with Continuous Tokens	arXiv	2025/03/17	-	-
OmniMamba	OmniMamba: Efficient and Unified Multimodal Understanding and Generation via State Space Models	arXiv	2025/03/11	Github	-
Janus-Pro	Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling	arXiv	2025/01/29	Github	Demo
Janus	Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation	arXiv	2024/10/17	Github	Demo

b-5: Hybrid Encoding (Joint)

Name	Title	Venue	Date	Code	Demo
Show-o2	Show-o2: Improved Native Unified Multimodal Models	arXiv	2025/06/15	Github	-
UniToken	UniToken: Harmonizing Multimodal Understanding and Generation through Unified Visual Encoding	CVPRW	2025/04/06	Github	-
VARGPT-v1.1	VARGPT-v1.1: Improve Visual Autoregressive Large Unified Model via Iterative Instruction Tuning and Reinforcement Learning	arXiv	2025/04/03	Github	-
ILLUME+	ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinement	arXiv	2025/04/02	Github	-
SemHiTok	SemHiTok: A Unified Image Tokenizer via Semantic-Guided Hierarchical Codebook for Multimodal Understanding and Generation	arXiv	2025/03/06	-	-
VARGPT	VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Model	arXiv	2025/01/21	Github	-
TokenFlow	TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation	CVPR	2024/12/04	Github	-
MUSE-VL	MUSE-VL: Modeling Unified VLM through Semantic Discrete Encoding	arXiv	2024/11/26	-	-

MLLM AR-Diffusion

c-1: Pxiel Encoding

Name	Title	Venue	Date	Code	Demo
TUNA	Tuna: Taming Unified Visual Representations for Native Unified Multimodal Models	arXiv	2025/12/01	Github	-
LMFusion	LMFusion: Adapting Pretrained Language Models for Multimodal Generation	arXiv	2024/12/19	-	-
MonoFormer	MonoFormer: One Transformer for Both Diffusion and Autoregression	arXiv	2024/09/24	Github	-
Show-o	Show-o: One Single Transformer to Unify Multimodal Understanding and Generation	ICLR	2024/08/22	Github	Demo
Transfusion	Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model	ICLR	2024/08/20	Github	-

c-2: Hybrid Encoding (Pseduo)

Name	Title	Venue	Date	Code	Demo
EMMA	EMMA: Efficient Multimodal Understanding, Generation, and Editing with a Unified Architecture	arXiv	2025/12/15	Github	-
HBridge	HBridge: H-Shape Bridging of Heterogeneous Experts for Unified Multimodal Understanding and Generation	arXiv	2025/11/25	-	-
LightFusion	LightFusion: A Light-weighted, Double Fusion Framework for Unified Multimodal Understanding and Generation	arXiv	2025/10/27	Github	-
BAGEL	Emerging Properties in Unified Multimodal Pretraining	arXiv	2025/05/20	Github	Demo
Mogao	Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generation	arXiv	2025/05/08	-	-
JanusFlow	JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation	arXiv	2024/11/12	Github	Demo

Any-to-Any Multimodal models

Name	Title	Venue	Date	Code	Demo
LongCat-Flash-Omni	LongCat-Flash-Omni Technical Report	arXiv	2025/11/28	Github	-
Ming-Flash-Omni	Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generation	arXiv	2025/10/28	Github	-
Qwen3-Omni	Qwen3-Omni Technical Report	arXiv	2025/09/22	Github	-
Ming-Omni	Ming-Omni: A Unified Multimodal Model for Perception and Generation	arXiv	2025/06/09	Github	-
M2-omni	M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performance	arXiv	2025/02/26	-	-
OmniFlow	OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flows	CVPR	2024/12/02	Github	-
Spider	Spider: Any-to-Many Multimodal LLM	arXiv	2024/11/14	Github	-
MIO	MIO: A Foundation Model on Multimodal Tokens	arXiv	2024/09/26	Github
X-VILA	X-VILA: Cross-Modality Alignment for Large Language Model	arXiv	2024/05/29	-	-
AnyGPT	AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling	arXiv	2024/02/19	Github	-
Video-LaVIT	Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization	ICML	2024/02/05	Github	-
Unified-IO 2	Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action	CVPR	2023/12/28	Github	-
NExT-GPT	NExT-GPT: Any-to-Any Multimodal LLM	ICML	2023/09/11	Github	-

Benchmark for Evaluation

Benchmarks on Understanding Tasks

Name	Paper	Venue	Date	Code
General-Bench	On Path to Multimodal Generalist: General-Level and General-Bench	ICML	2025/05/07	Github
MM-Vet v2	MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities	arXiv	2024/08/01	Github
OwlEval	mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality	arXiv	2024/04/27	Github
oVQA	Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy	ICLR	2024/02/11	Github
SEED-Bench-2	SEED-Bench-2: Benchmarking Multimodal Large Language Models	arXiv	2023/11/28	Github
MMMU	MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI	CVPR	2023/11/27	Github
MM-Vet	MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities	ICML	2023/08/04	Github
SEED-Bench	SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension	CVPR	2023/07/30	Github
MMBench	MMBench: Is Your Multi-modal Model an All-around Player?	ECCV	2023/07/12	Github
LAMM	LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark	NeurIPS	2023/06/11	Github
HaluEval	HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models	EMNLP	2023/05/19	Github
GQA	GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering	CVPR	2019/02/25	Github
VQA	VQA: Visual Question Answering	ICCV	2015/05/03	ProjectPage

Benchmarks on Image Generation Tasks

Name	Paper	Venue	Date	Code
GenExam	GenExam: A Multidisciplinary Text-to-Image Exam	arxiv	2025/09/18	Github
CVTG	TextCrafter: Accurately Rendering Multiple Texts in Complex Visual Scenes	arxiv	2025/08/05	Github
OneIG-Bench	OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation	arxiv	2025/06/26	Github
ComplexBench-Edit	ComplexBench-Edit: Benchmarking Complex Instruction-Driven Image Editing via Compositional Dependencies	arxiv	2025/06/15	Github
EditInspector	EditInspector: A Benchmark for Evaluation of Text-Guided Image Edits	arxiv	2025/06/11	-
ByteMorph-Bench	ByteMorph: Benchmarking Instruction-Guided Image Editing with Non-Rigid Motions	arxiv	2025/06/03	Github
RefEdit-Bench	RefEdit: A Benchmark and Method for Improving Instruction-based Image Editing Model on Referring Expressions	arxiv	2025/06/03	Github
WISE	WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generation	arxiv	2025/05/27	Github
ImgEdit-Bench	ImgEdit: A Unified Image Editing Dataset and Benchmark	arxiv	2025/05/26	Github
MMIG-Bench	MMGen-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Models	arxiv	2025/05/26	Github
KRIS-Bench	KRIS-Bench: Benchmarking Next-Level Intelligent Image Editing Models	NeurIPS 2025	2025/05/22	Github
CompBench	CompBench: Benchmarking Complex Instruction-guided Image Editing	arxiv	2025/05/18	-
WorldGenBench	WorldGenBench: A World-Knowledge-Integrated Benchmark for Reasoning-Driven Text-to-Image Generation	arxiv	2025/05/02	HuggingFace
GEdit-Bench	Step1X-Edit: A Practical Framework for General Image Editing	arXiv	2025/04/28	Github
DreamBench++	DreamBench++: A Human-Aligned Benchmark for Personalized Image Generation	ICLR	2025/03/09	Github
T2I-CompBench++	T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generation	TPAMI	2025/03/08	Github
IE-Bench	IE-Bench: Advancing the Measurement of Text-Driven Image Editing for Human Perception Alignment	arXiv	2025/01/17	-
AnyEdit	AnyEdit: Mastering Unified High-Quality Image Editing for Any Idea	CVPR	2024/11/24	Github
I2EBench	I2EBench: A Comprehensive Benchmark for Instruction-based Image Editing	NeurIPS	2024/08/26	Github
ConceptMix	ConceptMix: A Compositional Image Generation Benchmark with Controllable Difficulty	NeurIPS	2024/08/26	Github
GenAI-Bench	GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation	CVPR	2024/06/19	Github
Commonsense-T2I	Commonsense-T2I Challenge: Can Text-to-Image Generation Models Understand Commonsense?	COLM	2024/06/11	Github
HQ-Edit	HQ-Edit: A High-Quality Dataset for Instruction-based Image Editing	ICLR	2024/04/15	Github
VQAScore	Evaluating Text-to-Visual Generation with Image-to-Text Generation	ECCV	2024/04/01	Github
FlashEval	FlashEval: Towards Fast and Accurate Evaluation of Text-to-image Diffusion Generative Models	CVPR	2024/03/25	Github
DPG-Bench	ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment	arxiv	2024/03/08	Github
Reason-Edit	SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models	CVPR	2023/12/11	Github
Emu Edit	Emu Edit: Precise Image Editing via Recognition and Generation Tasks	CVPR	2023/11/16	HuggingFace
HEIM	Holistic Evaluation of Text-To-Image Models	NeurIPS	2023/11/07	Github
DSG-1k	Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation	ICLR	2023/10/27	Github
GenEval	GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment	NeurIPS	2023/10/17	Github
EditVal	EditVal: Benchmarking Diffusion Based Text-Guided Image Editing Methods	arXiv	2023/10/03	Github
T2I-CompBench	T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation	NeurIPS	2023/07/12	Github
DreamSim	DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data	NeurIPS	2023/06/15	Github
MagicBrush	MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing	NeurIPS	2023/06/16	Github
MultiGen-20M	UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild	NeurIPS	2023/05/18	Github
HRS-Bench	HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Models	ICCV	2023/04/11	Github
TIFA	TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering	ICCV	2023/03/21	Github
EditBench	Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting	CVPR	2022/12/13	ProjectPage
PartiPrompts	[Scaling Autoregre