Awesome-Multimodal-Modeling

Awesome Multimodal Modeling

📢 News

• [2026-10-19] Updated the curated model and design-analysis coverage; added a dedicated Benchmarks & Evaluation section and matching website collection, with explicit date and release-status qualifications.
• [2026-06-06] 🚀 We released two new open-source model collections:
  • Unified Multimodal Model Zoo (UMM Zoo) – a curated collection of unified multimodal models for understanding, generation, reasoning, and editing.
  • Native Multimodal Model Zoo (NMM Zoo) – a curated collection of native multimodal models that natively integrate perception, reasoning, and generation.
• [2026-04-13] ⭐ The repository has already gained over 100 stars in just one day! Thank you all for the incredible support. We will keep updating this list with more cutting-edge models and resources. Your continued stars and PRs are warmly welcomed!
• [2026-04-12] 🎉 We are excited to launch Awesome Multimodal Modeling — a curated reading list organized by architectural paradigms. A comprehensive survey paper is coming soon! Stay tuned.

Table of Contents

• Awesome Multimodal Modeling
  • 📢 News
  • Table of Contents
  • About This List
    • At a Glance
    • Curation Principles
  • 1. Introduction & Definitions
    • 1.1 Multimodal Model Evolution Stages
      • Traditional Multimodal Models
      • Multimodal Large Language Models (MLLMs)
      • Unified Multimodal Models (UMMs)
      • Native Multimodal Models (NMMs)
        • NMM — Early Fusion
        • NMM — Late Fusion
    • 1.2 Scope & Taxonomy
    • 1.3 Architecture Diagrams
  • 2. Traditional Multimodal Models
    • 2.1 Multimodal Representations & Alignment
      • Multimodal Representations
      • Multimodal Fusion
      • Multimodal Alignment
    • 2.2 Multimodal Pretraining
  • 3. Multimodal Large Language Models (MLLMs)
    • 3.1 Taxonomy Based on Vision Adapter
      • MLP/Others Projector
      • Q-Former
      • Cross-Attention
      • Hybrid Adaptor
      • Interface Unspecified
    • 3.2 Omni MLLMs
  • 4. Unified Multimodal Models (UMMs)
    • 4.1 Taxonomy by Generation Paradigm
      • Diffusion-Based UMMs
      • Autoregressive (AR) UMMs
        • Pixel Encoding
        • Semantic Encoding
        • Learnable Query Encoding
        • Hybrid Encoding (Pseudo)
        • Hybrid Encoding (Joint)
      • Hybrid (AR + Diffusion) UMMs
        • Pixel Encoding
        • Hybrid Encoding
    • 4.2 Any-to-Any / Omni UMMs
    • 4.3 Related Generators (U+G Unverified)
  • 5. Native Multimodal Models (NMMs)
    • 5.1 Design Analyses & Scaling Laws
    • 5.2 Early Fusion NMMs
    • 5.3 Encoder-Based NMMs
    • 5.4 Any-to-Any / Omni NMMs
    • 5.5 Training-Native Models
  • 6. Closed-Source Multimodal Models
    • Year 2026
    • Year 2025
    • Year 2024
    • Year 2023
  • 7. Resources
    • 7.1 Related Awesome Lists
    • 7.2 Slides & Survey Papers
    • 7.3 Code Repositories & Tools
    • 7.4 Benchmarks & Evaluation
      • Unified Understanding & Generation
      • Interleaved & Any-to-Any Evaluation
      • General Multimodal & Robustness
      • Image Generation Evaluation
  • How to Contribute
    • Validation Rules
    • Entry Format
  • Citation
  • License

About This List

In this section: At a Glance · Curation Principles

This repository provides a structured, community-maintained survey of multimodal models, covering the full evolutionary arc from early fusion methods to today's natively-trained omni-models. We emphasize precise architectural definitions and classification, especially for the often-conflated categories of Unified Multimodal Models (UMMs) and Native Multimodal Models (NMMs).

Scope: Primary focus on image + text modalities; audio/video/3D are annotated where present. Omni/any-to-any models are marked with Omni.

At a Glance

Dimension	Coverage
Primary scope	Image + text multimodal models, with explicit annotations for video, audio, and omni extensions
Core taxonomy	Traditional multimodal models, MLLMs, UMMs, and NMMs with explicit architecture/training qualifications
Key distinction	U+G unification for UMMs; architectural and optimization coupling for NMMs, with overlapping membership allowed
What makes this repo different	Architecture-first categorization, fusion-aware definitions, and curated links to adjacent awesome lists
Intended audience	Researchers, students, and engineers building or surveying multimodal systems

Curation Principles

Coverage audit: the additions and release-status updates in this revision were checked against primary sources through 2026-10-19. This is a curated selection, not an exhaustive index of every paper published by that date.

Principle	Rule
Source quality	Prefer official conference proceedings, OpenReview, ACL Anthology, CVF Open Access, arXiv, and official project pages
Classification policy	Category assignment is based on this repository's architecture-first definitions, which may differ from authors' own branding
Venue policy	Prefer the published venue; label author-reported acceptances where proceedings have not been checked. Keep first-paper dates separate from conference years
Date policy	First public: YYYY-MM-DD records the earliest verified exact-date public announcement; retain month precision where only a month is known. Distinguish paper submission, API availability, weights release and revision dates. Website year filters use the venue/release year; chronological sorting uses the explicit first-public date where recorded, otherwise the dated venue/release
Availability policy	Distinguish code, downloadable weights, API previews and future promises; record model-weight licenses separately from code licenses
Scope discipline	Models, benchmarks, datasets, and analysis papers are tracked separately to avoid mixing artifacts
Inclusion bar	We prioritize landmark papers, broadly adopted benchmarks, open implementations, or papers that clarify important taxonomy boundaries

Classification note: for ambiguous models sitting between MLLM, UMM, and NMM, this list records the category that best matches the training recipe and architectural coupling, not just the paper title. Architecture-native and training-native qualifications distinguish fusion structure from training history.

---

1. Introduction & Definitions

In this section: 1.1 Multimodal Model Evolution Stages · 1.2 Scope & Taxonomy · 1.3 Architecture Diagrams

1.1 Multimodal Model Evolution Stages

Subtopics: Traditional Multimodal Models · Multimodal Large Language Models (MLLMs) · Unified Multimodal Models (UMMs) · Native Multimodal Models (NMMs)

We use the following precise, architecture-first definitions throughout this list. Understanding these distinctions is critical for correctly classifying modern models.

Traditional Multimodal Models

Pre-2023 mainstream era

Representation alignment or fusion of modality-specific processing streams, without a large language model backbone. Covers cross-modal retrieval, classification and captioning; alignment-only dual encoders and models with joint fusion layers are distinguished by their architectures. Examples: CLIP, ALIGN, ViLBERT, BLIP.

Multimodal Large Language Models (MLLMs)

Pretrained-backbone multimodal language models

Combine a pretrained visual backbone or visual abstractor (e.g., ViT/CLIP/SigLIP, Q-Former, cross-attention adapter) with a pretrained LLM through a connector. The defining property is inheritance from strong pretrained unimodal backbones rather than joint multimodal pretraining from scratch. These models are primarily text-output understanding/reasoning systems, even when auxiliary generators are attached externally.

Key characteristics:

• ✅ Pretrained visual encoder / abstractor
• ✅ Pretrained LLM backbone
• ✅ Connector layer or cross-attention bridge
• ❌ No end-to-end multimodal pretraining from scratch
• ❌ No native image generation inside the same backbone

Examples: LLaVA, Qwen-VL, InternVL, MiniCPM-V, CogVLM

Unified Multimodal Models (UMMs)

Single framework for Understanding + Generation (U+G)

A single framework that handles both multimodal understanding and visual generation. UMMs may reuse pretrained components or modular tokenizers; the defining feature is U+G unification, not whether the model is trained from scratch.

Key characteristics:

• ✅ Unified understanding + generation
• ✅ Shared model interface or shared backbone for U+G
• ⚠️ May use pretrained components
• ⚠️ May use decoupled encoders / modular tokenizers
• ⚠️ If a model is also natively trained from scratch, its architectural details belong primarily in NMMs (§5)

Examples: Show-o, Janus, OpenUni, BAGEL, BLIP3-o

Native Multimodal Models (NMMs)

Architectural integration and multimodal optimization

NMMs are described along two dimensions: architectural nativity captures how early and persistently modalities share core computation; optimization nativity captures when multimodal objectives enter training and which core parameters they optimize. Entries that establish only one dimension are qualified as architecture-native or training-native. Training from scratch provides strong optimization evidence, but is not assumed for every entry. UMM and NMM membership can overlap.

Key characteristics:

• ✅ Fusion structure and training history are recorded separately
• ✅ Pretrained components and initialization are annotated per model
• ✅ Architecture-native entries may inherit pretrained backbones
• ✅ Input: text tokens + image patches/tokens
• ✅ Output: text (understanding focus; generation optional)

NMMs are described by fusion topology and input architecture, with initialization recorded separately. §5.2 records supported early-fusion architectures; §5.3 groups encoder-based systems without inferring fusion depth from encoder presence. Training-native entries without a confirmed architectural grouping are listed separately in §5.5:

NMM — Early Fusion

Modality tokens or embeddings enter shared core computation before the first shared backbone block. Separate tokenizers, input encoders, modality-specific experts, or output decoders can remain; their presence does not by itself imply late fusion.

• Shared backbone computation over multimodal states
• Discrete tokens, continuous latents, or pixel patches
• Modality interaction from the first shared block
• Initialization and trainable components annotated separately
• Examples: Chameleon, Emu3, Transfusion; Show-o, Show-o2, OneCat, and Llama 4 carry architecture-native qualifications

NMM — Late Fusion

Cross-modal interaction begins only after modality-specific backbone layers, rather than at the first shared block. This is a claim about where modalities interact, independent of whether the components were pretrained or initialized from scratch.

• Identify the modality-specific stages and the first interaction layer
• Record initialization and multimodal optimization separately
• A dedicated vision encoder alone does not establish this topology; encoder-based systems with unverified fusion depth are grouped in §5.3

1.2 Scope & Taxonomy

Multimodal Models
├── 2. Traditional Multimodal Models
│   ├── 2.1 Multimodal Representations & Alignment
│   │   ├── Multimodal Representations
│   │   ├── Multimodal Fusion
│   │   └── Multimodal Alignment
│   └── 2.2 Multimodal Pretraining
├── 3. Multimodal Large Language Models (MLLMs)
│   ├── 3.1 Taxonomy Based on Vision Adapter
│   └── 3.2 Omni MLLMs
├── 4. Unified Multimodal Models (UMMs)
│   ├── 4.1 Taxonomy by Generation Paradigm
│   │   ├── Diffusion-Based UMMs
│   │   ├── Autoregressive (AR) UMMs
│   │   │   ├── Pixel Encoding
│   │   │   ├── Semantic Encoding
│   │   │   ├── Learnable Query Encoding
│   │   │   ├── Hybrid Encoding (Pseudo)
│   │   │   └── Hybrid Encoding (Joint)
│   │   └── Hybrid (AR + Diffusion) UMMs
│   │       ├── Pixel Encoding
│   │       └── Hybrid Encoding
│   ├── 4.2 Any-to-Any / Omni UMMs
│   └── 4.3 Related Generators (U+G Unverified)
└── 5. Native Multimodal Models (NMMs)
    ├── 5.1 Design Analyses & Scaling Laws
    ├── 5.2 Early Fusion NMMs
    ├── 5.3 Encoder-Based NMMs
    ├── 5.4 Any-to-Any / Omni NMMs
    └── 5.5 Training-Native Models

1.3 Architecture Diagrams

The NMM diagrams below distinguish fusion topology from input architecture. The early-fusion diagram illustrates a from-scratch variant; initialization and input interfaces are specified per model in §5. Encoder-based entries do not inherit a strict late-fusion label from these schematic examples.

┌─────────────────────────────────────────────────────────────────┐
│           TRADITIONAL MULTIMODAL MODEL                          │
│                                                                 │
│  [Image] ──► [CNN/ViT Encoder] ──┐                              │
│                                  ├──► [Fusion] ──► [Output]     │
│  [Text]  ──► [LSTM/BERT]       ──┘                              │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│           MLLM — PRETRAINED BACKBONES + VISUAL INTERFACE         │
│                                                                 │
│  [Image] ──► [Pretrained ViT/CLIP] ──► [Projector/Q-Former]     │
│                                                │                │
│                                                ▼                │
│  [Text]  ──────────────────────────► [Pretrained LLM] ──► [Text]│
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│           UMM — UNIFIED UNDERSTANDING + GENERATION              │
│                                                                 │
│  [Image/Text Input] ──► [Shared/Modular Tokenizer]              │
│                                │                                │
│                                ▼                                │
│                   [Unified Transformer]                         │
│                       │           │                             │
│                       ▼           ▼                             │
│             [Text Output]      [Image Output]                   │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│           NMM — EARLY FUSION (Trained from Scratch)             │
│                                                                 │
│  [Text tokens] ──┐                                              │
│                  └──► [Single Decoder Transformer] ──► [Text]   │
│  [Image patches ──► Linear Patchify] ──┘                        │
│   (raw pixels, minimal preprocessing)                           │
│   Multimodal interaction from Layer 1                           │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│           NMM — ENCODER-BASED INPUT ARCHITECTURE               │
│                                                                 │
│  [Image] ──► [Vision Encoder] ──► [Visual Interface]           │
│                                         │                       │
│                                         ▼                       │
│  [Text]  ──────────────────────► [Backbone] ──► [Text]         │
│   Initialization and first interaction layer recorded per model │
└─────────────────────────────────────────────────────────────────┘

---

2. Traditional Multimodal Models

In this section: 2.1 Multimodal Representations & Alignment · 2.2 Multimodal Pretraining

Pre-chat-MLLM and non-native multimodal systems that established the basic vocabulary of alignment, fusion, retrieval, captioning, and multimodal pretraining.

2.1 Multimodal Representations & Alignment

Subtopics: Multimodal Representations · Multimodal Fusion · Multimodal Alignment

Multimodal Representations

Paper	Venue	Links	Notes	Task
Identifiability Results for Multimodal Contrastive Learning	ICLR 2023	Paper	Theoretical identifiability analysis of contrastive multimodal learning	representation learning
Unpaired Vision-Language Pre-training via Cross-Modal CutMix	ICML 2022	Paper	Introduces CutMix-style augmentation for unpaired VLP	vision-language pretraining
Balanced Multimodal Learning via On-the-fly Gradient Modulation	CVPR 2022	Paper	Balances modality learning via dynamic gradient reweighting	multimodal optimization
FLAVA: A Foundational Language And Vision Alignment Model	arXiv 2021	Paper	Unified architecture for vision-language understanding and generation	foundation multimodal model
UniT: Multimodal Multitask Learning with a Unified Transformer	arXiv 2021	Paper	Single transformer for multiple multimodal tasks	multimodal multitask learning
Perceiver: General Perception with Iterative Attention	ICML 2021	Paper	General-purpose architecture for high-dimensional multimodal inputs	general multimodal architecture
Learning Transferable Visual Models From Natural Language Supervision	ICML 2021	Paper	Contrastive vision-language pretraining at scale	vision-language contrastive learning
VinVL: Revisiting Visual Representations in Vision-Language Models	arXiv 2021	Paper	Improved visual features for VL tasks	vision-language representation improvement
12-in-1: Multi-Task Vision and Language Representation Learning	CVPR 2020	Paper	Unified multi-task learning across 12 VL tasks	multi-task learning
Learning Video Representations using Contrastive Bidirectional Transformer	arXiv 2019	Paper	Contrastive transformer for video representation learning	video contrastive learning
OmniNet: A Unified Architecture for Multi-modal Multi-task Learning	arXiv 2019	Paper	Unified encoder-decoder for multimodal tasks	unified multimodal architecture
Learning Representations by Maximizing Mutual Information Across Views	arXiv 2019	Paper	InfoMax principle for cross-view representation learning	self-supervised learning
ViCo: Word Embeddings from Visual Co-occurrences	ICCV 2019	Paper	Learning word embeddings from visual context	vision-language embeddings
Learning Factorized Multimodal Representations	ICLR 2019	Paper	Factorized latent space for multimodal data	representation disentanglement
Deep Fragment Embeddings for Bidirectional Image Sentence Mapping	NeurIPS 2014	Paper	Fragment-level image-sentence alignment	vision-language alignment
DeViSE: A Deep Visual-Semantic Embedding Model	NeurIPS 2013	Paper	Early deep vision-to-language embedding model	vision-language embedding
Multimodal Deep Learning	ICML 2011	Paper	Foundational multimodal deep learning framework	multimodal deep learning

Multimodal Fusion

Paper	Venue	Links	Notes	Task
Robust Contrastive Learning against Noisy Views	arXiv 2022	Paper	Robust contrastive learning under noisy multi-view inputs	contrastive learning
Attention Bottlenecks for Multimodal Fusion	NeurIPS 2021	Paper	Introduces bottleneck attention mechanism for efficient multimodal fusion	multimodal fusion
VMLoc: Variational Fusion For Learning-Based Multimodal Camera Localization	AAAI 2021	Paper	Variational multimodal fusion for camera localization tasks	multimodal localization
Trusted Multi-View Classification	ICLR 2021	Paper	Confidence-aware weighting for multi-view classification	multi-view classification
Deep-HOSeq: Deep Higher-Order Sequence Fusion for Multimodal Sentiment Analysis	ICDM 2020	Paper	Higher-order sequence fusion for multimodal sentiment analysis	multimodal sentiment analysis
What Makes Training Multi-Modal Classification Networks Hard?	CVPR 2020	Paper	Analyzes optimization challenges in multimodal classification	theoretical/empirical analysis
DeepCU: Integrating Both Common and Unique Latent Information for Multimodal Sentiment Analysis	IJCAI 2019	Paper	Separates shared and private latent representations for fusion	multimodal sentiment analysis
XFlow: Cross-modal Deep Neural Networks for Audiovisual Classification	IEEE TNNLS 2019	Paper	Cross-modal feature exchange network for audio-visual tasks	audio-visual classification
MFAS: Multimodal Fusion Architecture Search	CVPR 2019	Paper	Neural architecture search for optimal multimodal fusion design	architecture search
The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision	ICLR 2019	Paper	Neuro-symbolic model combining perception and reasoning	neuro-symbolic learning
Efficient Low-rank Multimodal Fusion with Modality-Specific Factors	ACL 2018	Paper	Low-rank factorization for efficient multimodal fusion	efficient fusion
Memory Fusion Network for Multi-view Sequential Learning	AAAI 2018	Paper	Memory-based fusion across temporal multimodal sequences	sequential multimodal learning
Tensor Fusion Network for Multimodal Sentiment Analysis	EMNLP 2017	Paper	Tensor-based full interaction modeling across modalities	multimodal sentiment analysis

Multimodal Alignment

Paper	Venue	Links	Notes	Task
LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment	arXiv 2023	Paper Code	Language-centered alignment across image, video, audio, depth, thermal, and IMU modalities	multimodal alignment
CLIP	ICML 2021	Paper	400M+ image-text pairs; dual-encoder (Vision Transformer + Text Transformer); contrastive alignment of separate image/text embeddings; no joint image-text backbone	zero-shot image classification, retrieval
CoMIR: Contrastive Multimodal Image Representation for Registration	NeurIPS 2020	Paper	Contrastive learning for multimodal image registration alignment	multimodal alignment
Multimodal Transformer for Unaligned Multimodal Language Sequences	ACL 2019	Paper	Transformer-based alignment for unaligned multimodal sequences	sequence alignment
Temporal Cycle-Consistency Learning	CVPR 2019	Paper	Uses cycle-consistency for temporal cross-modal alignment	temporal alignment
Deep Canonical Correlation Analysis	ICML 2013	Paper	Deep learning extension of CCA for cross-view representation alignment	representation alignment

2.2 Multimodal Pretraining

Paper	Venue	Links	Notes	Task
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation	NeurIPS 2021 Spotlight	Paper	Momentum distillation for aligning vision-language representations before fusion	vision-language pretraining
Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling	CVPR 2021	Paper	Sparse frame sampling for efficient video-language pretraining	video-language pretraining
UniT: Multimodal Multitask Learning with a Unified Transformer	arXiv 2021	Paper	Unified transformer for multitask multimodal learning	unified multimodal pretraining
Large-Scale Adversarial Training for Vision-and-Language Representation Learning	NeurIPS 2020	Paper	Adversarial training improves robustness of vision-language representations	robust multimodal pretraining
Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision	EMNLP 2020	Paper	Grounds language tokens in visual context via voken supervision	vision-grounded language modeling
Integrating Multimodal Information in Large Pretrained Transformers	ACL 2020	Paper	Injects multimodal signals into large pretrained transformer architectures	multimodal transformer pretraining
VL-BERT: Pre-training of Generic Visual-Linguistic Representations	arXiv 2019	Paper	Joint vision-language BERT-style pretraining	vision-language pretraining
VisualBERT: A Simple and Performant Baseline for Vision and Language	arXiv 2019	Paper	Early unified transformer for vision-language understanding	vision-language pretraining
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks	NeurIPS 2019	Paper	Two-stream transformer for cross-modal vision-language learning	vision-language pretraining
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training	arXiv 2019	Paper	Cross-modal encoder for universal vision-language representations	vision-language pretraining
LXMERT: Learning Cross-Modality Encoder Representations from Transformers	EMNLP 2019	Paper	Cross-modality transformer encoder for vision-language reasoning	vision-language pretraining
VideoBERT: A Joint Model for Video and Language Representation Learning	ICCV 2019	Paper	Joint discrete token modeling for video and language	video-language pretraining

---

3. Multimodal Large Language Models (MLLMs)

In this section: 3.1 Taxonomy Based on Vision Adapter · 3.2 Omni MLLMs

Models that connect a pretrained visual encoder / abstractor to a pretrained LLM. Primarily text-output understanding and reasoning systems, defined by inherited pretrained unimodal backbones rather than multimodal pretraining from scratch.

3.1 Taxonomy Based on Vision Adapter

These groups describe the vision-to-language interface, rather than attention variants inside the language backbone. MLP/Others includes linear projections, spatial merging and pixel shuffle; Q-Former covers query-transformer connectors; Cross-Attention refers to bridges attending to separate visual features. Hybrid Adaptor covers combinations such as learned pooling plus projection, or projection plus visual experts inside the language backbone.

MLP/Others Projector

Paper	Venue	Links	Notes	Task
WeVisDoc: From Coverage to Capability for Robust End-to-End Document Parsing	arXiv 2026-09-17	Paper Code HF	Paper first submitted: 2026-09-17. Fine-tunes Qwen3-VL-2B/4B-Instruct, retaining their visual encoder and merger/projector; page images and instructions produce structured Markdown, LaTeX formulas and HTML tables. Coverage-driven and residual-error-driven training target degraded documents. Code and 2B/4B checkpoints available as of 2026-10-19.	document parsing, OCR, table/formula recognition, robustness
VideoScout: Learning Agentic Active Exploration with Adaptive Reasoning Pacing for Long Video Understanding	arXiv 2026-09-14	Paper	Paper first submitted: 2026-09-14. Qwen2.5-VL-7B initialization retains its vision encoder and projector; SFT and trajectory-level DAPO learn adaptive viewing speed, revisiting and note taking for long-video answers. Text/video input produces text and viewing actions; code and checkpoint release not established in the checked primary source as of 2026-10-19.	long-video understanding, active exploration, temporal reasoning
LFM2.5-VL-3B	Model release 2026-08-12	Announcement HF Implementation	First public: 2026-08-12. Pretrained LFM2.5-2.6B with SigLIP2 NaFlex 400M, pixel-unshuffle and a two-layer MLP projector; text/image/multi-image inputs produce text, grounding coordinates and tool calls. Weights and inference implementations available under the Liquid AI model license; aimed at low-latency edge use.	image understanding, multi-image reasoning, OCR, grounding, tool use, edge deployment
PaddleOCR-VL-1.6: Expanding the Frontier of Document Parsing with Under-Optimized Region Refinement and Progressive Post-Training	arXiv 2026-06-02	Paper Code HF	First public: 2026-05-28 (weights); paper first submitted 2026-06-02. 0.9B document VLM retaining the 1.5 native-resolution encoder, adaptive MLP connector and ERNIE language backbone; region refinement and progressive post-training improve recognition. Document-image/text inputs produce structured text, tables, formulas and spotting coordinates. Code and weights available.	document parsing, multilingual OCR, tables, formulas, text spotting
PaddleOCR-VL-1.5: Towards a Multi-Task 0.9B VLM for Robust In-the-Wild Document Parsing	arXiv 2026-01-29	Paper Code HF	First public: 2026-01-29 (paper and weights). 0.9B VLM combines a native-resolution visual encoder, adaptive MLP connector and ERNIE-4.5-0.3B; document-image/text inputs produce structured text and localized recognition. Adds seal recognition, text spotting and robustness training; full page parsing uses the accompanying layout pipeline. Code and Apache-2.0 checkpoints available.	document parsing, multilingual OCR, seal recognition, text spotting
GLM-4.6V / GLM-4.6V-Flash	Model release 2025-12-08	Announcement Code HF Flash	First public: 2025-12-08. 106B and 9B vision-language variants with 128K trained context; visual patch merging and gated MLP projection connect the vision path to the language backbone. Text/image/video inputs produce text and tool calls, with visual observations passed directly to tools. Code and MIT-licensed weights available; native tool use does not establish from-scratch multimodal training.	image/video understanding, document understanding, visual tool use
SmolVLM2	Model release 2025-02-20	Announcement HF Paper	First public: 2025-02-20. Video-trained 256M, 500M and 2.2B family extending the SigLIP/SmolLM projector design; text/image/multi-image/video inputs produce text. Uses video/image training mixtures and efficient visual compression for device deployment. Checkpoints and inference/fine-tuning support available under Apache 2.0; the shared family report was submitted 2025-04-07.	image/video understanding, multi-image reasoning, efficiency, on-device deployment
Idefics3	arXiv 2024	Paper HF	SigLIP vision encoder with pixel-shuffle spatial compression and projection to a Llama 3 backbone; replaces Idefics2’s Perceiver resampler.	visual understanding, document understanding
MiniMax-01: Scaling Foundation Models with Lightning Attention	arXiv 2025	Paper	MiniMax-VL-01 uses a 303M ViT, a two-layer MLP visual projector and MiniMax-Text-01; Lightning/Softmax hybrid attention describes the language backbone.	visual understanding
VideoJev: System One Model for Video Understanding and Interaction	arXiv 2026	Paper Code	4B / 35B-A3B; initialized from Qwen3.5-4B / Qwen3.6-35B-A3B, retaining their pretrained ViT, visual projection and language backbone; candidate-scoring heads unify temporal grounding and multiple-choice QA.	video QA, temporal grounding
Video-o3: Native Interleaved Clue Seeking for Long Video Multi-Hop Reasoning	ICML 2026	Paper Code	Qwen2.5-VL-7B (also Qwen3-VL-4B); inherits the pretrained Qwen ViT, visual projector and language backbone; interleaves reasoning with VideoCrop tool calls and learns from SFT plus trajectory-guided RL.	long-video reasoning, visual tool use
VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding	arXiv 2026	Paper Code	4B; I3D-ViT initialized from image-pretrained MoonViT, with a lightweight projector and Qwen3-4B text backbone; spatiotemporal visual compression supports generalist offline and proactive streaming understanding.	video understanding, temporal grounding, proactive streaming
TimeLens2: Generalist Video Temporal Grounding with Multimodal LLMs	arXiv 2026	Paper Code	2B / 4B / 8B; inherits the pretrained Qwen3-VL ViT, visual projector and language backbones; verified SFT and temporal-grounding GRPO adapt them to generalist interval localization.	video temporal grounding
OneStreamer: Unifying Perception, Memory, and Proactive Response in Streaming Video Interaction	arXiv 2026	Paper Code	4B; initialized from Qwen3-VL-4B-Instruct with its pretrained vision encoder frozen and the multimodal projector and language model fine-tuned; hierarchical caption memory and control states support proactive streaming interaction.	streaming video understanding, memory, proactive response
Ling-3.0-flash-VL	Model release 2026	HF	Open-weight VLM (MIT) extending Ling-3.0-flash with a ViT and a two-layer MLP projector; supports image and video inputs with text outputs. BF16 and quantized checkpoints belong to the same model entry	image/video understanding, document understanding, visual reasoning, GUI agents
LLaDA-UI	Model release 2026	Code HF	SigLIP-initialized native-resolution ViT connects to LLaDA2.0-mini-base through spatial 4-to-1 feature grouping and a two-layer MLP; block-wise diffusion generates text reasoning and GUI actions. Code and weights are available; checkpoint license is unspecified in the model card as of 2026-09-15	screenshot understanding, GUI grounding, mobile/desktop/web agents
Muse Glimmer-30B	Model release 2026	HF Implementation	Open-weight 30B model (Apache 2.0), distilled from Muse Spark; a dedicated Perception Encoder connects to the language backbone through an MLP adapter and linear projection; text and image inputs, text outputs	visual understanding, visual reasoning, coding, agentic tool use
DiffusionGemma	Google DeepMind, 2026	Model Card HF Implementation	Open-weight Gemma 4-based model (Apache 2.0); normalized visual features enter the language model through a linear projection, while an encoder-decoder architecture uses block-wise discrete diffusion for text generation. Image/video understanding with text output; no native image generation	visual understanding, video understanding, reasoning, efficient text generation
Mistral Medium 3.5	Mistral AI, 2026	Announcement HF Implementation	Open-weight 128B dense VLM under a Modified MIT license; a from-scratch vision encoder connects through a patch merger and MLP projector to the language backbone. Supports text/image inputs and text outputs; whole-model from-scratch multimodal training is not established	visual understanding, reasoning, coding, long-horizon agents
Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model	arXiv 2026	Paper Project Model	4B codec-native streaming VLM combining a from-scratch Mage-ViT visual encoder, two-layer MLP projector, and pretrained Qwen3-4B-Instruct-2507 decoder; codec-guided token selection reduces visual tokens by over 75%	image/video understanding, long video, proactive streaming
StepX-Edge: An On-Device UI Vision-Language Model via Architecture-Training-Deployment Co-Design	arXiv 2026	Paper	0.9B on-device UI VLM with UI-aware layered visual encoding and a progressive dimensionality projection connector; quantized deployment is validated on Snapdragon 8 Gen5	UI understanding, OCR, grounding, on-device deployment
MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe	arXiv 2025	Paper Code Model	Efficient 8B MLLM built with Qwen3-8B, SigLIP2-400M and a unified 3D-Resampler for compact image-video encoding	visual understanding, document/OCR, video understanding, edge deployment
Moondream 3.1	Model Release 2026	Project Model	Efficient sparse MoE VLM with 9B total and 2B active parameters; supports visual reasoning, query, caption, detection, pointing and segmentation	visual understanding, grounding, segmentation, edge deployment
LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training	arXiv 2025	Paper Code HF	4B / 8B; 85M mid-training data + 22M instruction data; ViT–MLP–LLM architecture with pretrained RICE-ViT, a two-layer MLP projector, and Qwen3 backbone; supports image, multi-image, and video	visual understanding, video
SAIL-VL2 Technical Report	arXiv 2025	Paper	Open-suite 2B/8B vision-language foundation model with SAIL-ViT, progressive multimodal training, SFT-RL thinking fusion, and strong image/video reasoning across 106 datasets	visual understanding, video, reasoning
Kwai Keye-VL 2.0 Technical Report	arXiv 2026	Paper	achieves state-of-the-art performance among models of similar scale, particularly excelling in fine-grained temporal localization	video understanding
Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders	arXiv 2026	Paper	LLM-initialized vision encoder (non-CLIP); text-to-vision weight reuse, generative-aligned visual features, optimized for dense perception.	visual understanding
Youtu-VL: Unleashing Visual Potential via Unified Vision-Language Supervision	arXiv 2026	Paper	SigLIP2 vision encoder with 2×2 Spatial Merge and a two-layer MLP projector to Youtu-LLM; unified autoregressive supervision over visual and text targets supports fine-grained understanding and dense prediction.	visual understanding, grounding, segmentation, depth estimation
STEP3-VL-10B Technical Report	arXiv 2026	Paper	10B-scale foundation multimodal; unified unfrozen pre-training + PaCoRe test-time scaling, frontier-level reasoning with compact footprint.	visual understanding
GLM-OCR	arXiv 2026	Paper	GLM-OCR is an efficient 0.9B-parameter compact multimodal model designed for real-world document understanding.	OCR, structured extraction
Kimi K2.5	arXiv 2026	Paper	joint text-vision pretraining, Agent Swarm framework; coding, vision, reasoning, agentic tasks; reduces latency by up to 4.5x	visual agentic intelligence, agentic, reasoning
Kwai Keye-VL 1.5 Technical Report	arXiv 2025	Paper	Adaptive Slow-Fast encoding; 8B parameter scale with 128K long-context; SOTA video reasoning & human-preference aligned.	visual understanding
olmOCR / olmOCR-2	arXiv 2025	Paper	Efficient low-VRAM OCR model based on Qwen2.5-VL fine-tune; excels at preserving semantic structure and markdown output	OCR, structured extraction
PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Model	arXiv 2025-10-16	Paper Code HF	Paper first submitted: 2025-10-16. Original 0.9B document VLM with a NaViT-style dynamic-resolution encoder, adaptive MLP connector and ERNIE-4.5-0.3B; document-image/text inputs produce multilingual text, table, formula and chart representations. Code and weights available; later 1.5 and 1.6 releases are listed separately.	OCR, multilingual document parsing
DeepSeek-OCR	arXiv 2025	Paper HF	Lightweight ~3B MoE vision model optimized for high-volume OCR, document digitization, charts and formulas; efficient inference	OCR, document
Kimi-VL	arXiv 2025	Paper HF	Projector + MoE backbone; long video/PDF/GUI, agentic capabilities, chain-of-thought vision reasoning	visual understanding, agentic, video
Seed1.5-VL Technical Report	arXiv 2025	Paper	20B MoE + 532M ViT; native-resolution vision-language foundation model; efficient asymmetric architecture.	visual understanding
Qwen3-VL	arXiv 2025	Paper HF	Frontier-grade vision/OCR (32+ languages), video analysis, agentic capabilities, strong multimodal reasoning; includes large MoE variants (e.g., 235B)	visual understanding, video, omni
SmolVLM: Redefining small and efficient multimodal models	arXiv 2025-04-07	Paper Announcement HF 256M	First public: 2024-11-26 (original release); family paper submitted 2025-04-07. Compact image-first SigLIP/SmolLM family using pixel-shuffle compression and projection; text/image/multi-image inputs produce text, with sampled-frame video explored in the original release. Code and Apache-2.0 checkpoints available; SmolVLM2 separately records dedicated video training and release.	visual understanding, efficiency, on-device deployment
LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning	arXiv 2025	Paper	Diffusion LLM as LLM backbone; vision encoder: SigLIP	visual understanding
jina-vlm	arXiv 2025	Paper HF	SigLIP2 + Qwen backbone with custom projector; optimized for semantic VQA, diagrams, scans and document semantics	visual understanding, VQA, document
Phi-4-Multimodal	arXiv 2025	Paper HF	Small-parameter (LoRA + projectors) multimodal; vision + speech support, efficient on-device deployment	visual understanding, on-device
Molmo / PixMo	CVPR 2025	Paper Code	Strong open-data/open-weight VLM pipeline	visual understanding
FastVLM: Efficient Vision Encoding for Vision Language Models	CVPR 2025	Paper	efficient multimodal visual encoding for on-device deployment	visual understanding, on-device
Qwen2.5-VL: Technical Report	arXiv 2025	Paper HF	Stronger document, grounding, and video capabilities	visual understanding
General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model	arXiv 2024	Paper HF/Code	Specialized end-to-end OCR model with grounding (boxes + points); strong on scientific papers, slides, and mixed visual-text docs	OCR, grounding
LLaVA-OneVision: Easy Visual Task Transfer	arXiv 2024	Paper Code	Single model for image, multi-image, and video transfer	visual understanding
MiniCPM-V: A GPT-4V Level MLLM on Your Phone	arXiv 2024	Paper Code	On-device efficient MLLM	visual understanding
NVILA: Efficient Frontier Visual Language Models	CVPR 2025	Paper	Efficient general purpose multimodal LLM; spatial and temporal "Scale then compress" design; vision encoder: SigLIP	visual understanding
GLM-4V	arXiv 2024	Paper	ViT-based vision encoder (EVA-02-CLIP-L); high-resolution input support (up to 1120x1120) via image tiling and a visual interface to the language model, optimized for document and OCR tasks	visual understanding
xGen-MM (BLIP-3)	arXiv 2024	Paper	Open training recipe, datasets, and safety-tuned variants	visual understanding
DeepSeek-VL2: Mixture-of-Experts Vision-Language Models	arXiv 2024	Paper Code	MoE VLM with dynamic tiling and efficient inference	visual understanding
Pixtral	arXiv 2024	Paper HF	12B open-weight model with strong instruction following, image+text understanding; competitive with larger open VLMs	visual understanding
Qwen2-VL	arXiv 2024	Paper HF	Dynamic resolution; native video	visual understanding
Cambrian-1: A Fully Open, Vision-Centric Exploration	NeurIPS 2024	Paper Code	Spatial Vision Aggregator	visual understanding
PaliGemma: A Versatile 3B VLM for Transfer	arXiv 2024	Paper HF	SigLIP encoder + Gemma backbone; strong transfer model	visual understanding
InternLM-XComposer2	arXiv 2024	Paper Code	Compositional visual grounding	visual understanding
Phi-3-Vision	arXiv 2024	Paper [HF](https://huggingface.co/microsoft/Phi-3-v