awesome-vlm-architectures

👁️‍🗨️ Awesome VLM Architectures

Awesome VLM Architectures is a citation-first visual catalog of 200 Vision-Language Model (VLM/MLLM) architectures, spanning contrastive encoders, multimodal LLMs, native multimodal models, unified understanding and generation, video, OCR, GUI agents, and embodied AI. Each entry links to primary sources and summarizes the model architecture, modality alignment or fusion, training stages, datasets, and distinctive design choices, with an architectural figure when available.

Use this repository to compare multimodal model families, trace architectural ideas over time, or retrieve grounded references for research and AI-agent workflows. The catalog includes a source-linked release timeline reviewed through October 27, 2026 and covers foundational systems such as CLIP and Flamingo alongside current multimodal reasoning and agentic models. Expand any model panel for its detailed architecture summary.

• 📙 Visit my other repo to try Vision Language Models on ComfyUI

Last reviewed: October 27, 2026 (Europe/Istanbul).

October 2026 review: additions since the August 1 review, earlier omissions, family updates, date evidence, and availability caveats. This is a curated architecture catalog, not an exhaustive index of every fine-tune or API version.

Contents

• Citation
• Models
• Release Timeline
• Architectures
• Important References

Citation

If this repository is useful in your work, you may cite it below. Please also cite the original paper for claims about any individual model; this catalog is a guide to the literature, not a substitute for it.

Created and maintained by Gökay Aydoğan at fal.ai (ORCID; [email protected]).

Architecture images are credited individually in the figure credits and remain subject to the rights described in the figure notice.

@misc{aydogan2024awesomevlmarchitectures,
  author       = {Gökay Aydoğan},
  title        = {Awesome VLM Architectures},
  year         = {2024},
  howpublished = {\url{https://github.com/gokayfem/awesome-vlm-architectures}},
  note         = {GitHub repository, fal.ai},
  url          = {https://github.com/gokayfem/awesome-vlm-architectures}
}

Models

All architecture panels are ordered by release date, newest first. Models released on the same day retain editorial catalog order.

2026: Nemotron 3.5 Parse | Nemotron 3.5 Super VL | Muse Spark Open | Step 5 Preview | Liquid d1 | GroundAnything | Holo4 | MiMo-V2.6 | Qwen3.8-Omni-Flash | Taichu 5.0 | DeepSeek-V4.1-Flash | LLaDA-UI | Ling-3.0-flash-VL | Jina-OCR-v1 | NeoMME | Qwen-Drive-1.0 | GLM-5.3-Flash | Qwen3.8-Flash-Next | dots3-note-preview | North Micro Vision | Muse Glimmer | Shieldstral | MODUS | Argus-Unified | Kimi K3 | Mage-VL | Apertus 1.5 | Audio-Visual Flamingo | Inkling | Hy-Embodied-VLM | MonkeyOCRv2 | DiffusionGemma | InternVideo3 | Keye-VL 2.0 | Zamba2-VL | MiniMax M3 | Cosmos 3 | Fara1.5 | Lance | ZAYA1-VL | LensVLM | Falcon Perception | Granite Vision 4.1 | GLM-5V-Turbo | Nemotron 3 Nano Omni | SenseNova-U1 and U1.5 | PLaMo 2.1-VL | Qwen3.5-Omni | EXAONE 4.5 | BidirLM and BidirLM-Omni | Gemma 4 | Penguin-VL | Phi-4-Reasoning-Vision | V-SONAR and V-LCM | Qwen3.5 | Youtu-VL | Kimi K2.5 and K2.6 | LightOnOCR-2 and LightOnOCR-3 | Step3-VL-10B

2025: Molmo2 | Mistral Large 3 | Ministral 3 | ERNIE 5.0 | Emu3.5 | Nemotron Nano V2 VL | DeepSeek-OCR | OmniVinci | PaddleOCR-VL | Qwen3-Omni | Qwen3-VL | Moondream 3 | Intern-S1 | LFM2-VL and LFM2.5-VL | Step3 | GLM-4.1V-Thinking | ERNIE 4.5-VL | Show-o2 | MiMo-VL | BAGEL | Seed1.5-VL | PerceptionLM | Perception Encoder | InternVL3 and InternVL3.5 | Kimi-VL | Llama 4 Scout and Maverick | Qwen2.5-Omni | Gemma 3 | Aya Vision | Phi-4-multimodal | SigLIP 2 | EVEv2 | Qwen2.5-VL | VideoLLaMA 3 | UI-TARS | MiniMax-01 | MiniCPM-o-2.6 | Eagle 2 | Sa2VA

2024: VideoChat-Flash | OmniVLM | Apollo | DeepSeek-VL2 | Maya | InternVL 2.5 | PaliGemma 2 | ShowUI | SmolVLM | AIMv2 | LLaVA-CoT | LLM2CLIP | Tarsier2 | Janus and Janus-Pro | ARIA | Emu3 | Molmo and PixMo | Llama 3.2-Vision | NVLM | Pixtral 12B | VILA-U | Qwen2-VL | EAGLE | Show-o | Idefics3-8B | Transfusion | mPLUG-Owl3 | VITA | LLaVA-OneVision | VILA² | INF-LLaVA | SlowFast-LLaVA | EVLM | InternLM-XComposer-2.5 | OMG-LLaVA | Cambrian-1 | EVE | Ovis | Parrot | ConvLLaVA | Phi-3-Vision and Phi-3.5-Vision | CogVLM2 | Chameleon | PaliGemma | xGen-MM (BLIP-3) | MANTIS | Moondream-next | Idefics2 | InternLM-XComposer2-4KHD | MM1 | DeepSeek-VL | AnyGPT | SPHINX-X | LLaVA 1.6 | MiniCPM-V | MouSi | InternLM-XComposer2 | MoE-LLaVA | moondream1 and moondream2 | FireLLaVA | COSMO

2023: TinyGPT-V | MobileVLM | Alpha-CLIP | Nous-Hermes-2-Vision - Mistral 7B | SPHINX | Florence-2 | u-LLaVA | LLaVA-Plus | OtterHD | CoVLM | GLaMM | Fuyu-8B | PaLI-3 Vision Language Models | MiniGPT-v2 | BakLLaVA | Ferret | LLaVA 1.5 | CogVLM | MetaCLIP | Qwen-VL | IDEFICS | BLIVA | KOSMOS-2 | LaVIN | InstructBLIP | ImageBind | LLaVA | MiniGPT-4 | SigLIP | OpenFlamingo | PaLM-E | KOSMOS-1 | BLIP-2

2022: MULTIINSTRUCT | PaLI | Flamingo | BLIP

2021: GLIP | FROZEN | CLIP

2020: ViT

Release Timeline

Dates use the first documented official model release; when none is available, they use the paper's arXiv v1 submission or first technical report. Family point releases are folded into their first architecture release, and same-day entries retain catalog order.

Date	Architecture	Distinctive contribution
2026-10-17	Nemotron 3.5 Parse	Compact visual-prefix decoding for structured documents
2026-10-15	Nemotron 3.5 Super VL	Hybrid state-space MoE for long-context visual agents
2026-10-15	Muse Spark Open	Sparse multimodal experts with a shared perception encoder
2026-10-15	Step 5 Preview	Deep sparse experts with gated sliding-window and sparse GQA
2026-10-07	Liquid d1	Single-pass multimodal decisions without text decoding
2026-09-30	GroundAnything	Autoregressive and block-diffusion spatial grounding
2026-09-28	Holo4	Visual computer-use agents trained across GUI and tool environments
2026-09-22	MiMo-V2.6	Hybrid sliding-window omni encoders and multi-teacher distillation
2026-09-18	Qwen3.8-Omni-Flash	Native omni agents with spatial audio and sparse long-context reasoning
2026-09-15	Taichu 5.0	Entropy-gated recurrent latent visual reasoning
2026-09-10	DeepSeek-V4.1-Flash	Causal encoder-decoder multimodality with shared sparse KV memory
2026-09-09	LLaDA-UI	Block-diffusion reasoning and actions for GUI agents
2026-09-04	Ling-3.0-flash-VL	Kimi Delta Attention and gated latent attention for vision
2026-09-02	Jina-OCR-v1	Compressed visual parsing with recursive multi-token drafting
2026-08-31	NeoMME	Single-tower bidirectional multimodal embeddings from masked diffusion
2026-08-31	Qwen-Drive-1.0	Shared visual reasoning with external 3D perception and planning
2026-08-26	GLM-5.3-Flash	Native multimodal sparse and linear attention with mHC
2026-08-26	Qwen3.8-Flash-Next	Sparse hybrid attention and off-accelerator n-gram memory
2026-08-14	dots3-note-preview	Omnimodal MoE with a sparse vision expert encoder
2026-08-12	North Micro Vision	Compact native-resolution vision with multilayer feature injection
2026-08-10	Muse Glimmer	Dense visual agents with local-global attention and diffusion drafting
2026-08-04	Shieldstral	Policy-conditioned visual moderation in one forward pass
2026-07-28	MODUS	Shared decoder context with autoregressive and flow-matching modality experts
2026-07-28	Argus-Unified	Hybrid continuous and discrete visual tokens for economical understanding and generation
2026-07-27	Kimi K3	Kimi Delta Attention, Attention Residuals, and extremely sparse LatentMoE routing
2026-07-27	Mage-VL	Codec-native selective video tokenization with a proactive event gate
2026-07-24	Apertus 1.5	Early fusion of discrete visual and audio tokens
2026-07-17	Audio-Visual Flamingo	Temporally aligned audio-video reasoning over long clips
2026-07-15	Inkling	Relative-position million-context multimodal MoE trained from scratch
2026-07-15	Hy-Embodied-VLM	Action-centric sparse-MoE reasoning for physical-world agents
2026-07-11	MonkeyOCRv2	Joint image-to-text and pixel-reconstruction pretraining for document vision
2026-06-10	DiffusionGemma	Cached causal context with a block-diffusion text decoder
2026-06-10	InternVideo3	Token-preserving latent KV compression and closed-loop video reasoning
2026-06-09	Keye-VL 2.0	DeepSeek Sparse Attention adapted to GQA-based long-video multimodality
2026-06-02	Zamba2-VL	Hybrid Mamba-2 and shared-attention blocks for efficient VLM inference
2026-06-01	MiniMax M3	Native multimodality with block-sparse grouped-query attention at million-token context
2026-05-31	Cosmos 3	Coupled autoregressive reasoner and diffusion generator for physical AI
2026-05-21	Fara1.5	Verified synthetic environments for screenshot-based computer use
2026-05-18	Lance	Shared-sequence understanding, generation, and editing with modality experts
2026-05-08	ZAYA1-VL	Vision-conditional LoRA and compressed convolutional attention in an open-data MoE
2026-05-07	LensVLM	Selective expansion of visually compressed text context
2026-05-03	Falcon Perception	Early fusion with hybrid attention and continuous mask heads
2026-04-29	Granite Vision 4.1	Window Q-Formers and dual DeepStack feature injection
2026-04-29	GLM-5V-Turbo	Perception integrated into reasoning, planning, tools, and execution
2026-04-28	Nemotron 3 Nano Omni	Hybrid MoE with dynamic visual resolution and native audio
2026-04-27	SenseNova-U1 and U1.5	Native pixel-space unified understanding and generation
2026-04-21	PLaMo 2.1-VL	Compact Japanese VQA and grounding for edge deployment
2026-04-17	Qwen3.5-Omni	Hybrid-attention Thinker-Talker and adaptive speech alignment
2026-04-09	EXAONE 4.5	Native multimodal pretraining with document-focused data and 256K context
2026-04-02	BidirLM and BidirLM-Omni	Converting causal decoders into bidirectional multimodal encoders
2026-03-31	Gemma 4	Dense and MoE native multimodality, including an encoder-free 12B design
2026-03-06	Penguin-VL	Text-LLM-initialized vision encoder and priority-aware token compression
2026-03-04	Phi-4-Reasoning-Vision	Mid-fusion compact VLM with explicit reasoning and direct-answer modes
2026-03-01	V-SONAR and V-LCM	Vision-language alignment and prediction in multilingual concept space
2026-02-16	Qwen3.5	Native early fusion with hybrid linear/full attention and sparse MoE variants
2026-01-27	Youtu-VL	Unified autoregressive visual tokens that emit dense vision outputs without task heads
2026-01-27	Kimi K2.5 and K2.6	Trillion-parameter native multimodal MoE for agents and computer use
2026-01-20	LightOnOCR-2 and LightOnOCR-3	Compact end-to-end OCR with document grounding
2026-01-14	Step3-VL-10B	Language-aligned perception encoder with 16-fold visual-token compression
2025-12-11	Molmo2	Open video understanding with pointing and object tracking
2025-12-02	Mistral Large 3	Large sparse multimodal experts with an integrated visual encoder
2025-12-02	Ministral 3	Dense multimodal backbones with compute-efficient training
2025-11-13	ERNIE 5.0	One autoregressive sparse MoE for text, images, video, audio, and generation
2025-10-30	Emu3.5	Native multimodal next-token learning with diffusion adaptation
2025-10-28	Nemotron Nano V2 VL	Hybrid state-space visual reasoning with efficient video sampling
2025-10-20	DeepSeek-OCR	DeepEncoder compresses high-resolution documents into very short visual contexts
2025-10-17	OmniVinci	Joint audio-visual alignment with grouped temporal embeddings
2025-10-16	PaddleOCR-VL	NaViT-style dynamic resolution with a compact ERNIE decoder for document parsing
2025-09-22	Qwen3-Omni	Thinker-Talker MoE with streaming multicodebook speech
2025-09-22	Qwen3-VL	DeepStack multi-level ViT fusion and explicit video timestamp alignment
2025-09-18	Moondream 3	Compact sparse visual experts with learned attention temperature
2025-08-21	Intern-S1	Scientific multimodality with dynamic tokenization and expert reasoning
2025-08-12	LFM2-VL and LFM2.5-VL	Short-convolution language backbones with native-resolution vision
2025-07-25	Step3	Model-system co-design for communication-efficient sparse-MoE multimodality
2025-07-01	GLM-4.1V-Thinking	Curriculum-sampled reinforcement learning for multimodal reasoning
2025-06-30	ERNIE 4.5-VL	Heterogeneous shared and modality-specific experts with isolated routing
2025-06-19	Show-o2	Unified text prediction and continuous visual flow matching
2025-06-04	MiMo-VL	Four-stage multimodal pretraining followed by mixed on-policy RL
2025-05-20	BAGEL	Mixture-of-Transformer-Experts for understanding and generation
2025-05-11	Seed1.5-VL	Compact vision encoder with a 20B-active MoE for reasoning and agents
2025-04-17	PerceptionLM	Open data and training for detailed image-video understanding
2025-04-17	Perception Encoder	Contrastive visual pretraining with aligned language and spatial features
2025-04-11	InternVL3 and InternVL3.5	Native multimodal pretraining, later extended with adaptive resolution and cascade RL
2025-04-10	Kimi-VL	MoonViT native-resolution packing with a sparse MoE decoder
2025-04-05	Llama 4 Scout and Maverick	Early-fusion native multimodality in sparse-MoE Scout and Maverick models
2025-03-26	Qwen2.5-Omni	Streaming Thinker-Talker architecture for multimodal input and speech output
2025-03-12	Gemma 3	Efficient local/global attention with long-context image understanding
2025-03-04	Aya Vision	Cross-modal model merging for multilingual multimodality without language forgetting
2025-03-03	Phi-4-multimodal	Mixture-of-LoRAs for text, vision, and speech
2025-02-20	SigLIP 2	Multilingual, localization-aware, native-aspect-ratio vision-language encoding
2025-02-08	EVEv2	Improved Baselines for Encoder-Free Vision-Language Models
2025-01-26	Qwen2.5-VL	Enhanced Vision-Language Capabilities in the Qwen Series
2025-01-21	VideoLLaMA 3	Frontier Multimodal Foundation Models for Image and Video Understanding
2025-01-20	UI-TARS	Pioneering Automated GUI Interaction with Native Agents
2025-01-14	MiniMax-01	Scaling Foundation Models with Lightning Attention
2025-01-12	MiniCPM-o-2.6	A GPT-4o Level MLLM for Vision, Speech and Multimodal Live Streaming
2025-01-10	Eagle 2	Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models
2025-01-07	Sa2VA	Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos
2024-12-31	VideoChat-Flash	Hierarchical Compression for Long-Context Video Modeling
2024-12-16	OmniVLM	A Token-Compressed, Sub-Billion-Parameter Vision-Language Model for Efficient On-Device Inference
2024-12-13	Apollo	An Exploration of Video Understanding in Large Multimodal Models
2024-12-13	DeepSeek-VL2	Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding
2024-12-10	Maya	An Instruction Finetuned Multilingual Multimodal Model
2024-12-05	InternVL 2.5	Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
2024-12-04	PaliGemma 2	A Family of Versatile VLMs for Transfer
2024-11-26	ShowUI	UI-guided visual-token selection and interleaved action histories
2024-11-26	SmolVLM	A Small, Efficient, and Open-Source Vision-Language Model
2024-11-21	AIMv2	Multimodal Autoregressive Pre-training of Large Vision Encoders
2024-11-15	LLaVA-CoT	Let Vision Language Models Reason Step-by-Step
2024-11-06	LLM2CLIP	Powerful Language Model Unlocks Richer Visual Representation
2024-11-05	Tarsier2	Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understanding
2024-10-17	Janus and Janus-Pro	Decoupled visual encoders for understanding and generation with one transformer
2024-10-08	ARIA	An Open Multimodal Native Mixture-of-Experts Model
2024-09-27	Emu3	One next-token objective over discrete text, image, and video tokens
2024-09-25	Molmo and PixMo	Open data pipeline with human captions and grounded pointing supervision
2024-09-25	Llama 3.2-Vision	Enhanced Multimodal Capabilities Built on Llama 3
2024-09-17	NVLM	Open Frontier-Class Multimodal LLMs
2024-09-11	Pixtral 12B	A Cutting-Edge Open Multimodal Language Model
2024-09-06	VILA-U	Shared discrete visual tokens for autoregressive understanding and generation
2024-08-29	Qwen2-VL	A Powerful Open-Source Vision-Language Model for Image and Video Understanding
2024-08-28	EAGLE	Exploring The Design Space for Multimodal LLMs with Mixture of Encoders
2024-08-22	Show-o	Autoregressive language and discrete-diffusion image generation in one transformer
2024-08-22	Idefics3-8B	Building and Better Understanding Vision-Language Models
2024-08-20	Transfusion	Autoregressive text and continuous image diffusion in one transformer
2024-08-09	mPLUG-Owl3	Hyper-attention for long image sequences and video
2024-08-09	VITA	Towards Open-Source Interactive Omni Multimodal LLM
2024-08-05	LLaVA-OneVision	Easy Visual Task Transfer
2024-07-24	VILA²	VILA Augmented VILA
2024-07-23	INF-LLaVA	High-Resolution Image Perception for Multimodal Large Language Models
2024-07-22	SlowFast-LLaVA	A Strong Training-Free Baseline for Video Large Language Models
2024-07-19	EVLM	An Efficient Vision-Language Model for Visual Understanding
2024-07-03	InternLM-XComposer-2.5	A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output
2024-06-27	OMG-LLaVA	Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding
2024-06-24	Cambrian-1	Spatial Vision Aggregator and systematic multi-encoder study
2024-06-17	EVE	Unveiling Encoder-Free Vision-Language Models
2024-06-14	Ovis	Learnable visual vocabulary for structural visual-text embedding alignment
2024-06-04	Parrot	Multilingual Visual Instruction Tuning
2024-05-24	ConvLLaVA	Hierarchical Backbones as Visual Encoder for Large Multimodal Models
2024-05-21	Phi-3-Vision and Phi-3.5-Vision	Compact dynamic-resolution VLM with 128K context
2024-05-20	CogVLM2	Enhanced Vision-Language Models for Image and Video Understanding
2024-05-16	Chameleon	Mixed-modal early fusion over a shared token sequence
2024-05-14	PaliGemma	A Versatile and Transferable 3B Vision-Language Model
2024-05-06	xGen-MM (BLIP-3)	An Open-Source Framework for Building Powerful and Responsible Large Multimodal Models
2024-05-02	MANTIS	Mastering Multi-Image Understanding Through Interleaved Instruction Tuning
2024-04-19	Moondream-next	Compact Vision-Language Model with Enhanced Capabilities
2024-04-15	Idefics2	Open 8B VLM with native-resolution inputs and strong OCR and document understanding
2024-04-09	InternLM-XComposer2-4KHD	A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD
2024-03-14	MM1	Controlled study of encoders, connectors, token counts, and data mixtures
2024-03-08	DeepSeek-VL	Towards Real-World Vision-Language Understanding
2024-02-19	AnyGPT	Any-to-any autoregression over discrete text, image, speech, and music tokens
2024-02-08	SPHINX-X	Scaling Data and Parameters for a Family of Multi-modal Large Language Models
2024-01-30	LLaVA 1.6	LLaVA-NeXT Improved reasoning, OCR, and world knowledge
2024-01-30	MiniCPM-V	A GPT-4V Level MLLM on Your Phone
2024-01-30	MouSi	Poly-Visual-Expert Vision-Language Models
2024-01-29	InternLM-XComposer2	Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model
2024-01-29	MoE-LLaVA	Mixture of Experts for Large Vision-Language Models
2024-01-20	moondream1 and moondream2	Compact SigLIP–Phi VLMs optimized for efficient edge inference
2024-01-05	FireLLaVA	LLaVA derivative trained rapidly on a curated multimodal instruction mixture
2024-01-01	COSMO	COntrastive Streamlined MultimOdal Model with Interleaved Pre-Training
2023-12-28	TinyGPT-V	Efficient Multimodal Large Language Model via Small Backbones
2023-12-28	MobileVLM	A Fast, Strong and Open Vision Language Assistant for Mobile Devices
2023-12-06	Alpha-CLIP	A CLIP Model Focusing on Wherever You Want
2023-11-28	Nous-Hermes-2-Vision - Mistral 7B	SigLIP-equipped Mistral VLM with OCR and function-calling data
2023-11-13	SPHINX	The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models
2023-11-10	Florence-2	A Deep Dive into its Unified Architecture and Multi-Task Capabilities
2023-11-09	u-LLaVA	Unifying Multi-Modal Tasks via Large Language Model
2023-11-09	LLaVA-Plus	Learning to Use Tools for Creating Multimodal Agents
2023-11-07	OtterHD	A High-Resolution Multi-modality Model
2023-11-06	CoVLM	Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding
2023-11-06	GLaMM	Pixel Grounding Large Multimodal Model
2023-10-17	Fuyu-8B	A Multimodal Architecture for AI Agents
2023-10-13	PaLI-3 Vision Language Models	Smaller, Faster, Stronger
2023-10-13	MiniGPT-v2	large language model as a unified interface for vision-language multi-task learning
2023-10-12	BakLLaVA	Mistral-based LLaVA variant with a CLIP vision encoder and projection adapter
2023-10-11	Ferret	Refer and Ground Anything Anywhere at Any Granularity
2023-10-05	LLaVA 1.5	Improved Baselines with Visual Instruction Tuning
2023-10-05	CogVLM	Visual Expert for Pretrained Language Models
2023-09-28	MetaCLIP	Demystifying CLIP Data
2023-08-24	Qwen-VL	A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
2023-08-22	IDEFICS	Open Flamingo-style model for interleaved image-text generation
2023-08-19	BLIVA	A Simple Multimodal LLM for Better Handling of Text-rich Visual Questions
2023-06-26	KOSMOS-2	Grounding Multimodal Large Language Models to the World
2023-05-24	LaVIN	Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Models
2023-05-11	InstructBLIP	Towards General-purpose Vision-Language Models with Instruction Tuning
2023-05-09	ImageBind	One Embedding Space To Bind Them All
2023-04-17	LLaVA	Large Language and Vision Assistant - Visual Instruction Tuning
2023-04-16	MiniGPT-4	Enhancing Vision-Language Understanding with Advanced Large Language Models
2023-03-27	SigLIP	Sigmoid Loss for Language Image Pre-Training
2023-03-14	OpenFlamingo	An Open-Source Framework for Training Large Autoregressive Vision-Language Models
2023-03-06	PaLM-E	An Embodied Multimodal Language Model
2023-02-27	KOSMOS-1	Language Is Not All You Need: Aligning Perception with Language Models
2023-01-30	BLIP-2	Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
2022-12-21	MULTIINSTRUCT	Improving Multi-Modal Zero-Shot Learning via Instruction Tuning
2022-09-14	PaLI	A Jointly-Scaled Multilingual Language-Image Model
2022-04-28	Flamingo	a Visual Language Model for Few-Shot Learning
2022-01-28	BLIP	Bootstrapping Language-Image Pre-training
2021-12-07	GLIP	Grounded Language-Image Pre-training
2021-06-25	FROZEN	Multimodal Few-Shot Learning with Frozen Language Models
2021-01-05	CLIP	Contrastive Language-Image Pre-training
2020-10-22	ViT	An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Architectures

Nemotron 3.5 Parse: Compact visual-prefix decoding for structured documents

A compact C-RADIO and mBART system generates document text, element classes, and bounding boxes with a causal visual prefix.

NVIDIA Released: 2026-10-17

Architecture figure: The October 17 model card describes C-RADIO, convolutional compression, and mBART in text; no model-specific architecture diagram or paper was found in the cited release.

C-RADIOv4-SO400M processes rectangular NaViT inputs. Convolutional adapters compress its features before a ten-layer mBART causal decoder with grouped-query attention. The approximately 0.631B parameters include a one-step prediction head. Task tokens select text, layout classes, and box extraction.

Supervision pairs document images with OCR text and layout annotations. The released decoder supports ordinary generation; the accompanying serving plugin can use the bundled prediction head for speculative decoding. The card does not disclose a complete stage-by-stage training recipe.

Training combines rendered documents, scientific PDFs, tables, and synthetic word/character pages with automated and human labels. Outputs can be converted into Markdown, HTML, or structured records. Weights use OpenMDW 1.1; tokenizer and benchmark components have separately documented terms.

Nemotron 3.5 Super VL: Hybrid state-space MoE for long-context visual agents

A Mamba-Transformer mixture-of-experts backbone combines a C-RADIO visual encoder with long-context image and video reasoning.

NVIDIA Released: 2026-10-15

The 120B/A12B model combines Mamba-2, attention, and LatentMoE blocks. A C-RADIO encoder, projection layers, and temporal video embedding connect visual inputs to the language backbone. Multi-token prediction supports speculative decoding; the advertised context is 1M tokens.

Training starts from Nemotron 3 Super, proceeds through continued pretraining and multimodal supervised tuning, then combines verifiable-reward and preference optimization with multi-teacher on-policy distillation. A final stage improves the prediction head.

Inputs are text, images, and video; output is text. The recipe discloses data mixtures and stages, but some supervised/RL datasets remain pending release. Weights use OpenMDW 1.1. Encoder-version labels differ across official artifacts, so the family name C-RADIO is used here.

Muse Spark Open: Sparse multimodal experts with a shared perception encoder

A large sparse multimodal transformer integrates perceptual inputs with agentic language reasoning over a million-token context.

Meta Superintelligence Labs Released: 2026-10-15

Architecture figure: The Muse Spark Open model card provides architecture specifications but no model-specific diagram. The cited evaluation report is not an architecture paper.

The 816B/42B-active model has 62 decoder layers, gated grouped-query attention, and 256 experts with eight selected per token. A 1.8B perception encoder combines learned absolute and 2D rotary positions with sparse/full attention. Supported inputs are text, images, video, and audio; output is text.

The downloadable version includes additional safety post-training relative to the hosted system. The card details the architecture and evaluations but does not publish a complete reproducible pretraining or alignment recipe.

Data sources include public web material, third-party collections, Meta services, and synthetic multimodal examples, with filtering and deduplication. BF16 and NVFP4 checkpoints are offered through gated access under the Muse Model License. October 15 is explicitly recorded as the release date.

Step 5 Preview: Deep sparse experts with gated sliding-window and sparse GQA

A narrow, deep multimodal MoE interleaves gated local attention with sparse grouped-query attention for long agent trajectories.

StepFun Released: 2026-10-15

Timeline basis: Dated primary architecture/training article; earlier API release day remains unresolved.

The 600B/27B-active model uses 92 transformer blocks and a 3:1 ratio of gated sliding-window to sparse full-attention layers. Sparse GQA adds blockwise token merging to reduce indexer cost. The public checkpoint supports text and image input with text output and up to 1M context.

The technical article describes precision and gradient-handling changes for deep-model stability. An agent-native synthesis system constructs tasks under expert-defined controls, followed by agentic reinforcement learning.

The release targets coding and knowledge-work trajectories. Full training data are not published. The card’s introduction also advertises video, while its input table lists text/image; this discrepancy is left explicit. October 15 is the dated architecture disclosure used here, not a claim about the earliest hosted API availability.

Liquid d1: Single-pass multimodal decisions without text decoding

A decision head returns typed, calibrated answers in one forward pass instead of generating an autoregressive response.

Liquid AI Released: 2026-10-07

d1-3B adapts LFM2.5-VL-3B to score yes/no questions, named options, and ratings. The experimental d1-omni-600M instead combines a bidirectional LFM encoder with SigLIP2 vision and FastConformer audio pathways. Its supported combinations are text-plus-image or text-plus-audio, not simultaneous image-plus-audio.

Decision-focused post-training teaches calibrated typed outputs from a shared state and multiple questions. The interface reads scores directly from decision positions, eliminating output-token decoding.

The release provides inference examples and evaluations across text, visual, and audio decisions. A complete training-data inventory is not published in the cards. Both checkpoints use the custom LFM license. These are discriminative decision models, distinct from the chat-generating LFM2.5-VL base.

GroundAnything: Autoregressive and block-diffusion spatial grounding

A shared visual-language backbone supports separate autoregressive and parallel diffusion checkpoints for spatial predictions.

Qize Yu et al., GroundingPI and collaborators Released: 2026-09-30

Timeline basis: First paper publication; code and weights followed in October.

MoonViT-V2/Kimi-K3 perception feeds Qwen3-4B through 2×2 spatial aggregation and a two-layer MLP. A vocabulary of 1,000 coordinate tokens represents spatial outputs. The diffusion variant adds a mask token and bidirectional response-block denoising; GroundAnything-VLM is the autoregressive checkpoint.

The training recipe develops grounded perception before converting the decoder to diffusion. Entropy-guided decoding refines spatial hypotheses in parallel. An optional self-speculative mode offers a separate speed/accuracy operating point.

Tasks span object localization, referring expressions, pointing, OCR, document layout, and GUI grounding. The paper appeared September 30, code October 3, and ModelScope weights October 14. Original contributions use Apache 2.0, while upstream Kimi-derived material retains its license conditions.

Holo4: Visual computer-use agents trained across GUI and tool environments

A family of visual agents combines screenshot-based interaction with code, API, and MCP tools through task-factory training.

H Company Released: 2026-09-28

Holo4-27B builds on dense Qwen3.8, while Holo4-35B-A3B uses Qwen3.6 MoE. Both retain their base visual-language pathways and learn action policies spanning browser, desktop, and mobile environments. The contribution is the agent training system rather than a new shared vision tower.

A task factory creates environments and trajectories for supervised learning. Separate LoRA reinforcement-learning experts train on desktop/web tasks and terminal/MCP/API tasks, then merge with equal weighting. This connects GUI actions with structured tools within one agent.

The release documents task construction and provides evaluation trajectories; those traces are not the complete training corpus. Availability differs by checkpoint: 35B-A3B uses Apache 2.0, whereas 27B carries research/noncommercial terms. The family release date does not imply identical licenses.

MiMo-V2.6: Hybrid sliding-window omni encoders and multi-teacher distillation

Visual and audio encoders feed a hybrid sliding-window MoE, with specialist reinforcement learning consolidated through multi-teacher on-policy distillation.

Xiaomi MiMo Released: 2026-09-22

Flash has 309B total/15B active parameters and Pro 1.02T/42B. A 681M vision transformer uses local/global attention and spatial patch merging; audio tokenization and a patch encoder provide another input path. The language backbone alternates sliding-window and global attention and includes multi-token prediction.

Multimodal midtraining precedes large-scale agentic reinforcement learning. MOPD2 combines specialist teachers with rollout, teacher-prefix, and supervised-prefix training to consolidate capabilities and reduce failures such as repeated tool calls.

The report details coding and general-agent task construction, verifiers, and multimodal mixtures. Text, image, audio, and video inputs produce text with up to 1M context. RL and distilled checkpoints are separate releases under MIT; they should not be conflated in comparisons.

Qwen3.8-Omni-Flash: Native omni agents with spatial audio and sparse long-context reasoning

A sparse hybrid Thinker combines vision, general audio, and spatial audio encoders for long-context multimodal agent work.

Qwen Team Released: 2026-09-18

The backbone inherits Qwen3.8-Next’s Gated DeltaNet and Qwen Sparse Attention. Alongside the vision encoder, AuT encodes general audio while Spatial AuT processes multichannel spatial cues. Explicit timestamps preserve cross-modal correspondence; post-training extends native 256K context to 1M.

Early mixed-modality co-training is followed by long-context reasoning and interactive agent training, then multi-teacher distillation. The report includes realtime text/speech interaction and selective retrieval of audio-video segments.

Training spans pure text and paired/interleaved audio, image, and video data. Qwen-MM-Plugins and Qwen-Live-Harness are released tooling; they are not downloadable model weights. The report documents hosted-model architecture and agent integration.

Taichu 5.0: Entropy-gated recurrent latent visual reasoning

A compact visual reasoner adds adaptive recurrent latent computation to a Qwen-based model with C-RADIO perception.

Taichu AI Released: 2026-09-15

Timeline basis: Explicit Initial release commit.

Architecture figure: The Taichu 5.0 project page and model card describe adaptive latent recurrence but provide examples and benchmarks rather than a published architecture diagram.

ZDTaichu5.0-9B combines Qwen3.5-9B with C-RADIOv4-H and adaptive image resolution. Entropy-gated adaptive recurrence allocates additional latent reasoning computation when needed. The supported context is 128K.

The release targets general visual understanding alongside spatial, multi-view, and embodied reasoning. Public architecture descriptions explain adaptive recurrence, but do not disclose a complete training corpus or sufficiently detailed stage schedule for independent reproduction.

Inputs include text and visual observations; outputs are textual answers and reasoning. Official code and a model checkpoint are linked. The timeline follows the explicitly titled Initial release commit on September 15, not the earlier mode