Awesome-VLM-Streaming-Video

Awesome-VLM-Streaming-Video 🎬

📒 Introduction

📚 A curated collection of papers and open-source code repositories dedicated to the application of Vision-Language Models (VLMs) for streaming video.

📖 Contents

• Awesome-VLM-Streaming-Video 🎬
  • 📒 Introduction
  • 📖 Contents
  • 🛠️ Project
  • 📋 Technical Report
  • 💬 Proactive Interaction
    • Auxiliary Response Head
    • Generative Token-based Trigger
    • RL-optimized Proactive Response
    • Training-free / Feature-based Trigger
    • Hybrid Trigger Framework
    • Learned Response Timing
  • 🧠 Long-term Memory Management
    • Hierarchical Multi-level Memory
    • Sliding-window / Eviction
    • Token Compression / Pruning
    • KV-Cache Compression / Retrieval / Reuse
    • Retrieval-augmented Memory
    • Semantic/Textual Abstraction
    • Event-centric Structured Memory
    • Spatial / 3D Memory
    • Parametric / Fast-weight Memory
  • ⚡ Real-time Inference
    • Encoding-Decoding Parallelism
    • Selective Model Invocation
    • Visual Token Reduction
    • KV-Cache Optimization
    • Serving / Deployment
  • 💭 Streaming with Thinking
  • 📊 Benchmarks
  • 📦 Training Datasets
  • 📝 Survey
  • 🔗 Resources
  • ☎️ We're Hiring!

🛠️ Project

Date	Title	Paper	Code	Comment
2026.06	JoyAI-VL-Interaction	[html]	[GitHub]	An Open Real-time Video-Language Interaction System
2025.11	Live VLM WebUI	[docs]	[GitHub]	Real-Time Vision Language Model Interaction with Webcam Streaming
-	火山引擎（实时音视频）	[html]	-	turn-based chat; 周期性发送 ExternalTextToLLM 触发主动响应

📋 Technical Report

Date	Title	Paper	Code	Demo	Comment
2026.09	Realtime-Venus: A Full-Duplex Interaction System with Asynchronous Delegation	[pdf]	[GitHub]	[html]	Full-duplex Interaction System; Asynchronous Delegation
2026.09	Qwen3.8-Omni-Flash-Realtime	[pdf]	[GitHub]	[html]	Omni Senses and Agentic Delivery; Live Harness
2026.09	Omni Interaction Agent Technical Report	[pdf]	[GitHub]	[demo]	Multimodal Duplex Interaction Agent
2026.09	Gemini 3.8 Live	[html]	-	[html]	Parallel Reasoning; Streaming Agent for Complex Workflows
2026.08	MOSS-VL Technical Report	[pdf]	[GitHub]	[demo]	Cross-attention vision-language architecture for full-duplex streaming interaction
2026.08	SeedRealtime: An Audio-Visual Full-Duplex LLM	[html]	-	-	Native audio-visual full-duplex interaction
2026.06	JoyAI-VL-Interaction	[pdf]	[GitHub]	[demo]	An Open Real-time Video-Language Interaction System
2026.05	Interaction Models: A Scalable Approach to Human-AI Collaboration (Thinking Machines Lab)	[html]	-	-	Time-aligned micro-turns
2026.04	MiniCPM-o 4.5	[pdf]	[GitHub]	[demo]	New Full-Duplex and Proactive Multimodal Live Streaming Capability
2026.03	Gemini 3.1 Flash Live	[html]	-	[demo]	turn-based chat
2026.04	Qwen3.5-Omni	[pdf]	-	[demo]	Adaptive Rate Interleave Alignment
2025.09	Qwen3-Omni	[pdf]	[GitHub] ; [aliyun]	[demo]	turn-based chat
2026.02	Seed2.0	[pdf]	[GitHub]	-	Doubao Video Calling; OVBench & LiveSports-3K & OVOBench & ODVBench & ViSpeak
2025.12	Seed1.8	[pdf]	[GitHub]	-	Doubao Video Calling; OVBench & LiveSports-3K & OVOBench & ViSpeak & StreamingBench & OmniMMI
2025.03	Qwen2.5-Omni Technical Report	[pdf]	[GitHub]	-	Block-Wise Audio/Visual Encoding; Time-Aligned Multimodal RoPE; Thinker-Talker Architecture with Streaming Speech Generation
2024.12	InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions	[pdf]	[GitHub]	-	Disentangled Streaming Perception, Multimodal Long Memory & Reasoning for Continuous Audio-Video Interaction
2024.08	VITA: Towards Open-Source Interactive Omni Multimodal LLM	[pdf]	-	[html]	Interactive Video, Image, Text & Audio Understanding; Open-Source Omni Multimodal Model

💬 Proactive Interaction

Auxiliary Response Head

Date	Title	Paper	Code	Comment
2026.05	StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggering	[pdf]	-	Cross-Attention Trigger over early-decoding hidden states with a trained Respond/Wait classification head
2026.03	Proact-VL: A Proactive VideoLLM for Real-Time AI Companions	[pdf]	[GitHub]	<|FLAG|>Token Response Head with Transition-Smoothed Classification & Stability Regularization
2026.03	StreamReady: Learning What to Answer and When in Long Streaming Videos	[pdf]	-	Learnable <RDY> Token with Readiness Head for Evidence-Gated Response Triggering
2025.05	StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant	[pdf]	[GitHub]	Activation Model via <ACT> Token with Binary Score Head
2025.03	ViSpeak: Visual Instruction Feedback in Streaming Videos	[pdf]	[GitHub]	Informative Head for <seg> Token
2025.03	StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognition	[pdf]	[GitHub]	Cognition Gate Network (Shallow Layer Transfer from LLM) for Binary </response> / </silence> Classification
2025.01	Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction	[pdf]	[GitHub]	Binary Classification Head on <TODO> Token Embedding (BCE Loss); <ANS> for History Marking; <SILENT> for Reaction-Stage Filtering
2024.11	VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Format	[pdf]	[GitHub]	Informative Head & Relevance Head

Generative Token-based Trigger

Date	Title	Paper	Code	Comment
2026.10	OneStreamer: Unifying Perception, Memory, and Proactive Response in Streaming Video Interaction	[pdf]	[html]	Proactive Hierarchical Caption Memory with </Observe>/</Summary>; </Silence>/</Standby>/</Response> control tokens; OneStreamer-1M
2026.06	Harnessing Streaming Video in the Wild	[pdf]	-	Training Objective Targeting
2026.04	AURA: Always-On Understanding and Real-Time Assistance via Video Streams	[pdf]	[GitHub]	Unified <|silent|> Token for Silent Observation with Real-Time QA / Proactive QA / Multi-Response QA; Silent-Speech Balanced Loss
2026.03	STRIDE: When to Speak Meets Sequence Denoising for Streaming Video Understanding	[pdf]	[GitHub]	Activation Token Denoising over 0/1/[M]; Sequence Duplication; Selective Re-masking
2025.12	Streaming Video Instruction Tuning	[pdf]	[GitHub]	Three-State Response Tokens (</Silence>, </Standby>,</Response>) via Unified Next-Token Prediction; Focal Loss for Response Imbalance
2025.06	Proactive Assistant Dialogue Generation from Streaming Egocentric Videos	[pdf]	[GitHub]	Frame-Level [EOS] Silence Prediction with Negative Frame Sub-Sampling
2025.03	LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistant	[pdf]	[GitHub]	Streaming EOS Prediction via Fast-Slow Dual-Path; Token Aggregation & Dropping Router for Visual Feature Compression
2025.03	AssistPDA: An Online Video Surveillance Assistant for Video Anomaly Prediction, Detection, and Analysis	[pdf]	-	EOS Prediction with Task-Adaptive Threshold for Anomaly Alerting
2024.07	What to Say and When to Say it: Live Fitness Coaching as a Testbed for Situated Interaction	[pdf]	[GitHub]	Action Tokens <next> and <feedback>
2024.06	VideoLLM-online: Online Video Large Language Model for Streaming Video	[pdf]	[GitHub]	Streaming EOS Token Prediction

RL-optimized Proactive Response

Date	Title	Paper	Code	Comment
2026.10	StreamVibe: Proactive Video Agents for Evolving User Intent	[pdf]	[html]	Adaptive Streaming SFT; Localized Trajectory GRPO with Point-Level Timing / Content Rewards and Trajectory-Level Coherence Reward
2026.10	SyncPro: Synchronous Reasoning and Responding for Proactive Streaming Video Understanding	[pdf]	[GitHub]	Independent Inner / Response Activation; Timing-Aware SFT with Action Temporal Ranking; Trajectory-Level RL with Hard-Window F1 Reward
2026.09	Watch-Think-Interact: Bootstrapping Long-Horizon Multi-Turn Streaming Video Reasoning with Reinforcement Learning	[pdf]	-	Stream-GDPO Optimizes Multi-Question Rollouts with Answer / Continue-Watching / Past-Interval Recall Decisions and Trajectory-Level Feedback
2026.05	StreamPro: From Reactive Perception to Proactive Decision-Making in Streaming Video	[pdf]	-	Two-stage proactive training with CB-Stream Loss for SFT and GRPO with turn-level & trajectory-level rewards
2026.03	Thinking in Streaming Video	[pdf]	[GitHub]	Watch-Think-Speak with <silent> & <response> Action Tokens; Streaming RLVR (Format + Time + Accuracy Reward)
2026.01	Learning to Respond: A Large-Scale Benchmark and Progressive Learning Framework for Trigger-Centric Online Video Understanding	[pdf]	-	Trigger-Centric Online Video Understanding
2025.12	MMDuet2: Enhancing Proactive Interaction of Video MLLMs with Multi-Turn Reinforcement Learning	[pdf]	[GitHub]	Text-to-Text "NO REPLY" Response Decision with Multi-Objective Reward (PAUC + Replication + In-Span + Prefix) GRPO

Training-free / Feature-based Trigger

Date	Title	Paper	Code	Comment
2026.10	Foresight: Planning Future Perception in Streaming VLMs without Retraining	[pdf]	[GitHub]	Training-free Siamese Frozen Qwen3-VL-8B with Shared KV Cache Planning When to Reason, What to Check & Sampling FPS
2026.06	LiveStarPro: Proactive Streaming Video Understanding with Hierarchical Memory for Long-Horizon Streams	[pdf]	[GitHub]	Inference-Time Streaming Verification Decoding (SVeD) for Response/Silence Decisions
2026.05	Response-G1: Explicit Scene Graph Modeling for Proactive Streaming Video Understanding	[pdf]	-	Fine-Tuning-Free Retrieval-Augmented Trigger Prompting with Query-Guided Scene Graph Evidence
2026.03	FluxMem: Adaptive Hierarchical Memory for Streaming Video Understanding	[pdf]	[GitHub]	Scene-Change Ratio Trigger Reusing Temporal Adjacency Selection Statistics
2026.01	QueryStream: Advancing Streaming Video Understanding with Query-Aware Pruning and Proactive Response	[pdf]	-	Relevance-Triggered Active Response
2026.01	Event-VStream: Event-Driven Real-Time Understanding for Long Video Streams	[pdf]	-	Motion-Semantic-Prediction Boundary Score; Adaptive Threshold; Event-triggered Decoding
2025.11	LiveStar: Live Streaming Assistant for Real-World Online Video Understanding	[pdf]	[GitHub]	Streaming Verification Decoding Perplexity Gate for Response & Silence
2025.08	StreamAgent: Towards Anticipatory Agents for Streaming Video Understanding	[pdf]	-	Reactive/Proactive/Speculative Planning; Heuristic Trigger; Tool-Guided Information Hunting

Hybrid Trigger Framework

Date	Title	Paper	Code	Comment
2026.03	Em-Garde: A Propose-Match Framework for Proactive Streaming Video Understanding	[pdf]	[GitHub]	Query-Time Instruction-Guided Visual Proposal Generation (SFT+GRPO); Lightweight Embedding Cosine Similarity Surge Triggering
2026.03	StreamingClaw Technical Report	[pdf]	-	Training-Free (Reminder Node); Training-Based (Scenario-Specific Trigger Tokens)

Learned Response Timing

Date	Title	Paper	Code	Comment
2026.05	An Efficient Streaming Video Understanding Framework with Agentic Control	[pdf]	-	R3-Streaming Readiness Head Emits <Routine> to Defer Responses until Sufficient Evidence Arrives
2026.04	Progressive Online Video Understanding with Evidence-Aligned Timing and Transparent Decisions	[pdf]	-	Thinking-QwenVL with Active Thinking Decision Maker (progress/confidence) for Evidence-Aligned Response Timing
2025.10	Eyes Wide Open: Ego Proactive Video-LLM for Streaming Video	[pdf]	[GitHub]	Weighted Interval Supervision & Uncertainty-Guided High-Resolution Requests
2025.02	EgoSpeak: Learning When to Speak for Egocentric Conversational Agents in the Wild	[pdf]	[GitHub]	Standalone Audio-Visual Frame-Level Three-Way Classifier (Background / Self / Other) with Anticipatory Prediction

🧠 Long-term Memory Management

Hierarchical Multi-level Memory

Date	Title	Paper	Code	Comment
2026.10	StreamVibe: Proactive Video Agents for Evolving User Intent	[pdf]	[html]	StreamVibe-Harness: Persistent Intent States with Flexible Lifecycles; Recent Visual Context, Mid-Term Summaries & Long-Term Text Memory with Intent-Aware Retrieval
2026.10	OneStreamer: Unifying Perception, Memory, and Proactive Response in Streaming Video Interaction	[pdf]	[html]	Recent-N visual window with Proactive Hierarchical Caption Memory (PHCM) for long-range streaming context
2026.08	StreamFlow: Dynamic Memory Flows for Streaming Video Understanding	[pdf]	-	Dynamics-Aware Mid-Term Memory with Pre-Encoding Redundancy Filtering + Fixed-Capacity Latent Long-Term Memory
2026.06	LiveStarPro: Proactive Streaming Video Understanding with Hierarchical Memory for Long-Horizon Streams	[pdf]	[GitHub]	Tree-Structured Hierarchical Memory (TSHM) for Long-Horizon Streaming on Existing LiveStar-8B Weights
2026.06	Harnessing Streaming Video in the Wild	[pdf]	-	short-term memory; mid-term memory; long-term memory; enabling up to 12 hours of context
2026.05	StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggering	[pdf]	-	Multimodal evidence-guided long-short term memory with dense recent observations and sparse historical evidence under a fixed budget
2026.05	Semantic-Aware Adaptive Visual Memory for Streaming Video Understanding	[pdf]	[GitHub]	Training-Free Three-Tier Memory with Semantic-Prior-Guided Selective Forgetting & Query-Adaptive Retrieval Scope
2026.04	Progressive Online Video Understanding with Evidence-Aligned Timing and Transparent Decisions	[pdf]	-	Hierarchical Progressive Semantic Integration (HPSI) with Learnable Multi-Level Aggregation Tokens Propagated across Clips
2026.03	CurveStream: Boosting Streaming Video Understanding in MLLMs via Curvature-Aware Hierarchical Visual Memory Management	[pdf]	-	Curvature-Aware Scorer (Motion Variation + Geometric Curvature); EMA-Based K-Sigma Dynamic Thresholding; Hierarchical Clear/Blurred/Discard Memory with FIFO Eviction
2026.03	StreamingClaw Technical Report	[pdf]	-	Hierarchical Memory Evolution
2026.03	StreamReady: Learning What to Answer and When in Long Streaming Videos	[pdf]	-	3-Level Visual Memory Tree (FIFO-Centroid-Prototype) & Contextual Memory Bank
2026.03	WAT: Online Video Understanding Needs Watching Before Thinking	[pdf]	-	Dual-Level Memory: Short-Term FIFO Queue + Long-Term Memory with Redundancy-Aware Eviction Policy
2026.03	Scaling the Long Video Understanding of Multimodal Large Language Models via Visual Memory Mechanism	[pdf]	[GitHub]	Dual-Pathway Compression for Context Memory and Local Memory; Visual KV-Cache Memory Bank; Cross-attention Memory Recall & MemIndex
2026.03	FluxMem: Adaptive Hierarchical Memory for Streaming Video Understanding	[pdf]	[GitHub]	Three-Level Hierarchical Memory (Short/Mid/Long-Term); Temporal Adjacency Selection & Spatial Domain Consolidation
2026.02	FreshMem: Brain-Inspired Frequency-Space Hybrid Memory for Streaming Video Understanding	[pdf]	-	Short-term Sliding Window + Multi-scale Frequency Memory + Space Thumbnail Memory
2026.01	HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding	[pdf]	[GitHub]	Hierarchical KV-Cache (Sensory-Working-Long-Term Memory); Exponential Forgetting Curve & Frame-Level Anchor Tokens; Cross-Layer Memory Smoothing & Position Re-Indexing
2025.12	Venus: An Efficient Edge Memory-and-Retrieval System for VLM-based Online Video Understanding	[pdf]	-	Hierarchical Raw Data Layer & Semantic Index Layer; Scene Segmentation and Incremental Clustering for Sparse Memory Construction
2025.08	StreamAgent: Towards Anticipatory Agents for Streaming Video Understanding	[pdf]	-	Streaming KV-Cache; CPU Long-Term & GPU Short-Term Hierarchical Memory; Layer-Adaptive KV Recall
2025.07	StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modeling	[pdf]	[GitHub]	Sliding-Window KV-Cache with Selective Offloading; 3D Voxel-Based Spatial Token Pruning for Cross-Frame Redundancy Removal
2025.01	Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge	[pdf]	[GitHub]	Hierarchical Memory: Ebbinghaus Forgetting Curve Short-Term + Tree-Structured Clustering Long-Term + FAISS Dialogue Memory; Forgetting Probability-Based Frame Sampling
2024.12	InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions	[pdf]	[GitHub]	Short-Term / Long-Term Multimodal Memory with Compression and Retrieval, Coordinated with Streaming Perception and Reasoning
2024.09	VideoLLaMB: Long Streaming Video Understanding with Recurrent Memory Bridges	[pdf]	[GitHub]	Recurrent Memory Bridge with Self-Attention Recursive Update & Retrieval Attention; SceneTiling Semantic Segmentation; Linear Memory Scaling

Sliding-window / Eviction

Date	Title	Paper	Code	Comment
2026.04	AURA: Always-On Understanding and Real-Time Assistance via Video Streams	[pdf]	[GitHub]	Dual Sliding-Window Context over Recent Video and QA History; Out-of-Window Video Chunks and <|silent|> Tokens Discarded
2026.04	A Simple Baseline for Streaming Video Understanding	[pdf]	[GitHub]	Fixed Recent-frame Sliding Window; Old-frame Eviction without External Memory
2026.03	Proact-VL: A Proactive VideoLLM for Real-Time AI Companions	[pdf]	[GitHub]	Dual-Cache (System & Streaming) Sliding Window with Reverse-RoPE Eviction for Infinite Streaming
2026.03	STRIDE: When to Speak Meets Sequence Denoising for Streaming Video Understanding	[pdf]	[GitHub]	Bounded Visual Cache Eviction; Sliding-window Frame Retention

Token Compression / Pruning

Date	Title	Paper	Code	Comment
2026.06	Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understanding	[pdf]	[GitHub]	Training-Free Fixed-Budget Memory Bank with Online Semantic Basis for Causal Visual Token Selection
2026.05	An Efficient Streaming Video Understanding Framework with Agentic Control	[pdf]	-	Training-Free Active Forgetting with High-Fidelity Recent Tokens and Aggressive Historical Compression
2025.10	Recurrent Attention-based Token Selection for Efficient Streaming Video-LLMs	[pdf]	-	Attention-Score-Based Visual Token Selection with Recurrent FIFO Token Queue; Maximal Marginal Relevance Caption Retrieval
2025.05	StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant	[pdf]	[GitHub]	Producer-Consumer Memory Buffer with Conditional Round-Decayed Token Compression (Prioritizing Recent Frames)
2025.04	TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videos	[pdf]	[GitHub]	Differential Token Drop (Primary); FIFO Slimmed Token Memory Bank (Supplementary)
2025.03	VideoScan: Enabling Efficient Streaming Video Understanding via Frame-level Semantic Carriers	[pdf]	[GitHub]	Single Semantic Carrier Token per Frame (Avg-Pool Embedding + Reused KV Aggregation); Cosine Similarity-Based Dynamic Memory Bank Eviction

KV-Cache Compression / Retrieval / Reuse

Date	Title	Paper	Code	Comment
2026.09	ShallowStream: Index Shallow then Answer Deep for Streaming Video Understanding	[pdf]	[GitHub]	Always-On Shallow-Layer KV Index; Query-Time Attention Scoring and Diversity-Aware Frame Retrieval before Deep Answering
2026.05	MuKV: Multi-Grained KV Cache Compression for Long Streaming Video Question-Answering	[pdf]	[GitHub]	Multi-Grained KV-Cache Compression at Patch, Frame & Segment Levels with Attention- and Frequency-Guided Token Selection; Semi-Hierarchical Retrieval for Online QA
2026.05	CoRDS: Coreset-based Representative and Diverse Selection for Streaming Video Understanding	[pdf]	[GitHub]	Training-Free KV-Cache Coreset Selection with Joint K/V Coverage, Diversity & Recency Protection
2026.05	Decouple and Cache: KV Cache Construction for Streaming Video Understanding	[pdf]	[GitHub]	Decoupled Streaming Cache (DSCache): Cumulative Past KV Cache + On-Demand Instant Cache with Position-Agnostic Encoding
2026.02	Going Down Memory Lane: Scaling Tokens for Video Stream Understanding with Dynamic KV-Cache Memory	[pdf]	[GitHub]	Adaptive Key Selection for Sparse Sliding-Window Encoding; Training-Free Retrieval MoE via Reciprocal Rank Fusion of Internal & External Signals
2025.12	V-Rex: Real-Time Streaming Video LLM Acceleration via Dynamic KV Cache Retrieval	[pdf]	-	Hash-Bit Hamming Clustering & Weighted Cumulative Sum Early-Exit Thresholding for Dynamic KV-Cache Retrieval; Hierarchical Memory Offloading
2025.11	StreamKV: Streaming Video Question-Answering with Segment-based KV Cache Retrieval and Compression	[pdf]	[GitHub]	Cosine Similarity-Based Dynamic Semantic Segmentation; Summary-Vector Representative Key Retrieval; Guidance-Prompt-Driven KV Compression with Layer-Adaptive Budget Allocation
2025.11	LiveStar: Live Streaming Assistant for Real-World Online Video Understanding	[pdf]	[GitHub]	Peak-End Memory Compression & Dual-Level Streaming KV Cache
2025.10	StreamingTOM: Streaming Token Compression for Efficient Video Understanding	[pdf]	[GitHub]	Causal Temporal Token Reduction (Static-Dynamic DPC & Attention Selection) + Online 4-bit Quantized KV Memory with Representative-Key Retrieval
2025.08	StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understanding	[pdf]	-	Chat Template Proxy Query for Query-Agnostic KV-Cache Pruning & Weighted Merging
2025.06	InfiniPot-V: Memory-Constrained KV Cache Compression for Streaming Video Understanding	[pdf]	[GitHub]	KV-Cache Compression with Temporal-Axis Redundancy Pruning & Value-Norm Ranking with Layer-Adaptive Pooling
2025.05	LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrieval	[pdf]	-	Streaming-Oriented KV Cache with Video-Specific KV Compression; Frame-Wise KV Merging & FIFO KV Chunk Memory

Retrieval-augmented Memory

Date	Title	Paper	Code	Comment
2026.09	ShallowStream: Index Shallow then Answer Deep for Streaming Video Understanding	[pdf]	[GitHub]	Shallow-Layer Attention Scores Retrieve Diverse Evidence Frames for Query-Time Full-Depth Answering
2026.09	Watch-Think-Interact: Bootstrapping Long-Horizon Multi-Turn Streaming Video Reasoning with Reinforcement Learning	[pdf]	-	Time-Tagged Natural-Language Memory Anchors Selective Recall of Past Video Intervals when Retained Context Is Insufficient
2026.08	Dynamic Hub-and-Spoke Memory for Streaming Video Understanding	[pdf]	-	Training-Free Entity-Centered Hub-and-Spoke Text Memory; Query-Aware Retrieval and Link Expansion Combined with Recent Visual Tokens
2026.06	What Should a Streaming Video Model Remember?	[pdf]	-	SelectStream with Surprise-Driven Writing, Priority-Preserving Latent Memory Graph & Query-Conditioned Graph Retrieval
2026.05	Response-G1: Explicit Scene Graph Modeling for Proactive Streaming Video Understanding	[pdf]	-	Query-Guided Scene Graph Memory with Top-K Similarity Retrieval
2026.04	OASIS: On-Demand Hierarchical Event Memory for Streaming Video Reasoning	[pdf]	[GitHub]	On-Demand Retrieval over Hierarchical Event Memory; Uncertainty-Triggered Controlled Refinement
2026.03	PEARL: Personalized Streaming Video Understanding Model	[pdf]	[GitHub]	Dual-Grained Memory (Streaming Memory + Concept Memory); Concept-Aware Retrieval with Query Rewriting for Personalized Concept Grounding
2026.02	Vista: Scene-Aware Optimization for Streaming Video Question Answering under Post-Hoc Queries	[pdf]	-	Scene-aware Segmentation; Temporal-Spatial Scene Token Compression; CPU Offloaded Full Frames with Query-conditioned Top-k Recall
2026.02	WeaveTime: Stream from Earlier Frames into Emergent Memory in VideoLLMs	[pdf]	[GitHub]	Uncertainty-Gated Hierarchical Retrieval with Entropy-Threshold Triggered Past Context Access
2025.12	CogStream: Context-guided Streaming Video Question Answering	[pdf]	[GitHub]	Temporal-Semantic Clustering with Question-Aware Event Compression; Historical Dialogue Retrieval via LLM Selection
2025.11	CacheFlow: Compressive Streaming Memory for Efficient Long-Form Video Understanding	[pdf]	-	Dynamic Token Dropping; GRU-Based Compressive Memory; KV Offloading and Rehydration; Consensus-First Retrieval
2025.06	Flash-VStream: Efficient Real-Time Understanding for Long Video Streams	[pdf]	[GitHub]	K-Means Clustered Context Synopsis Memory & Feature-Centric Detail Augmentation Memory with Disk-Offloadable Feature Bank
2025.03	Streaming Video Question-Answering with In-context Video KV-Cache Retrieval	[pdf]	[GitHub]	Sliding-Window Encoding with KV-Cache Offloading to RAM/Disk; Internal (Self-Attention Key Averaging) & External (CLIP Cosine Similarity) Frame-Level Retrieval

Semantic/Textual Abstraction

Date	Title	Paper	Code	Comment
2026.08	Dynamic Hub-and-Spoke Memory for Streaming Video Understanding	[pdf]	-	Typed Textual Observations Store Distant History in Entity Hubs and Evidence Spokes; Recent Frames Remain Visual Tokens
2026.07	FOLIO: Focused Semantic Memory for Streaming Video Understanding	[pdf]	-	Training-Free Focus-Aware Writing with Short-Term Visual Buffer, Entity-Centered Semantic Memory & Evidence Cache
2026.04	StreamMeCo: Long-Term Agent Memory Compression for Efficient Streaming Video Understanding	[pdf]	[GitHub]	Connectivity-Aware Compression of Text/Face/Voice Memory Graph Nodes with Time-Decay Retrieval
2026.03	Click-to-Ask: An AI Live Streaming Assistant with Offline Copywriting and Online Interactive QA	[pdf]	-	Streaming Event Segmentation; Event-level Historical Caption Memory; Knowledge Extraction Accelerator
2026.03	Thinking in Streaming Video	[pdf]	[GitHub]	Reasoning-Compressed Streaming Memory: Visual Sliding Window + Reasoning Tokens as Long-Term Semantic Anchors
2026.03	Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneously	[pdf]	[GitHub]	Short-Term Visual Sliding Window & Long-Term Textual Streaming-Thought Memory with FIFO Eviction; Recursive Temporal Segmentation
2026.02	Exploring Multimodal LMMs for Online Episodic Memory Question Answering on the Edge	[pdf]	-	Convert Video into a Lightweight Textual Memory
2025.06	Proactive Assistant Dialogue Generation from Streaming Egocentric Videos	[pdf]	[GitHub]	Iterative Progress Summarization as Summary-Based Memory Compression
2025.04	Memory-efficient Streaming VideoLLMs for Real-time Procedural Video Understanding	[pdf]	[GitHub]	Multimodal Interleaved Cache: Online Verbalization of Visual-to-Text for Long-Term & Short-Term Visual Tokens

Event-centric Structured Memory

Date	Title	Paper	Code	Comment
2026.08	StreamArena: Toward Continuous, Interactive, and Long-Horizon Agentic Streaming Video Understanding	[pdf]	-	StreamMind Hierarchical Event Memory + Entity Relation Graph + Key Frames with Asynchronous Memory Writing
2026.08	StreamSoccer: Event-Driven Memory for Streaming Soccer Commentary	[pdf]	-	Fixed-Budget Active Event Memory; Completed-Event Consolidation and Historical Retrieval for Current, Recent & Historical Soccer Commentary
2026.07	ObjectStream: Latent Objects as Memory Anchors for Streaming Video Understanding	[pdf]	[GitHub]	Training-Free Latent Object Anchors with Cross-Frame Association, Persistent Tracks & State-Change Retention
2026.07	Imprint: Online Memory Compression for Long-Horizon Egocentric QA	[pdf]	-	Online Interaction Records Consolidated into Recurring Patterns using Recurrence, Recency & Distinctiveness for Long-Horizon Egocentric QA
2026.04	OASIS: On-Demand Hierarchical Event Memory for Streaming Video Reasoning	[pdf]	[GitHub]	Hierarchical Event Memory with Intent-Driven Retrieval and Plug-and-Play Training-Free Reasoning
2026.02	EventMemAgent: Hierarchical Event-Centric Memory for Online Video Understanding with Adaptive Tool Use	[pdf]	[GitHub]	Event-Centric Dual-Layer Memory (STM with Online Event Segmentation & Reservoir Sampling + LTM with Structured Event Tuples)
2026.01	Event-VStream: Event-Driven Real-Time Understanding for Long Video Streams	[pdf]	-	Event-level Memory Bank; Merge-or-Append Event Consolidation
2025.12	VideoScaffold: Elastic-Scale Visual Hierarchies for Streaming Video Understanding in MLLMs	[pdf]	[GitHub]	Prediction-Guided Elastic-Scale Event Segmentation & Hierarchical Cross-Attention Event Consolidation
2025.09	StreamForest: Efficient Online Video Understanding with Persistent Event Memory	[pdf]	[GitHub]	Event-Level Tree Hierarchy with Adaptive Merging via Similarity & Merge-Count & Temporal Penalty; Short-Term Spatiotemporal Sliding Window

Spatial / 3D Memory

Date	Title	Paper	Code	Comment
2026.06	Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priors	[pdf]	[html]	Incremental geometry priors with Visual-Spatial Feature Integration and Geometry-Adaptive Voxel Compression for streaming 3D understanding
2026.01	OnlineSI: Taming Large Language Model for Online 3D Understanding and Grounding	[pdf]	[GitHub]	Fixed-size Spatial Memory with Time-adaptive Sampling and Concatenation; Explicit Point Cloud and Semantic Memory

Parametric / Fast-weight Memory

Date	Title	Paper	Code	Comment
2026.10	Memory as Weights: Internalizing Long-Term History for Streaming Videos	[pdf]	-	Recurrent HyperNetwork Consolidates Streaming Video History into Fixed-Size LoRA Weights while Keeping Current Observations as Input
2026.08	StreamTTT: Reconciling Real-Time Perception and Long-Term Memory in Streaming VLMs	[pdf]	-	Parallel Gated TTT Fast-Weight Memory for Long-Range History outside the Short Sliding Attention Context
2025.10	video-SALMONN S: Streaming Audio-Visual LLMs Beyond Length Limits via Memory	[pdf]	[GitHub]	Test-Time Training Fast-Weight MLP as Streaming Memory with Dual (Reconstruction + Long-Span Prediction) Objective; Cosine Similarity Token Discarding; Prompt-Dependent Modality-Aware KV-Cache Chunk Reading

⚡ Real-time Inference

Encoding-Decoding Parallelism

Date	Title	Paper	Code	Comment
2026.03	Think-as-You-See: Streaming Chain-of-Thought Reasoning for Large Vision-Language Models	[pdf]	[GitHub]	Parallel Dual KV-Cache with Merge-Generate-Split Loop for Concurrent Encoding-Decoding; Decoupled Cross-Modal RoPE; Near-Zero Time-to-First-Token
2026.03	Think While Watching: Online Streaming Segment-Level Memory for Multi-Turn Video Reasoning in MLLMs	[pdf]	[GitHub]	Threaded Parallel Watch-Think Pipeline with Async Segment Prefetch & Adaptive Attention Backend
2026.01	Speak While Watching: Unleashing TRUE Real-Time Video Understanding Capability of Multimodal Large Language Models	[pdf]	[GitHub]	Decoupled Positional Encoding (Overlapped / Group-Decoupled / Gap-Isolated) for Parallel Perception-Generation Streaming

Selective Model Invocation

Date	Title	Paper	Code	Comment
2026.09	StreamScout: Learning When to Look Deeper for Streaming Video Understanding	[pdf]	-	Query-Adaptive Three-Stage Evidence Cascade (Recent Glance, Uniform Look-Back & Query-Salient Retrieval) with Self-Distilled and GRPO-Refined Stop-or-Escalate Decisions
2026.09	ReactVAU: A Slow-Fast Decoupled Framework for Streaming Video Anomaly Understanding	[pdf]	[html]	Lightweight Spatial Grid Folding for Continuous Anomaly Filtering; Anomaly-Aware Persistent Memory; Suspicious Events Wake Heavyweight Semantic Reasoning
2026.05	An Efficient Streaming Video Understanding Framework with Agentic Control	[pdf]	-	TB-GRPO Fast/Slow Model Routing with Target-Band Control over Heavy-Model Escalation Frequency
2026.03	STRIDE: When to Speak Meets Sequence Denoising for Streaming Video Understanding	[pdf]	[GitHub]	Two-stage Activation-to-Generation Pipeline; Event-gated Downstream Video-LLM Invocation
2026.03	Color When It Counts: Grayscale-Guided Online Triggering for Always-On Streaming Video Sensing	[pdf]	[GitHub]	Windowed Grayscale Affinity Analysis with Quadratic Programming; Credit-Budgeted RGB Activation; Dynamic Token Router with Asymmetric Grayscale and RGB Token Capacity
2026.03	StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognition	[pdf]	[GitHub]	SSM-Based Single-Token Perception with Event-Gated Sparse LLM Invocation
2026.01	Event-VStream: Event-Driven Real-Time Understanding for Long Video Streams	[pdf]	-	Boundary-aware Event Pooling; Event-triggered Sparse Decoding; Hysteresis Pacing Control
2025.09	Open-ended Hierarchical Streaming Video Understanding with Vision Language Models	[pdf]	-	Lightweight RNN Streaming Module with Event-Gated Sparse Frozen VLM Invocation
2025.03	LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistant	[pdf]	[GitHub]	Routing-Based Response Determination via Fast-Slow Dual-Path; Token Aggregation & Dropping for High-FPS Routing

Visual Token Reduction

Date	Title	Paper	Code	Comment
2026.09	CoFiE: Coarse-to-Fine Evidence Selection for Efficient Streaming Video Understanding	[pdf]	-	Novelty-Guided Frame Filtering before Vision Encoding plus Query-Specific Evidence Refinement during LLM Prefill
2026.08	StreamFlow: Dynamic Memory Flows for Streaming Video Understanding	[pdf]	-	Raw-Frame Temporal-Residual Filtering before Visual Encoding for Lower Latency and Peak Memory
2026.07	OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models	[pdf]	[GitHub]	Training-Free Audio-Video Compression with Modality-Specific Budgets; Anchor-Delta Visual Pruning and Per-Second Audio Token Merging
2026.06	Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understanding	[pdf]	[GitHub]	Online Semantic-Basis Redundancy Estimation with Informative Visual Token Retention under a Fixed Budget
2026.05	Semantic-Aware Adaptive Visual Memory for Streaming Video Understanding	[pdf]	[GitHub]	Pseudo-Question Semantic Prior for Temporal Pruning, Spatial Selection & Constant-Budget Selective Forgetting
2026.04	CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference	[pdf]	-	Single-pass Compressed-bitstream Ingestion; Motion Vector-guided Patch Pruning; I-frame Anchor KV-Cache Refresh with RoPE-based Position Correction
2026.04	A Simple Baseline for Streaming Video Understanding	[pdf]	[GitHub]	Fixed Recent-frame Window; Bounded-memory Low-latency Inference
2026.03	Thinking in Streaming Video	[pdf]	[GitHub]	Eager Prefill + CUDA Graph Decode-and-Prune
2026.03	FluxMem: Adaptive Hierarchical Memory for Streaming Video Understanding	[pdf]	[GitHub]	Visual Token Compression via TAS & SDC with Otsu-Based Adaptive Thresholding for Latency & Memory Reduction
2026.02	QueryStream: Advancing Streaming Video Understanding with Query-Aware Pruning and Proactive Response	[pdf]	-	Query-Aware Differential Pruning (QDP) & Relevance-Triggered Active Response (RTAR) Scheduling
2026.02	OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence	[pdf]	-	Codec Patchification Selects Predictive Visual Regions; Shared 3D RoPE Supports Sparse Spatial-Temporal Video Encoding
2025.12	StreamingAssistant: Efficient Visual Token Pruning for Accelerating Online Video Understanding	[pdf]	-	Checkerboard-Masked Parallel Spatial Pruning with Adjacency-Constrained R