标签：多模态

自托管的私有 ChatGPT 替代品，把文档 RAG、Agent 和多用户聊天打包进一个 Docker 镜像

一本把 Agent 拆成 LLM+上下文+工具的开源书，配 109 个可跑实验，顶替零散博客和论文综述

字节的多模态 GUI Agent 栈，用 MCP 把模型接到真实桌面和浏览器操作上

把视觉编码器接到 LLaMA 系语言模型上的视觉指令微调参考实现，也是多数开源 VLM 的对照基线

本地常驻录屏录音，把屏幕和麦克风变成可查询的 MCP 上下文源，替代手工截图和事后回忆

一个仓库覆盖 600+ LLM 与 400+ 多模态模型的 SFT/DPO/GRPO 训练到量化部署，省掉自己拼 PEFT+并行+推理后端

中文 RAG 全栈教程，用来替代零散博客拼凑学习路径，不是可直接部署的库

把网页和 PDF 当图片切片建索引，用视觉 embedding 做检索，替代 HTML 解析型 RAG

一套 C++ 内核加多语言绑定，把 LLM、语音、视觉、RAG 塞进端侧 App，替代自己拼 llama.cpp 加各平台推理后端

批流一体的分布式数据同步层，用一套连接器配置顶替 Flink CDC 加自研同步脚本

把 GUI 操作能力做成可替换的模型层，用来顶掉自己写 adb 脚本或 Playwright 硬编码流程

用一套分块压缩数组格式同时承载多模态训练数据和向量检索，替代「对象存储 + 向量库 + 标注工具」三件套

把 Python 推理函数直接变成可容器化的 REST 服务，顶替手写 FastAPI 加 Dockerfile 那一层胶水

在 M 系列 Mac 上把 TTS/STT 从云端 API 换成本地推理的 MLX 实现

vLLM 官方多模态推理扩展，把扩散、音频、视频模型纳入同一套 OpenAI 兼容服务和 KV 缓存调度

Google 出品的多语言 AI 应用框架，用统一接口把模型调用、工具、RAG 和可观测性串成一条链路

在本地把整台机器的文件按语义建索引，替代 Spotlight 那种只认文件名的搜索

用 GRPO 替代 SFT 来训练 VLM 的参考实现，主要价值在于提供一套可复现的奖励函数和训练脚本

用 Rust 内核替代 Spark/PySpark 做多模态 ETL 和 embedding 批处理的数据引擎

用 YAML 把检索、重排、生成串成 MCP 服务，替代手写胶水代码搭 RAG 流程

The first vision plugin for DeepSeek Harness, and the vision bridge for every text-only coding agent. Paste an image, get structured JSON evidence (OCR, layout, semantics). | 全网最强 DeepSeek Harness 外挂视觉插件，为 DeepSeek、GLM 等纯文本模型外挂视觉能力，粘贴图片即得结构化 JSON 证据（OCR、版面、语义）。

Align Anything: Training All-modality Model with Feedback

The Ruby AI framework.

One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks

An open-source model family for long-form speech, dialogue synthesis, voice design, sound effects, and real-time streaming TTS

Fengshenbang-LM(封神榜大模型)是IDEA研究院认知计算与自然语言研究中心主导的大模型开源体系，成为中文AIGC和认知智能的基础设施。

[ICML'26] SimpleMem: Efficient Lifelong Memory for LLM Agents — Text & Multimodal

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

An extensible, state-of-the-art framework for columnar compression, and the fastest FOSS columnar file format. Formerly at @spiraldb, now an Incubation Stage project at LFAI&Data, part of the Linux Foundation.

[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure

Images to inference with no labeling (use foundation models to train supervised models).

streamline the fine-tuning process for multimodal models: PaliGemma 2, Florence-2, and Qwen2.5-VL

[EMNLP-2024] Build multimodal language agents for fast prototype and production

Comprehensive resources on Generative AI, including a detailed roadmap, projects, use cases, interview preparation, and coding preparation.

A curated collection of papers, benchmarks, surveys, and tools for model quantization, covering low-bit networks, LLMs, multimodal and generative models, vector and lattice quantization, and efficient deployment.

(ෆ`꒳´ෆ) A Survey on Text-to-Image Generation/Synthesis.

Let Claude (or any LLM) actually watch a video — scene-aware, deduplicated frames + transcript, from a URL or local file. Runs locally, MIT.

On-device, real-time multimodal AI with features similar to GPT-Live

[ICLR & NeurIPS 2025] Repository for Show-o series, One Single Transformer to Unify Multimodal Understanding and Generation.

An open-source implementaion for fine-tuning Qwen-VL series by Alibaba Cloud.

A collection of original, innovative ideas and algorithms towards Advanced Literate Machinery. This project is maintained by the OCR Team in the Language Technology Lab, Tongyi Lab, Alibaba Group.

Synthesizing Graphics Programs for Scientific Figures and Sketches with TikZ.

High Performance Serving Runtime For Interactive Super Intelligence System

Fast Multimodal LLM on Mobile Devices

A novel Multimodal Large Language Model (MLLM) architecture, designed to structurally align visual and textual embeddings.

Get clean data from tricky documents, powered by vision-language models ⚡

Multimodal RL training framework for diffusion & omni models

Give Claude the ability to watch and understand videos — Claude Code plugin with frame extraction and multimodal audio analysis