登录

OmAgent

om-ai-labApache-2.0活跃维护

一个多模态语言 Agent 开发框架，把视觉模型和 LLM 编排在一起，帮你快速搭出能看能听的原型并推向生产。

[EMNLP-2024] Build multimodal language agents for fast prototype and production

v0.2.4发布于 2025-02-24 08:16 UTC2,671star289fork同步于 2026-11-10 17:45 UTC
多模态AgentAgent框架LLM编排视觉语言模型原型开发
对比 GitHub 官网agentchatbotgeminigptgpt4

怎么上手

由 README 提炼，执行前请确认
原文 
pip install omagent-core
cd examples/step1_simpleVQA && python compile_container.py
python run_webpage.py
硬件要求按 README 说明

README 未说明具体显存或 GPU 型号要求；支持通过 Ollama 或 LocalAI 在本地部署自有模型。

下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼
没人工点评？写下你的经验

轻量级 Python 库，快速搭建可处理图像、视频和音频的多模态语言智能体

一个多模态语言 Agent 开发框架，把视觉模型和 LLM 编排在一起，帮你快速搭出能看能听的原型并推向生产。

核心能力

  • 用图结构工作流引擎编排智能体节点与任务
  • 内置多种记忆类型支持上下文推理
  • 原生支持 VLM 模型、实时 API 与移动端连接
  • 提供 ReAct、CoT、SC-CoT 等现成智能体算法
  • 支持 Lite 模式，无需部署中间件

容易踩的坑

需先运行 compile_container.py 生成 container.yaml 才能启动示例；运行前要设置 custom_openai_key 和 custom_openai_endpoint 环境变量；网页示例默认监听 127.0.0.1:7860。

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

最近发版

5 个版本v0.2.4
全部 release
  1. v0.2.4最新正式版2025-02-24 · 2025-02-24 08:16 UTC

    What's Changed

    New Contributors

    Full Changelog: https://github.com/om-ai-lab/OmAgent/compare/v0.2.3...v0.2.4

  2. v0.2.32025-01-26 · 2025-01-26 09:29 UTC
    查看更新说明

    What's Changed

    Full Changelog: https://github.com/om-ai-lab/OmAgent/compare/v0.2.2...v0.2.3

  3. v0.2.22025-01-14 · 2025-01-14 09:03 UTC
    查看更新说明

    What's Changed

    New Contributors

    Full Changelog: https://github.com/om-ai-lab/OmAgent/compare/v0.2.1...v0.2.2

  4. v0.2.12024-12-25 · 2024-12-25 06:44 UTC
    查看更新说明
    1. New debug mode, features include:
      • More detailed log information
      • Stops all running instances of the same workflow when a workflow is started
      • Disables error retries
    2. New Tavily search tool
    3. New device:
      • WebpageClient provides a Gradio-based web UI
      • ProgrammaticClient offers a concise programming interface, allowing you to call workflows as Python functions
    4. Use the conductor's logging system instead of the local logging system.
    5. New in-memory STM, enabling STM functionality without relying on external components
    6. Optimized large language model invocation methods, especially for multimodal invocation.
    7. Optimized app backend code
  5. v0.2.02024-12-04 · 2024-12-04 11:23 UTC
    查看更新说明

    What's Changed

用户点评

登录用户的真实使用记录，可按版本和硬件筛选
共 0 条

    还没有人点评。第一个说出真实体验的人最有价值。

    README 原文

    未翻译
    在 GitHub 打开

    📖 Introduction

    OmAgent is python library for building multimodal language agents with ease. We try to keep the library simple without too much overhead like other agent framework.

    • We wrap the complex engineering (worker orchestration, task queue, node optimization, etc.) behind the scene and only leave you with a super-easy-to-use interface to define your agent.
    • We further enable useful abstractions for reusable agent components, so you can build complex agents aggregating from those basic components.
    • We also provides features required for multimodal agents, such as native support for VLM models, video processing, and mobile device connection to make it easy for developers and researchers building agents that can reason over not only text, but image, video and audio inputs.

    🔑 Key Features

    • A flexible agent architecture that provides graph-based workflow orchestration engine and various memory type enabling contextual reasoning.
    • Native multimodal interaction support include VLM models, real-time API, computer vision models, mobile connection and etc.
    • A suite of state-of-the-art unimodal and multimodal agent algorithms that goes beyond simple LLM reasoning, e.g. ReAct, CoT, SC-Cot etc.
    • Supports local deployment of models. You can deploy your own models locally by using OllamaOllama or LocalAI.
    • Fully distributed architecture, supports custom scaling. Also supports Lite mode, eliminating the need for middleware deployment.

    🛠️ How To Install

    • python >= 3.10
    • Install omagent_core
      Use pip to install omagent_core latest release. 
      pip install omagent-core
      Or install the latest version from the source code like below. 
      pip install -e omagent-core

    🚀 Quick Start

    Configuration

    The container.yaml file is a configuration file that manages dependencies and settings for different components of the system. To set up your configuration:

    1. Generate the container.yaml file:

      cd examples/step1_simpleVQA
python compile_container.py

      This will create a container.yaml file with default settings under examples/step1_simpleVQA. For more information about the container.yaml configuration, please refer to the container module

    2. Configure your LLM settings in configs/llms/gpt.yml:

      • Set your OpenAI API key or compatible endpoint through environment variable or by directly modifying the yml file
      export custom_openai_key="your_openai_api_key"
export custom_openai_endpoint="your_openai_endpoint"

      You can use a locally deployed Ollama to call your own language model. The tutorial is here.

    Run the demo

    1. Run the simple VQA demo with webpage GUI:

      For WebpageClient usage: Input and output are in the webpage

      cd examples/step1_simpleVQA
python run_webpage.py

      Open the webpage at http://127.0.0.1:7860, you will see the following interface:

    🤖 Example Projects

    1. Video QA Agents

    Build a system that can answer any questions about uploaded videos with video understanding agents. we provide a gradio based application, see details here.

    More about the video understanding agent can be found in paper.

    2. Mobile Personal Assistant

    Build your personal mulitmodal assistant just like Google Astral in 2 minutes. See Details here.

    3. Agentic Operators

    We define reusable agentic workflows, e.g. CoT, ReAct, and etc as agent operators. This project compares various recently proposed reasoning agent operators with the same LLM choice and test datasets. How do they perform? See details here.

    AlgorithmLLMAveragegsm8k-scoregsm8k-cost($)AQuA-scoreAQuA-cost($)
    SC-COTgpt-3.5-turbo73.6980.065.022767.320.6491
    COTgpt-3.5-turbo69.8678.700.678861.020.0957
    ReAct-Progpt-3.5-turbo69.7474.913.463364.570.4928
    POTgpt-3.5-turbo64.4276.880.690251.970.1557
    IO*gpt-3.5-turbo38.4037.830.332838.980.0380

    *IO: Input-Output Direct Prompting (Baseline)

    More Details in our new repo open-agent-leaderboard and Hugging Face space

    💻 Documentation

    More detailed documentation is available here.

    🤝 Contributing

    For more information on how to contribute, see here.
    We value and appreciate the contributions of our community. Special thanks to our contributors for helping us improve OmAgent.

    🔔 Follow us

    You can follow us on X, Discord and WeChat group for more updates and discussions.

    🔗 Related works

    If you are intrigued by multimodal large language models, and agent technologies, we invite you to delve deeper into our research endeavors:
    🔆 How to Evaluate the Generalization of Detection? A Benchmark for Comprehensive Open-Vocabulary Detection (AAAI24)
    🏠 GitHub Repository

    🔆 OmDet: Large-scale vision-language multi-dataset pre-training with multimodal detection network (IET Computer Vision)
    🏠 Github Repository

    ⭐️ Citation

    If you find our repository beneficial, please cite our paper:

    @article{zhang2024omagent,
  title={OmAgent: A Multi-modal Agent Framework for Complex Video Understanding with Task Divide-and-Conquer},
  author={Zhang, Lu and Zhao, Tiancheng and Ying, Heting and Ma, Yibo and Lee, Kyusong},
  journal={arXiv preprint arXiv:2406.16620},
  year={2024}
}

    相关项目

    同分类里 star 最高的项目
    对比全部 →
    MiniCPM-V
    OpenBMB·Python

    MiniCPM-V 是面壁智能开源的多模态视觉语言模型，能在手机、PC 等本地设备上完成看图、听音和对话交互，无需依赖云端。

    多模态+4视觉语言模型
    27126+412.1k202505272026-11-09 07:37 UTC
    LLaVA
    haotian-liu·Python

    开源的多模态视觉指令微调模型，让 LLaMA 类大模型能看图对话，效果对标 GPT-4V，适合做图文问答和视觉助手。

    视觉指令微调+4视觉语言模型
    250772.8kv1.2.02024-08-12 09:52 UTC
    screenpipe
    screenpipe·Rust

    持续录制电脑屏幕和音频，把日常工作流整理成可检索的记录，帮你找出值得自动化的环节，并为 AI 智能体提供上下文。

    屏幕录制+4视觉语言模型
    22335+762.3kapp-v2.7.1452026-11-10 00:40 UTC
    Qwen3-VL
    QwenLM·Jupyter Notebook

    阿里云通义团队开源的多模态大模型系列，能同时理解图像和文本，用于视觉问答、图文推理等视觉语言任务。

    多模态大模型+4视觉语言模型
    20190+151.9k2026-01-30 04:47 UTC