模型评测

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

A command-line benchmarking tool

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.

Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

The LLM Evaluation Framework

Supercharge Your LLM Application Evaluations 🚀

A framework for few-shot evaluation of language models.

OpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety.

🧊 Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 🍓

🐢 Open-Source Evaluation & Testing library for LLM Agents

Next-generation AI Agent Optimization Platform: Cozeloop addresses challenges in AI agent development by providing full-lifecycle management capabilities from development, debugging, and evaluation to monitoring.

One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks

Python package for the evaluation of odometry and SLAM

中文语言理解测评基准 Chinese Language Understanding Evaluation Benchmark: datasets, baselines, pre-trained models, corpus and leaderboard

A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.

⚡FlashRAG: A Python Toolkit for Efficient RAG Research (WWW2025 Resource)

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Laminar - open-source observability platform purpose-built for AI agents. YC S24.

[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

Inspect: A framework for large language model evaluations

An open-source visual programming environment for battle-testing prompts to LLMs.

Holistic Evaluation of Language Models (HELM) is an open source Python framework created by the Center for Research on Foundation Models (CRFM) at Stanford for holistic, reproducible and transparent evaluation of foundation models, including large language models (LLMs) and multimodal models.

A harness optimized to smaller LLMs

[CVPR2024 Highlight] VBench - We Evaluate Video Generation

Efficient Retrieval Augmentation and Generation Framework

:bar_chart: Benchmark multiple object trackers (MOT) in Python

[NeurIPS '25] Knowledge Graph Generation from Any Text

A framework for comprehensive diagnosis and optimization of agents using simulated, realistic synthetic interactions

Benchmark for vector databases.

A desktop MCP client designed as a tool unitary utility integration, accelerating AI adoption through the Model Context Protocol (MCP) and enabling cross-vendor LLM API orchestration.

lzbench is an in-memory benchmark of open-source compressors

🐰 Bencher - Continuous Benchmarking

[NeurIPS 2025 Spotlight] OpenCUA: Open Foundations for Computer-Use Agents

[EMNLP 2026] MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research · 浏览器里运行的安卓模拟器 · Browser-hosted Android Simulator · Verifiable Evaluation · Scalable Online RL Training

[EMNLP 2024 & AAAI 2026] A powerful toolkit for compressing large models including LLMs, VLMs, and video generative models.

This repo contains the code for "VLM2Vec / MMEB" [ICLR 2025], "VLM2Vec-V2 / MMEB-V2" [TMLR 2026], and "MMEB-V3" [COLM 2026]

ParseBench - A Document Parsing Benchmark for AI Agents

Dataset and benchmark for RAG on company internal documents.

Peak compute, memory bandwidth and latency of CPUs, GPUs and NPUs: the same tests through every API that reaches a device. Command line and app for Windows, macOS, Linux and Android

[ICLR'25] OpenRCA: Can Large Language Models Locate the Root Cause of Software Failures?

OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks

A unified framework for training, fine-tuning, and evaluating World Action Models

[ICML 2026 Oral] Holi-Spatial: Evolving Video Streams into Holistic 3D Spatial Intelligence

Framework for benchmarking vector search engines

Complete AI governance and LLM Evals platform with support for EU AI Act, ISO 42001, NIST AI RMF and 20+ more AI frameworks and regulations. Join our Discord channel: https://discord.com/invite/d3k3E4uEpR

[Neurips 2023 & TPAMI] T2I-CompBench (++) for Compositional Text-to-image Generation Evaluation

Python SDK for running evaluations on LLM generated responses

LLM 并发性能测试工具，支持自动化压力测试和性能报告生成。

VideoGen-Eval: Agent-based System for Video Generation Evaluation