模型评测

自托管 LLM 追踪与评测平台，替代散落的日志和自建 trace 表

替代手写 time 循环的命令行基准测试工具，把多次运行的统计和对比做成一条命令

实验追踪与模型注册的事实标准，现在把 LLM 调用链和评估也纳入同一套后端

用 YAML 声明式配置跑 LLM 评测与红队扫描的 CLI，顶替手写对比脚本和散落的 pytest 断言

自托管的 LLM 追踪与评测平台，替代 LangSmith 那套闭源 SaaS 的追踪、数据集和评估流水线

把 LLM 应用的评估写成 pytest 用例，用断言和指标替代人工看输出

用 LLM 当裁判给 RAG 和 Agent 流水线打分的评测框架，顶替人工标注和零散脚本拼出来的评估流程

跑学术基准复现分数的标准后端，Open LLM Leaderboard 用的就是它

OpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety.

🧊 Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 🍓

🐢 Open-Source Evaluation & Testing library for LLM Agents

Next-generation AI Agent Optimization Platform: Cozeloop addresses challenges in AI agent development by providing full-lifecycle management capabilities from development, debugging, and evaluation to monitoring.

One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks

Python package for the evaluation of odometry and SLAM

中文语言理解测评基准 Chinese Language Understanding Evaluation Benchmark: datasets, baselines, pre-trained models, corpus and leaderboard

A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.

⚡FlashRAG: A Python Toolkit for Efficient RAG Research (WWW2025 Resource)

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Laminar - open-source observability platform purpose-built for AI agents. YC S24.

[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

Inspect: A framework for large language model evaluations

An open-source visual programming environment for battle-testing prompts to LLMs.

Holistic Evaluation of Language Models (HELM) is an open source Python framework created by the Center for Research on Foundation Models (CRFM) at Stanford for holistic, reproducible and transparent evaluation of foundation models, including large language models (LLMs) and multimodal models.

A harness optimized to smaller LLMs

[CVPR2024 Highlight] VBench - We Evaluate Video Generation

Efficient Retrieval Augmentation and Generation Framework

:bar_chart: Benchmark multiple object trackers (MOT) in Python

[NeurIPS '25] Knowledge Graph Generation from Any Text

A framework for comprehensive diagnosis and optimization of agents using simulated, realistic synthetic interactions

Benchmark for vector databases.

A desktop MCP client designed as a tool unitary utility integration, accelerating AI adoption through the Model Context Protocol (MCP) and enabling cross-vendor LLM API orchestration.

lzbench is an in-memory benchmark of open-source compressors

🐰 Bencher - Continuous Benchmarking

[NeurIPS 2025 Spotlight] OpenCUA: Open Foundations for Computer-Use Agents

[EMNLP 2026] MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research · 浏览器里运行的安卓模拟器 · Browser-hosted Android Simulator · Verifiable Evaluation · Scalable Online RL Training

[EMNLP 2024 & AAAI 2026] A powerful toolkit for compressing large models including LLMs, VLMs, and video generative models.

This repo contains the code for "VLM2Vec / MMEB" [ICLR 2025], "VLM2Vec-V2 / MMEB-V2" [TMLR 2026], and "MMEB-V3" [COLM 2026]

ParseBench - A Document Parsing Benchmark for AI Agents

Dataset and benchmark for RAG on company internal documents.

Peak compute, memory bandwidth and latency of CPUs, GPUs and NPUs: the same tests through every API that reaches a device. Command line and app for Windows, macOS, Linux and Android

[ICLR'25] OpenRCA: Can Large Language Models Locate the Root Cause of Software Failures?

OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks

A unified framework for training, fine-tuning, and evaluating World Action Models

[ICML 2026 Oral] Holi-Spatial: Evolving Video Streams into Holistic 3D Spatial Intelligence

Framework for benchmarking vector search engines

Complete AI governance and LLM Evals platform with support for EU AI Act, ISO 42001, NIST AI RMF and 20+ more AI frameworks and regulations. Join our Discord channel: https://discord.com/invite/d3k3E4uEpR

[Neurips 2023 & TPAMI] T2I-CompBench (++) for Compositional Text-to-image Generation Evaluation

Python SDK for running evaluations on LLM generated responses

LLM 并发性能测试工具，支持自动化压力测试和性能报告生成。

VideoGen-Eval: Agent-based System for Video Generation Evaluation