登录

evalscope

modelscopeApache-2.0活跃维护

轻量可定制的大模型评测框架，支持 LLM、VLM、AIGC 和 RAG 的效果评估与性能压测，帮团队快速跑通基准测试。

A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.

v1.14.1发布于 2026-10-26 05:27 UTC3,678star +17 / 7 天564fork同步于 2026-11-10 11:40 UTC
模型评测性能压测大模型多模态评测RAG评估
对比 GitHub 官网evaluationllmperformanceragvlm

怎么上手

由 README 提炼，执行前请确认
原文 
pip install evalscope
evalscope eval --model your-model-name --api-url $OPENAI_API_BASE_URL --api-key $OPENAI_API_KEY --eval-type openai_api --datasets gsm8k --limit 5
硬件要求按 README 说明

README 未说明具体显存要求；评测在线 OpenAI 兼容 API 时无需 GPU，评测本地模型会自动从 ModelScope 下载权重。

纯 CPU 可运行
下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼
没人工点评？写下你的经验

一站式大模型评测框架，一条命令完成能力评测、推理压测与结果可视化

轻量可定制的大模型评测框架，支持 LLM、VLM、AIGC 和 RAG 的效果评估与性能压测，帮团队快速跑通基准测试。

核心能力

  • 内置 MMLU、C-Eval、GSM8K 等主流评测基准
  • 支持 LLM、VLM、Embedding、Reranker、AIGC 多类模型
  • 集成 OpenCompass、VLMEvalKit、RAGEval 多评测后端
  • 提供 TTFT、TPOT、吞吐等推理性能压测指标
  • Web Dashboard 做多模型对比与逐样本结果查看

容易踩的坑

可视化面板需额外安装 pip install 'evalscope[service]'，默认访问 http://127.0.0.1:9000；评测本地模型会自动下载权重，需提前配置好 ModelScope 访问；Python 版本需不低于 3.10。

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

最近发版

5 个版本v1.14.1
全部 release
  1. v1.14.1最新正式版2026-10-26 · 2026-10-26 05:27 UTC

    中文版

    基准测试数据集

    • 新增 SafetyBench 零样本安全评测，支持 zh、en、zh_subset，分别报告整体及七类准确率；无法解析或缺失的预测计错，不随机回退到某个答案。

    功能增强

    • Web 任务工作区：评测与压测共用模型连接，支持独立草稿、配置模板、导入导出及活动任务发现；刷新页面后可恢复监控，无需重复提交。
    • 任务辨识：通过可选别名和脱敏后的端点区分同名模型的运行，在监控、报告、对比和 HTML 中展示；别名不改变评测标识。
    • Bayes@N：新增二值均匀先验聚合，报告后验均值和单独的后验标准差，适用于完整固定重复次数的 accuracy / exact_match。
    • CLI：新增 --dry-run，解析并打印配置后退出，不执行推理。

    文档优化

    • 补充音频基准所需的服务端 input_audio 支持及 SGLang 版本要求。
    • 新增 SafetyBench、Bayes@N 和 Web 任务工作区说明，并更新缓存重新评分失败后的恢复方式。

    问题修复

    • 依赖：移除核心安装中的 LiteLLM 依赖；显式选择 eval_type='litellm' 时仍可单独安装使用。
    • 缓存一致性：修复 rerun_review=True 失败后普通续跑静默复用旧分数的问题，原子保存配置并持久化待完成评分状态，保留原始预测来源。
    • 数学评分：使用进程隔离为支持 fork 的非 daemon 进程中的数学评分调用增加计算与结果传输期限，超时样本按执行失败排除，避免退化输入长时间占用所有评分线程。
    • 模型响应：拒绝缺少完整结束状态的 OpenAI / LiteLLM 流，识别 HTTP 200 内嵌的 SSE 错误和错误响应体；流读取失败纳入重试，避免保存部分答案或把失败计为正常输出。
    • 代码评测：修复 MBPP+ 未执行扩展测试及 test_imports、HumanEval 本地超时过早开始计时，以及 SciCode 将代码块外文本送入执行的问题。
    • 多模态评测：按官方 leave-one-out 方式计算 VQAv2 分数；ScienceQA 加入题目 hint；ARC-AGI-2 评测每个 test 输入并按任务汇总；MMAU 使用有公开答案的 test_mini，取消缺省选择 A。
    • ASR：LibriSpeech、FLEURS、CommonVoice15 按全语料累计编辑数与参考词数计算 WER；TORGO 同时修复 corpus WER 和 CER。
    • 视频描述：MSR-VTT、MSVD 按完整子集计算 CIDEr 和 BLEU，避免指标随评分批大小变化。
    • 问答与分类：修复 DocMath 裁判问题及带符号数值答案、FRAMES 答案行、TriviaQA 官方规范化、Drivel 多标签、PubMedQA 单标签、HaluEval 与 CoinFlip 评分和汇总一致性。
    • 评分协议：修复 EQ-Bench v2 的排除与报告尺度；修正 SPaRC 的星形、正负多格骨牌规则提示。
    • 数据导出：修复 JSONL 转 CSV 时值与字段名未对齐的问题。

    升级说明

    • 需要 LiteLLM 后端的用户请显式执行 pip install litellm；基础 OpenAI、System One 等路径不再通过核心依赖安装它。第三方后端仍可能自行依赖 LiteLLM。
    • 失败或中断的重新评分会阻止普通缓存续跑；请使用 rerun_review=True / --rerun-review 完成恢复。预测文件仍可复用，不会把旧分数当作新评测结果。
    • 多项基准的输入或评分发生变化，请核对 evaluation_version。ScienceQA、SPaRC 等提示变化需要重新生成预测；评分修复可对旧预测重新评分。
    • 数学评分的 30 秒期限从子进程启动后开始，覆盖计算和完整结果传输，不覆盖预热、等待 fork 锁、启动、父进程解码和清理。Windows 等无 fork 环境及 daemon 进程仍为无期限的进程内路径。本次执行期限变化不会自动使旧数学 reviews 失效，需要验证新策略时请显式重新评分。
    • Bayes@N 当前仅支持一个选定子集，不支持 collection、多子集或缺失/重复试验；默认聚合方式不变。
    • Web 连接与模板默认不保存或导出 API key，携带密钥需显式选择并确认。任务注册表仍是进程内的，服务重启不提供持久调度或孤儿子进程托管。

    English Version

    Benchmark Datasets

    • Added zero-shot SafetyBench evaluation for zh, en and zh_subset, with separate overall and seven-category accuracies. Missing and unparsed predictions count as incorrect, without random answer fallback.

    Feature Enhancements

    • Task Workspace: Evaluation and performance testing share model connections, with independent drafts, reusable templates, import/export and active-task discovery. Reloading reconnects monitoring without resubmission.
    • Run Identity Display: Optional aliases and sanitized endpoints distinguish runs using the same model across monitors, reports, comparisons and HTML. Aliases do not change evaluation identity.
    • Bayes@N: Added binary uniform-prior aggregation with a posterior mean and a separate posterior standard deviation for complete fixed-repeat accuracy / exact_match trials.
    • CLI: Added --dry-run to parse and print configuration without running inference.

    Documentation

    • Documented server-side input_audio support and SGLang version requirements for audio benchmarks.
    • Added SafetyBench, Bayes@N and task-workspace guides, and documented recovery from failed review reruns.

    Bug Fixes

    • Dependencies: Removed LiteLLM from the core installation. Users selecting eval_type='litellm' can install it separately.
    • Cache Consistency: Fixed stale review reuse after a failed rerun_review=True by atomically publishing configuration and persisting pending review state while preserving prediction provenance.
    • Mathematical Scoring: Added process isolation and a computation/result-transfer deadline in fork-capable, non-daemonic processes. Timed-out samples are excluded as execution failures rather than freezing scoring threads on degenerate inputs.
    • Model Responses: Reject incomplete OpenAI/LiteLLM streams and detect in-band SSE errors and error response bodies. Stream-consumption failures participate in retries instead of persisting partial answers or treating failures as valid output.
    • Code Evaluation: Fixed MBPP+ ext
  2. v1.14.02026-10-20 · 2026-10-20 10:31 UTC
    查看更新说明

    中文版

    基准测试数据集

    • 决策评测：新增 JevBench、OpenDecisions、STS-B 和自定义 JSONL 数据集 general_decision，覆盖 Choice、Noul 和 Score 任务。
    • 代码与数据库评测：新增 DS-1000 数据科学代码生成和 BIRD-SQL Mini-Dev 文本转 SQL 评测，使用执行结果评分。
    • 表格与金融评测：新增 TableBench、FinQA 和 TAT-QA，分别支持官方 Mix_Metric、程序执行/等价性评分及考虑数值尺度的 EM/F1。
    • 空间推理评测：新增 SPaRC，评估二维网格中的路径规划和规则约束满足能力。
    • 多模态评测：新增 VisualOverload，保留官方提示与原始图像，并导出 visualoverload_submission.json 供官方服务器评分。

    功能增强

    • 决策模型：扩展 System One 并新增 AutoTrust API，统一校验结构化决策请求、答案类型与概率分布；支持 BoolQ 的 Noul 转换和 BANKING77 的固定语义类别。
    • 概率指标：新增分类、概率校准、相关性、序数评分及风险/覆盖率与 AURC 分析；保留原始概率，并单独记录用于评分的分布。
    • 决策可视化与压测：Web UI 支持查看逐样本选项概率、预测与参考答案；新增 perf --api systemone / autotrust 及 open_decisions 压测数据集。
    • 数据集加载：通过共享 DatasetSource 自动解析已声明的 Hugging Face 数据集 ID、版本与文件路径；覆盖基准加载及相关压测插件，保留本地路径优先级。
    • 共享前缀池：新增 --prefix-count，按请求索引轮询分配随机 token 前缀；固定配置与种子时，串行和多进程生成保持一致。
    • 报告展示：CLI、HTML 和 Web 指标标签显示 Context、Depth、Threshold、Min/Max tokens 等数值维度名称。

    文档优化

    • 新增中英文决策模型评测实践、自定义决策数据、API 协议及各新增基准说明。
    • 更新共享前缀池、跨数据源加载和指标维度的用法及示例。
    • 修正离线模型下载命令、mini-swe-agent 和 ms-swift 链接，并同步 AI2D 中文说明。

    问题修复

    • 修复 Needle Haystack 动态指标名导致的语义缺失、主指标缺失与热力图生成失败，按 context/depth 维度保留分格评分。
    • 修复 MMBench 的 nan 提示及提示与问题拼接；修复 MMMU 开放题数值与多参考答案评分。
    • 修复 BigCodeBench / BigCodeBench-Hard 仅定义而未执行 TestCases，导致错误代码被判为通过的问题。
    • 修复 ZebraLogicBench 未解析答案未计入部分分母及多行 JSON 解析；修复 ToolBench 不适用指标被按零分参与平均的问题。
    • 修复 OlympiadBench 全角逗号答案、数学标记包裹的选择题标签、加粗或不同大小写的 ANSWER: 行，以及 ANLS 对 JSON 字面量参考答案的处理。
    • 修复 OmniDocBench 小样本未同时覆盖中英文时的缺失指标访问，以及 general_mcq 自定义提示模板被默认值覆盖的问题。
    • 修复 Seed-TTS-Eval 混合语言样本共用首个样本的 WER 语言配置，以及无法评分的音频被计为完美 WER 0 的问题。
    • 修复 CLIP 零样本分类在新 NumPy 下的标量转换、limit 截断和 skip_existing 提前终止整个任务的问题。
    • 修复一条记录生成多个样本时，repeats 将不同问题分到同一重复组，影响 pass@k / vote@k 的问题。
    • 修复预测缓存恢复时同一样本重复评分，保留每个样本最后一条有效预测；支持带 UTF-8 BOM 的本地 CSV、TSV 和 JSONL。
    • 修复并发数、超时、流式开关及重试等传输配置变化导致缓存评测标识不必要变化的问题。
    • 修复逐次调用的 GenerateConfig 默认值覆盖用户生成配置，以及 LiteLLM 流式延迟未包含流读取时间的问题。
    • 修复 HTTP 200 的 SSE 流中错误事件被计为成功请求，以及批量评分子集全部失败时聚合崩溃的问题。
    • 修复 Web 页面刷新后丢失活动任务监控，恢复进度、日志与停止操作；仅在会话存储中保留任务 ID。
    • 修复工具 Schema 对 dataclass 字符串类型注解的解析，排除伪字段，并在类型注解无法解析时保留字段及回退行为。

    升级说明

    • 多项基准的评测语义已更新，请核对 evaluation_version。MMBench v1.1 改变了输入提示，需重新生成预测；评分类修复可使用 rerun_review=True / --rerun-review 重新评分。
    • 缓存标识对生成配置做了规范化。升级后旧缓存可能需要一次 rerun_review=True；之后仅调整并发或传输参数不会使相同评测请求失配。
    • VisualOverload 不提供本地答案准确率；response_present 仅表示响应是否非空。请使用 repeats=1，并将完整提交文件上传至官方评分服务。
    • TableBench 当前包含 836 道文本答案题，不包含需要执行绘图代码的 Visualization 题。
    • DS-1000 默认在本地子进程执行代码，不构成安全沙箱。运行不可信代码请启用 sandbox，并使用对应依赖环境；TensorFlow 子集的预置容器要求 Linux/amd64。
    • 决策 API 仅支持文本任务；AutoTrust 的 Score 协议固定为六级（0-5），不会隐式重映射其他评分尺度。

    English Version

    Benchmark Datasets

    • Decision Evaluation: Added JevBench, OpenDecisions, STS-B and custom JSONL datasets through general_decision, covering Choice, Noul and Score tasks.
    • Code and Database Evaluation: Added DS-1000 data science code generation and BIRD-SQL Mini-Dev text-to-SQL evaluation with execution-based scoring.
    • Table and Financial Evaluation: Added TableBench, FinQA and TAT-QA, using the official Mix_Metric, program execution/equivalence metrics, and scale-aware EM/F1 respectively.
    • Spatial Reasoning: Added SPaRC for two-dimensional grid pathfinding under rule constraints.
    • Multimodal Evaluation: Added VisualOverload with official prompts, original images and visualoverload_submission.json export for official server-side grading.

    Feature Enhancements

    • Decision Models: Extended System One and added AutoTrust APIs with shared validation of typed requests, answer kinds and probability distributions. Added BoolQ Noul conversion and stable semantic class identities for BANKING77.
    • Probability Metrics: Added classification, calibration, correlation, ordinal scoring, risk/coverage and AURC analysis. Raw probabilities remain available separately from scoring distributions.
    • Decision UI and Performance Testing: Inspect per-sample probabilities, predictions and references in the Web UI. Added perf --api systemone / autotrust and the `open_deci
  3. v1.13.02026-10-13 · 2026-10-13 10:27 UTC
    查看更新说明

    中文版

    基准测试数据集

    • 多轮对话评测：新增 MT-Bench，支持两轮对话和可配置的 Judge，遵循官方评分流程。
    • Agent 评测：新增 Terminal-Bench 4.0（terminal_bench_v4），固定官方数据版本 4.0.0，支持任务筛选、重复试验及独立验证环境。
    • 决策能力评测：新增 ANLI、BoolQ、BANKING77、ContractNLI、RewardBench v1，默认从 ModelScope 加载，支持普通 LLM 和 System One Choice 评测。

    功能增强

    • System One Choice：新增 eval_type='systemone_api'，通过现有模型接口支持文本单选决策模型，并保留选项概率、置信度和请求信息。
    • 数学评分：统一使用固定版本 Math-Verify 进行答案提取、解析与等价判断，覆盖 29 个基准，保留各基准的容差、部分得分及多答案规则。
    • 基准加载：适配器改为按需加载，减少 CLI 和库导入时加载整个基准目录的开销。
    • 数据集缓存：Native 加载支持复用 ModelScope SDK 本地快照；优化临时 Arrow 缓存、并发写入和中断恢复。
    • 多模态性能测试：新增真实 MMMU 多图数据模式，保留提示中的图像占位顺序。
    • 多轮性能测试：支持会话 ID 与逐轮请求 ID，SWE-Smith 数据构建支持副本生成，并减少延迟指标收集开销。
    • SLA 自动调优：支持显式 s / ms 阈值单位，校验指标有效性，使用准确请求计数，并改进约束与优化搜索。
    • Dashboard：展示已安装的 EvalScope 版本，并提供本地化说明。

    文档优化

    • 更新 System One、Terminal-Bench 4.0、SLA 调优、多轮性能测试和数据缓存使用指南。
    • 新增 DeepSearchQA 的 Parallel Search MCP 示例与 Qwen-Image 文生图评测示例。
    • 修复性能结果分析、函数调用 API key、Python 3.10 示例以及多处外部文档链接。

    问题修复

    • 修复 AI2D 图像版本按图而非逐题选择的问题，依据官方 abcLabel 标注选择原图或字母标注图。
    • 修复 IFBench 关键词/姓名子串匹配、部分校验器与上游不一致及无单词回复的除零问题。
    • 修复 AlpacaEval 胜负被计为 0.75/0.25、HallusionBench 分组、OCRBench-v2 类别宏平均、PubMedQA 未识别答案分母及 collection 宏平均问题。
    • 修复 BFCL-v3、ACEBench 和 OCRBench-v2 汇总行重复计入总分/样本数，以及 vote@k 将空答案纳入投票的问题。
    • 修复 C-Eval 答案格式解析、中文多选分隔符、MMLU-Redux 多答案标注和 BBH 自由答案括号保留问题。
    • 修复 NER 重复实体与 BIO 边界、LiveCodeBench sandbox 输出比较和视频压缩包文件匹配问题。
    • 修复流式性能测试中的工具调用 TTFT、推理文本 token 估算、Responses ITL/失败识别/缓存用量、单 token TPOT、分位数精度与 NaN、推测解码估算及 embedding/rerank 响应体延迟统计。
    • 修复 Anthropic 图片 URL 和 thinking/signature 历史保留、Responses 中断重试及自定义 finish reason 处理。
    • 修复 Agent 提交数值 0、临时 HOME 清理、工具 Schema 字面量、缺省样本 metadata、本地音视频路径，以及未测量 PD 指标导致性能详情 API 返回 500 等问题。
    • 修复 Terminal-Bench 的 Codex max_turns 兼容问题，并更新 Full CI 的文本/视觉模型配置。

    升级说明

    • 数学评分涉及的 29 个基准、22 个 NER 基准及 AI2D、IFBench 等评分修复均更新了相应评测版本。比较结果时请核对 evaluation_version；旧预测可通过 rerun_review=True / --rerun-review 重新评分，旧 reviews 不能直接跨评测版本复用。
    • AI2D v1.1 使用修正后的逐题图像，结果与 v1.0 不直接可比。Terminal-Bench 4.0 结果也不直接与 2.0/2.1 比较。
    • Math-Verify 使用 ANTLR 4.13.2；当前 ms-vlmeval==0.0.20 要求 ANTLR 4.11.1，因此 vlmeval / all extras 的联合依赖安装存在冲突，请使用独立环境。数学解析与验证不提供硬超时。
    • System One 仅支持文本单选；RewardBench v1 的整体 accuracy 按样本加权，ContractNLI 当前评估文档分类 accuracy，请勿将它们视为完整复现官方排行榜的全部指标。

    English Version

    Benchmark Datasets

    • Multi-turn Evaluation: Added MT-Bench with two-turn conversations, configurable judges, and the official judging flow.
    • Agent Evaluation: Added Terminal-Bench 4.0 (terminal_bench_v4), pinned to official revision 4.0.0, with task filtering, repeated trials, and separate verifier environments.
    • Decision Evaluation: Added ANLI, BoolQ, BANKING77, ContractNLI, and RewardBench v1, defaulting to ModelScope and supporting both standard LLM and System One Choice evaluation.

    Feature Enhancements

    • System One Choice: Added eval_type='systemone_api' for text single-choice decision models through the existing model interface, retaining option probabilities, confidence, and request metadata.
    • Mathematical Scoring: Unified extraction, parsing, and equivalence checking with pinned Math-Verify across 29 benchmarks, preserving benchmark-specific tolerances, partial credit, and multiple-answer policies.
    • Benchmark Loading: Load adapters on demand to reduce full-catalog import overhead for CLI and library use.
    • Dataset Caching: Reuse local ModelScope SDK snapshots in Native loading and improve temporary Arrow storage, concurrent cache publication, and interruption recovery.
    • Multimodal Performance Testing: Added a real MMMU multi-image mode with correct prompt/image placeholder ordering.
    • Multi-turn Performance Testing: Added conversation and per-turn request IDs, SWE-Smith dataset replication, and lower latency-metric collection overhead.
    • SLA Auto-tuning: Added explicit s / ms threshold units, metric-validity checks, exact request-count handling, and improved constraint/optimization searches.
    • Dashboard: Display the installed EvalScope version with localized guidance.

    Documentation

  4. v1.12.02026-09-16 · 2026-09-16 08:15 UTC
    查看更新说明

    中文版

    基准测试数据集

    • 语音评测：新增 THCHS-30 国际音标（IPA）音素识别评测，支持 Phone Error Rate（PER）指标。
    • 语音评测：LibriSpeech 新增 test-other 子集支持。

    功能增强

    • Agent 评测：新增 DeepSeek Harness Bridge Runner，支持通过 OpenAI Chat Completions Bridge 运行 deepseek-harness。
    • Agent 评测：新增可选的工具调用参数 Schema 校验，帮助区分模型参数生成错误与工具执行错误。
    • Agent 基准：Tau3-Bench 支持配置 Agent 最大执行步数 max_steps。
    • 性能测试：新增基于 AIPerf 的 AgentX 回放场景，支持数据版本校验、运行有效性记录和敏感信息脱敏。
    • 性能测试：新增可选 PD 分离式推理指标，包括 Steady ITL、PD Handoff Latency 和 PD Handoff Overhead。
    • 性能测试：AgentX 支持 tokenizer_trust_remote_code 与 benchmark_grace_period 配置。
    • Dashboard：报告列表支持按模型展示同一模型的多个报告，保持原始评测报告不变。
    • Dashboard：聚合结果表首屏按批次展示，优化大规模模型与基准组合下的页面性能。

    文档优化

    • 更新 Agent Bridge、AgentX 性能测试、Tau3-Bench、评测架构和基准贡献说明。
    • 修复基准 smoke evaluation 示例命令，并补充正确的 mock 评测配置。
    • 更新产品网站入口及相关使用指引。

    问题修复

    • 修复多参考答案在 inclusion-based 评分中被拼接为单个错误目标的问题，影响 TriviaQA、MMLU-Redux 等评测。
    • 修复 Judge 模板渲染、指标异常处理、包含匹配规则及 GeneralArena bootstrap 随机数处理，提升评测结果的正确性与可复现性。
    • 修复 GeneralArena 导入历史评测结果时按行错配、将已生成回复带入 Judge 输入的问题；评测版本升级至 v1.2。
    • 修复 Agent Bridge 丢失模型原始输出、性能指标和推理内容，以及上游生成失败未记录到 Agent Trace 的问题。
    • 修复 Agent sandbox stdin 隔离和显式 stdin 传递问题。
    • 修复 MCP 初始化错误被隐藏、service worker 无法运行 MCP stdio 客户端的问题。
    • 修复空 ASR 转写被错误计为 WER=0 的问题；Seed-TTS-Eval 评测版本升级至 v1.1。
    • 修复流式推理输出的 TTFT/ITL 统计、多轮 warmup 交接，以及带 token usage 的 Agent Trace 导致 Predictions API 返回 500 的问题。
    • 修复 few-shot 配置未在数据加载前校验、CLI benchmark-info 导入错误、Dashboard 空分数行模型名异常换行等问题。
    • 修复可用模块的导入期异常被误报为缺失依赖的问题，并移除会直接报错的 CLIP Benchmark 模板入口。

    English Version

    Benchmark Datasets

    • Speech Evaluation: Added THCHS-30 IPA phoneme recognition with Phone Error Rate (PER) support.
    • Speech Evaluation: Added the LibriSpeech test-other subset.

    Feature Enhancements

    • Agent Evaluation: Added the DeepSeek Harness Bridge Runner for running deepseek-harness through the OpenAI Chat Completions Bridge.
    • Agent Evaluation: Added optional tool-call argument Schema validation to distinguish invalid model arguments from tool execution failures.
    • Agent Benchmarks: Added configurable Agent max_steps support for Tau3-Bench.
    • Performance Testing: Added an AIPerf-based AgentX replay scenario with dataset-version validation, run-validity records, and secret redaction.
    • Performance Testing: Added optional PD-disaggregated serving metrics, including Steady ITL, PD Handoff Latency, and PD Handoff Overhead.
    • Performance Testing: Added tokenizer_trust_remote_code and benchmark_grace_period options for AgentX.
    • Dashboard: Added display-only grouping of reports from the same model without mutating source evaluation reports.
    • Dashboard: Added incremental rendering for aggregated results to improve large-scale dashboard performance.

    Documentation

    • Updated documentation for Agent Bridge, AgentX performance testing, Tau3-Bench, evaluation architecture, and benchmark contributions.
    • Fixed the benchmark smoke-evaluation example and documented the correct mock-evaluation configuration.
    • Updated product website entry points and related guidance.

    Bug Fixes

    • Fixed multi-reference answers being flattened into an invalid target in inclusion-based scoring, affecting benchmarks such as TriviaQA and MMLU-Redux.
    • Fixed Judge template rendering, metric-failure handling, inclusion matching, and GeneralArena bootstrap RNG behavior for more correct and reproducible evaluation results.
    • Fixed mispaired imported reviews and generated responses leaking into Judge inputs in GeneralArena; advanced its evaluation version to v1.2.
    • Fixed Agent Bridge loss of original model outputs, performance metrics, reasoning content, and upstream generation failures in Agent Traces.
    • Fixed Agent sandbox stdin isolation and explicit stdin forwarding.
    • Fixed hidden MCP initialization errors and MCP stdio clients failing in service workers.
    • Fixed empty ASR transcripts being scored as WER=0; advanced Seed-TTS-Eval to evaluation version v1.1.
    • Fixed TTFT/ITL timing for
  5. v1.11.12026-08-31 · 2026-08-31 09:07 UTC
    查看更新说明

    中文版

    基准测试数据集

    • 多模态评测：新增 SURDS、VLMs Are Biased、Ref-Adv-s、VisFactor、MedXpertQA、VTCBench 等视觉、多模态与长视频理解基准。
    • 推理与专业能力评测：新增 $OneMillion-Bench、PRBench、HMMT-Nov-2025 等 Agent、深度推理与数学评测基准。

    功能增强

    • API 契约：Web API 响应契约改由 Pydantic 模型自动生成，提升前后端类型一致性。
    • 缓存与数据集：隔离数据集缓存键，增强视频解码与 limit 参数校验。
    • 模型服务：支持识别 HTTP 200 响应中的网关错误载荷，并优化音频预处理、流式响应、用量统计与重试行为。
    • 性能测试：优化指标精度、关闭流程、闭环 warmup 交接，以及 SSE Unicode 内容处理。

    文档优化

    • 更新 README 与 Dashboard 使用说明。
    • 修复 Model API 文档中的过期说明。

    问题修复

    • 修复 ProcessBench 等运行只产生部分指标时的报告异常：主指标不可用将被明确标记，不再错误地替换为辅助分数。
    • 修复 General QA 在错误样本下的指标身份保留、指令级指标聚合、答案解析和 ROUGE 评分问题。
    • 修复 IFBench 重复评测版本及唯一词约束校验问题。
    • 修复 OmniDocBench 空页面指标、VTCBench HTML 标签解析、Toolathlon 任务生命周期与结果校验问题。
    • 修复 Judge 延迟初始化并发、NLTK 镜像文件校验、缓存恢复不完整记录等问题。

    English Version

    Benchmark Datasets

    • Multimodal Evaluation: Added SURDS, VLMs Are Biased, Ref-Adv-s, VisFactor, MedXpertQA, VTCBench, and other vision, multimodal, and long-video benchmarks.
    • Reasoning and Domain Evaluation: Added $OneMillion-Bench, PRBench, and HMMT-Nov-2025 for agent, deep-reasoning, and mathematical evaluation.

    Feature Enhancements

    • API Contracts: Web API response contracts are now generated from Pydantic models for stronger frontend/backend type consistency.
    • Cache and Datasets: Isolated dataset cache keys and improved video decoding and limit validation.
    • Model Services: Detects gateway-error payloads returned with HTTP 200, and improves audio preprocessing, streaming responses, usage accounting, and retry behavior.
    • Performance Testing: Improved metric accuracy, shutdown handling, closed-loop warmup handoff, and Unicode handling in SSE streams.

    Documentation

    • Updated README and Dashboard guidance.
    • Fixed stale Model API documentation.

    Bug Fixes

    • Fixed report failures when ProcessBench and similar runs emit only part of their metrics: unavailable primary metrics are now reported explicitly without substituting an auxiliary score.
    • Fixed metric-identity preservation for failed General QA samples, instruction-level aggregation, answer parsing, and ROUGE scoring.
    • Fixed duplicate evaluation versions and unique-word validation in IFBench.
    • Fixed empty-page metrics in OmniDocBench, HTML tag parsing in VTCBench, and Toolathlon job lifecycle and result validation.
    • Fixed concurrent lazy Judge initialization, NLTK mirror archive verification, and incomplete cache-resume records.

    What's Changed

用户点评

登录用户的真实使用记录，可按版本和硬件筛选
共 0 条

    还没有人点评。第一个说出真实体验的人最有价值。

    README 原文

    未翻译
    在 GitHub 打开

    ⭐ If you like this project, please click the "Star" button in the upper right corner to support us. Your support is our motivation to move forward!

    📝 Introduction

    EvalScope is a one-stop LLM evaluation framework built by the ModelScope Community. Just one command to start — it supports model capability evaluation, inference performance stress testing, and result visualization.

    pip install evalscope
evalscope eval --model your-model-name --api-url $OPENAI_API_BASE_URL --api-key $OPENAI_API_KEY --eval-type openai_api --datasets gsm8k --limit 5

    ✨ Key Features

    • 📚 Comprehensive Evaluation Benchmarks: Built-in multiple industry-recognized evaluation benchmarks including MMLU, C-Eval, GSM8K, and more.
    • 🧩 Multi-modal and Multi-domain Support: Supports evaluation of various model types including Large Language Models (LLM), Vision Language Models (VLM), Embedding, Reranker, AIGC, and more.
    • 🧭 System One Evaluation: Evaluate text Choice decision models via --eval-type systemone_api, with task instructions, few-shot examples and structured probabilities/confidence. See the usage guide.
    • 🚀 Multi-backend Integration: Seamlessly integrates multiple evaluation backends including OpenCompass, VLMEvalKit, RAGEval to meet different evaluation needs.
    • 🤖 Agent Evaluation Mode: Drives benchmarks (e.g. GSM8K, AIME, SWE-bench Agentic) inside a controlled multi-turn AgentLoop with pluggable strategies, tools and Docker sandbox; full per-sample Agent Trace is recorded and visualizable.
    • ⚡ Inference Performance Testing: Provides powerful model service stress testing tools, supporting multiple performance metrics such as TTFT, TPOT.
    • 📊 Interactive Reports: Provides a Web Dashboard for multi-dimensional model comparison, report overview and detailed inspection.
    • ⚔️ Arena Mode: Supports multi-model battles (Pairwise Battle), intuitively ranking and evaluating models.
    • 🔧 Highly Extensible: Developers can easily add custom datasets, models and evaluation metrics.

    📊 Visualization Preview

    EvalScope provides an interactive Web Dashboard for multi-dimensional model comparison and in-depth analysis.

    For details, please refer to 📖 Visualizing Evaluation Results.

    🎉 What's New

    • 🔥 [2026.10.09] Added System One Choice evaluation and ANLI, BoolQ, BANKING77, ContractNLI, RewardBench v1 benchmarks. All five default to ModelScope and also support standard LLM evaluation. See System One Evaluation.
    • 🔥 [2026.08.24] v1.11.0 Introduced published evaluation versions for reproducible benchmark results; improved report semantics and incomplete-run handling; strengthened multimodal media loading and task-config validation.
    • 🔥 [2026.08.13] Improved evaluation reports with unified metric semantics and more reliable Agent Trace step grouping and tool-call/result linking.
    • 🔥 [2026.08.10] Added AutomationBench, JobBench, MiniWoB, OmniDocBench-v1.6, PerceptionBench, ScreenSpot-Pro, PLawBench, PMC-VQA, HiPhO, LogicVista, and CC-OCR-V2 benchmarks.
    • 🔥 [2026.07.21] Added Claw-Eval, ResearchRubrics, Toolathlon (agent), TVBench (video), WideSearch, and PerspectiveGap benchmarks.
    • 🔥 [2026.07.03] Added CharXiv & BabyVision (chart understanding, visual cognition) and ERQA & WorldVQA (entity-recognition QA with LLM-judge + CoT) multimodal benchmarks.
    • 🔥 [2026.06.23] Major agent & code evaluation expansion: added BigCodeBench, SWE-bench Multilingual, BrowseComp, MCP-Atlas, GDPval benchmarks; added OpenCode / OpenHands runners; refactored adapter architecture with AudioLanguageAdapter, unified FunctionCallAdapter, and public run_agent_loop API.
    • 🔥 [2026.06.16] Added full-reference image quality metrics (SSIM, PSNR, etc.), long-context benchmarks (LoCoMo QA, LongMemEval), Caption & Maritime-OCR-Bench benchmarks; perf module now supports unified --data-source and parallelized request generation.
    • 🔥 [2026.06.02] Refactored RAG evaluation module: upgraded to MTEB 2.x and RAGAS 0.4.x, with unified Pydantic-based configs. See the RAGEval guide.
    • 🔥 [2026.05.27] Added Trie agentic trace replay for perf benchmarking: three new dataset plugins (trie_agentic_coding / trie_code_qa / trie_office_work) replay real multi-turn agent traces with per-turn token caps and tool-call latency simulation. Also introduced a --duration wall-clock budget for all benchmark modes and a Turn dataclass for per-turn overrides.
    • 🔥 [2026.05.27] Added Vendor Verifier benchmarks (k2_verifier, kimi_verifier, minimax_verifier) for validating whether third-party API deployments faithfully reproduce official model behavior, with a shared FunctionCallAdapter base class.
    • 🔥 [2026.05.26] Added the GAIA agent benchmark (multi-turn ReAct + bash in a Docker sandbox, official rule-based scorer) and generic MCP server support — any NativeAgentConfig-driven benchmark can now plug in stdio / HTTP / SSE MCP servers (fetch, web search, GitHub, ...) without per-benchmark wiring.
    • 🔥 [2026.05.22] Introduced the External Agent Bridge mode: evaluate off-the-shelf agent CLIs such as Anthropic's Claude Code and OpenAI's Codex directly through EvalScope. The bridge transparently forwards each CLI's LLM traffic (Anthropic Messages / OpenAI Chat / OpenAI Responses, including SSE streaming) to your evaluation model, while recording the full trajectory as an agent_trace. Bring-your-own-runner via @register_runner. See the External Agent Bridge guide.
    • 🔥 [2026.05.19] Added support for SWE-bench_Pro and τ³-bench: SWE-bench_Pro is a more challenging multilingual long-horizon software-engineering benchmark from Scale AI (recommended over the original SWE-bench for less data contamination and broader language coverage; per-instance Docker images are pulled directly from DockerHub, no local image build required); τ³-bench is the v1.0.0 release of the tau-bench family, extending τ²-bench with a new banking_knowledge retrieval domain (RAG), 75+ task fixes across existing domains, and pluggable retrieval pipelines (BM25 / embeddings / rerankers / sandbox shell).

    • 🔥 [2026.05.15] Introduced Agent Evaluation Mode: any benchmark based on DefaultDataAdapter (GSM8K, AIME, IFEval, etc.) can now be driven through a multi-turn AgentLoop with pluggable strategies (function_calling / react / swe_bench_*), tools (bash / python_exec / submit) and local / docker environments. Per-sample agent_trace is recorded and rendered step-by-step in the dashboard's Predictions tab. See the Agent Evaluation guide for details.

    • 🔥 [2026.05.08] Partnered with LightSeek to launch TokenSpeed, a speed-of-light LLM inference engine for agentic workloads. EvalScope provides the SWE-smith benchmarking pipeline — using real coding-agent traces to measure per-GPU throughput (TPM) and per-user latency (TPS) — serving as the official benchmark tool for TokenSpeed performance evaluation. Refer to the SWE-smith usage guide to get started.

    • 🔥 [2026.05.07] Replaced the Gradio-based WebUI with a new React + Vite web interface for better performance and user experience.

    • 🔥 [2026.04.23] Added support for recording performance (perf) metrics during evaluation tasks, enabling simultaneous tracking of model accuracy and inference efficiency metrics such as TTFT, TPOT, and throughput in a single evaluation run.

    • 🔥 [2026.04.17] Added support for multi-turn conversation performance stress testing, enabling load testing of dialogue-based model services with multi-turn context. Refer to the usage documentation.

    • 🔥 [2026.04.10] Added support for TIR-Bench (Thinking-with-Images Reasoning Benchmark), a multimodal benchmark evaluating agentic visual reasoning capabilities of vision-language models.

    • 🔥 [2026.03.24] Added support for Agent Skill. Any agent model that supports Skill/Tool calling can use natural language to drive EvalScope for model evaluation, performance benchmarking, and result visualization.

    • 🔥 [2026.03.09] Added support for evaluation progress tracking and HTML format visualization report generation.

    • 🔥 [2026.03.02] Added support for Anthropic Claude API evaluation. Use --eval-type anthropic_api to evaluate models via Anthropic API service.

    • 🔥 [2026.02.03] Comprehensive update to dataset documentation, adding data statistics, data samples, usage instructions and more.

    • 🔥 [2026.01.13] Added support for Embedding and Rerank model service stress testing.

    • 🔥 [2025.12.26] Added support for Terminal-Bench-2.0, which evaluates AI Agent performance on 89 real-world multi-step terminal tasks.

    • 🔥 [2025.12.18] Added support for SLA auto-tuning model API services.

    • 🔥 [2025.12.16] Added support for audio evaluation benchmarks such as Fleurs, LibriSpeech; added support for multilingual code evaluation benchmarks such as MultiplE, MBPP.

    • 🔥 [2025.12.02] Added support for custom multimodal VQA evaluation; added support for visualizing model service stress testing in ClearML.

    • 🔥 [2025.11.26] Added support for OpenAI-MRCR, GSM8K-V, MGSM, MicroVQA, IFBench, SciCode benchmarks.

    • 🔥 [2025.11.18] Added support for custom Function-Call (tool invocation) datasets to test whether models can timely and correctly call tools.

    • 🔥 [2025.11.14] Added support for SWE-bench_Verified, SWE-bench_Lite, SWE-bench_Verified_mini code evaluation benchmarks.

    • 🔥 [2025.11.12] Added pass@k, vote@k, pass^k and other metric aggregation methods; added support for multimodal evaluation benchmarks such as A_OKVQA, CMMU, ScienceQA, V*Bench.

    • 🔥 [2025.11.07] Added support for τ²-bench, an extended and enhanced version of τ-bench that includes a series of code fixes and adds telecom domain troubleshooting scenarios.

    • 🔥 [2025.10.30] Added support for BFCL-v4, enabling evaluation of agent capabilities including web search and long-term memory.

    • 🔥 [2025.10.27] Added support for LogiQA, HaluEval, MathQA, MRI-QA, PIQA, QASC, CommonsenseQA and other evaluation benchmarks. Thanks to @penguinwang96825 for the code implementation.

    • 🔥 [2025.10.26] Added support for Conll-2003, CrossNER, Copious, GeniaNER, HarveyNER, MIT-Movie-Trivia, MIT-Restaurant, OntoNotes5, WNUT2017 and other Named Entity Recognition evaluation benchmarks. Thanks to @penguinwang96825 for the code implementation.

    • 🔥 [2025.10.21] Optimized sandbox environment usage in code evaluation, supporting both local and remote operation modes.

    • 🔥 [2025.10.20] Added support for evaluation benchmarks including PolyMath, SimpleVQA, MathVerse, MathVision, AA-LCR; optimized evalscope perf performance to align with vLLM Bench.

    • 🔥 [2025.10.14] Added support for OCRBench, OCRBench-v2, DocVQA, InfoVQA, ChartQA, and BLINK multimodal image-text evaluation benchmarks.

    • 🔥 [2025.09.22] Code evaluation benchmarks (HumanEval, LiveCodeBench) now support running in a sandbox environment.

    • 🔥 [2025.09.19] Added support for multimodal image-text evaluation benchmarks including RealWorldQA, AI2D, MMStar, MMBench, and OmniBench, as well as pure text evaluation benchmarks such as Multi-IF, HealthBench, and AMC.

    • 🔥 [2025.09.05] Added support for vision-language multimodal model evaluation tasks, such as MathVista and MMMU.

    • 🔥 [2025.09.04] Added support for image editing task evaluation, including the GEdit-Bench benchmark.

    • 🔥 [2025.08.22] Version 1.0 Refactoring. Break changes, please refer to.

    • 🔥 [2025.07.18] The model stress testing now supports randomly generating image-text data for multimodal model evaluation.

    • 🔥 [2025.07.16] Support for τ-bench has been added.

    • 🔥 [2025.07.14] Support for "Humanity's Last Exam" (Humanity's-Last-Exam).

    • 🔥 [2025.07.03] Refactored Arena Mode.

    • 🔥 [2025.06.28] Optimized custom dataset evaluation; enhanced LLM judge usage.

    • 🔥 [2025.06.19] Added support for the BFCL-v3 benchmark.

    • 🔥 [2025.06.02] Added support for the Needle-in-a-Haystack test.

    • 🔥 [2025.05.29] Added support for two long document evaluation benchmarks: DocMath and FRAMES.

    • 🔥 [2025.05.16] Model service performance stress testing now supports setting various levels of concurrency.

    • 🔥 [2025.05.13] Added support for the ToolBench-Static dataset, DROP and Winogrande benchmarks.

    • 🔥 [2025.04.29] Added Qwen3 Evaluation Best Practices.

    • 🔥 [2025.04.27] Support for text-to-image evaluation.

    • 🔥 [2025.04.10] Model service stress testing tool now supports the /v1/completions endpoint.

    • 🔥 [2025.04.08] Support for evaluating embedding model services compatible with the OpenAI API has been added.

    • 🔥 [2025.03.27] Added support for AlpacaEval and ArenaHard evaluation benchmarks.

    • 🔥 [2025.03.20] The model inference service stress testing now supports generating prompts of specified length using random values.

    • 🔥 [2025.03.13] Added support for the LiveCodeBench code evaluation benchmark.

    • 🔥 [2025.03.11] Added support for the SimpleQA and Chinese SimpleQA evaluation benchmarks.

    • 🔥 [2025.03.07] Added support for the QwQ-32B model evaluation.

    • 🔥 [2025.03.04] Added support for the SuperGPQA dataset.

    • 🔥 [2025.03.03] Added support for evaluating the IQ and EQ of models.

    • 🔥 [2025.02.27] Added support for evaluating the reasoning efficiency of models.

    • 🔥 [2025.02.25] Added support for MuSR and ProcessBench benchmarks.

    • 🔥 [2025.02.18] Supports the AIME25 dataset.

    • 🔥 [2025.02.13] Added support for evaluating DeepSeek distilled models.

    • 🔥 [2025.01.20] Support for visualizing evaluation results.

    • 🔥 [2025.01.07] Native backend: Support for model API evaluation.

    • 🔥🔥 [2024.12.31] Support for adding benchmark evaluations.

    • 🔥 [2024.12.13] Model evaluation optimization.

    • 🔥 [2024.11.26] The model inference service performance evaluator has been completely refactored.

    • 🔥 [2024.10.31] The best practice for evaluating Multimodal-RAG has been updated.

    • 🔥 [2024.10.23] Supports multimodal RAG evaluation.

    • 🔥 [2024.10.8] Support for RAG evaluation.

    • 🔥 [2024.09.18] Documentation added blog module.

    • 🔥 [2024.09.12] Support for LongWriter evaluation.

    • 🔥 [2024.08.30] Support for custom dataset evaluations.

    • 🔥 [2024.08.20] Updated the official documentation.

    • 🔥 [2024.08.09] Simplified the installation process.

    • 🔥 [2024.07.31] Important change: The package name llmuses has been changed to evalscope.

    • 🔥 [2024.07.26] Support for VLMEvalKit as a third-party evaluation framework.

    • 🔥 [2024.06.29] Support for OpenCompass as a third-party evaluation framework.

    • 🔥 [2024.06.13] EvalScope integrates with SWIFT; Integrated the Agent evaluation dataset ToolBench.

    🚀 Quick Start

    Installation

    pip install evalscope

    For detailed installation instructions (source install, extra dependencies, etc.), please refer to the 📖 Installation Guide.

    Method 1. Evaluate an Online Model API (Recommended for beginners, no GPU required)

    Supports any OpenAI API-compatible model service. Just set $OPENAI_API_BASE_URL and $OPENAI_API_KEY and you are ready to go:

    evalscope eval \
 --model your-model-name \
 --api-url $OPENAI_API_BASE_URL \
 --api-key $OPENAI_API_KEY \
 --eval-type openai_api \
 --datasets gsm8k arc \
 --limit 5

    Method 2. Evaluate a Local Model

    Evaluate a local model (auto-downloaded from ModelScope):

    evalscope eval \
 --model Qwen/Qwen2.5-0.5B-Instruct \
 --datasets gsm8k arc \
 --limit 5

    Method 3. Using Python Code

    from evalscope import run_task, TaskConfig

task_cfg = TaskConfig(
    model='your-model-name',
    api_url='https://your-openai-compatible-endpoint/v1',
    api_key='your_api_key',
    eval_type='openai_api',
    datasets=['gsm8k', 'arc'],
    limit=5
)

run_task(task_cfg)

    Using Python Dictionary

    from evalscope.run import run_task

task_cfg = {
    'model': 'Qwen/Qwen2.5-0.5B-Instruct',
    'datasets': ['gsm8k', 'arc'],
    'limit': 5
}
run_task(task_cfg=task_cfg)

    Using YAML File (config.yaml)

    model: Qwen/Qwen2.5-0.5B-Instruct
datasets:
  - gsm8k
  - arc
limit: 5

    from evalscope.run import run_task

run_task(task_cfg="config.yaml")

    Output Results

    After evaluation completion, you will see a report in the terminal in the following format:

    ┌───────────────────────┬───────────┬────────────┬───────────────┬───────┬─────────┐
│ Model                 │ Dataset   │ Metric     │ Subset        │   Num │ Score   │
├───────────────────────┼───────────┼────────────┼───────────────┼───────┼─────────┤
│ Qwen2.5-0.5B-Instruct │ gsm8k     │ Accuracy ↑ │ main          │     5 │ 40%     │
├───────────────────────┼───────────┼────────────┼───────────────┼───────┼─────────┤
│ Qwen2.5-0.5B-Instruct │ arc       │ Accuracy ↑ │ ARC-Easy      │     5 │ 80%     │
├───────────────────────┼───────────┼────────────┼───────────────┼───────┼─────────┤
│ Qwen2.5-0.5B-Instruct │ arc       │ Accuracy ↑ │ ARC-Challenge │     5 │ 40%     │
└───────────────────────┴───────────┴────────────┴───────────────┴───────┴─────────┘

    Launch the visualization dashboard:

    pip install 'evalscope[service]'
evalscope service

    Visit http://127.0.0.1:9000 to open the visualization interface.

    📈 Advanced Usage

    Custom Evaluation Parameters

    You can fine-tune model loading, inference, and dataset configuration through command line parameters.

    evalscope eval \
 --model Qwen/Qwen3-0.6B \
 --model-args '{"revision": "master", "precision": "torch.float16", "device_map": "auto"}' \
 --generation-config '{"do_sample":true,"temperature":0.6,"max_tokens":512}' \
 --dataset-args '{"gsm8k": {"few_shot_num": 0, "few_shot_random": false}}' \
 --datasets gsm8k \
 --limit 10
    • --model-args: Model loading parameters such as revision, precision, etc.
    • --generation-config: Model generation parameters such as temperature, max_tokens, etc.
    • --dataset-args: Dataset configuration parameters such as few_shot_num, etc.

    For details, please refer to 📖 Complete Parameter Guide.

    ⚔️ Arena Mode

    Arena mode evaluates model performance through pairwise battles between models, providing win rates and rankings, perfect for horizontal comparison of multiple models.

    # Example evaluation results
Model           WinRate (%)  CI (%)
------------  -------------  ---------------
qwen2.5-72b            69.3  (-13.3 / +12.2)
qwen2.5-7b             50    (+0.0 / +0.0)
qwen2.5-0.5b            4.7  (-2.5 / +4.4)

    For details, please refer to 📖 Arena Mode Usage Guide.

    🖊️ Custom Dataset Evaluation

    EvalScope allows you to easily add and evaluate your own datasets. For details, please refer to 📖 Custom Dataset Evaluation Guide.

    ⚡ Inference Performance Evaluation Tool

    EvalScope provides a powerful stress testing tool for evaluating the performance of large language model services.

    • Key Metrics: Supports throughput (Tokens/s), first token latency (TTFT), token generation latency (TPOT), etc.
    • Result Recording: Supports recording results to wandb and swanlab.
    • Speed Benchmarks: Can generate speed benchmark results similar to official reports.

    For details, please refer to 📖 Performance Testing Usage Guide.

    🧪 Other Evaluation Backends

    EvalScope supports launching evaluation tasks through third-party evaluation frameworks (we call them "backends") to meet diverse evaluation needs.

    • Native: EvalScope's default evaluation framework with comprehensive functionality.
    • OpenCompass: Focuses on text-only evaluation. 📖 Usage Guide
    • VLMEvalKit: Focuses on multi-modal evaluation. 📖 Usage Guide
    • RAGEval: Focuses on RAG evaluation, supporting Embedding and Reranker models. 📖 Usage Guide
    • Third-party Evaluation Tools: Supports evaluation tasks like ToolBench.

    1. Input Layer

      • Model Sources: API models (OpenAI API), Local models (ModelScope)
      • Datasets: Standard evaluation benchmarks (MMLU/GSM8k etc.), Custom data (MCQ/QA)

    2. Core Functions

      • Multi-backend Evaluation: Native backend, OpenCompass, MTEB, VLMEvalKit, RAGAS
      • Performance Monitoring: Supports multiple model service APIs and data formats, tracking TTFT/TPOT and other metrics
      • Tool Extensions: Integrates Tool-Bench, Needle-in-a-Haystack, etc.

    3. Output Layer

      • Structured Reports: Supports JSON, Table, Logs
      • Visualization Platform: Supports Web Dashboard, Wandb, SwanLab

    ❤️ Community & Support

    Welcome to join our community to communicate with other developers and get help.

    Discord GroupWeChat GroupDingTalk Group

    👷‍♂️ Contributing

    We welcome any contributions from the community! If you want to add new evaluation benchmarks, models, or features, please refer to our Contributing Guide.

    Thanks to all developers who have contributed to EvalScope!

    📚 Citation

    If you use EvalScope in your research, please cite our work:

    @misc{evalscope_2024,
    title={{EvalScope}: Evaluation Framework for Large Models},
    author={ModelScope Team},
    year={2024},
    url={https://github.com/modelscope/evalscope}
}

    相关项目

    同分类里 star 最高的项目
    对比全部 →
    promptfoo
    promptfoo·TypeScript

    用声明式配置对提示词、Agent 和 RAG 做自动化评测与红队渗透测试，支持多模型横向对比，可接入命令行和 CI/CD。

    提示词评测+4模型评测
    27019+782.7k0.125.12026-11-09 21:45 UTC
    opik
    comet-ml·Python

    面向 LLM 应用、RAG 和 Agent 工作流的可观测与评测平台，提供全链路追踪、自动评估和生产级监控看板，帮团队定位问题并持续优化效果。

    LLM 可观测+4模型评测
    23298+461.9k2.2.1432026-11-10 09:38 UTC
    deepeval
    confident-ai·Python

    一个 Python 的 LLM 评测框架，提供开箱即用的评估指标，帮你在开发和上线前系统检验大模型输出质量。

    LLM评测+3模型评测
    19405+482.2ksetup-v0.1.02026-11-10 11:12 UTC
    ragas
    vibrantlabsai·Python

    面向 LLM 应用的自动化评测工具，用合成测试集和多种指标量化 RAG 与生成效果，帮你在上线前发现质量问题。

    LLM评测+4模型评测
    16280+231.8kv0.4.32026-02-24 07:47 UTC