opencompass

open-compassApache-2.0活跃维护

OpenCompass 是一个大模型评测平台，支持 OpenAI、Qwen、DeepSeek 等主流模型，覆盖知识、推理、代码、长上下文等 100+ 数据集，帮你系统化地跑分对比。

OpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety.

0.5.5发布于 2026-11-06 08:08 UTC7,580star +8 / 7 天904fork同步于 2026-11-10 11:32 UTC

大模型评测基准测试评测平台长上下文模型对比

对比 GitHub 官网 benchmark evaluation large-language-model llm llm-eval

怎么上手

由 README 提炼，执行前请确认

原文

conda create --name opencompass python=3.12 -y
conda activate opencompass
pip install -U opencompass

硬件要求按 README 说明

README 未说明具体显存要求；本地开源模型需依赖 HuggingFace、LMDeploy 或 vLLM 后端，API 模型评测无需本地 GPU。

下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼

没人工点评？写下你的经验

一站式大模型评测平台，统一跑通数据集配置、推理、打分与结果汇总

OpenCompass 是一个大模型评测平台，支持 OpenAI、Qwen、DeepSeek 等主流模型，覆盖知识、推理、代码、长上下文等 100+ 数据集，帮你系统化地跑分对比。

核心能力

集成 100+ 数据集，覆盖知识、推理、代码、科学、长文本与安全
支持 OpenAI、Anthropic、Gemini、Qwen、GLM、DeepSeek 等模型
对接 HuggingFace、LMDeploy、vLLM 多种推理后端
支持 zero-shot、few-shot、思维链、规则与 LLM-as-judge 评测
支持任务切分、并发与分布式执行，适配大规模评测

容易踩的坑

推理后端依赖常冲突，建议为 lmdeploy、vllm 单独建虚拟环境；代码执行类数据集依赖 pyext，需用 Python 3.10，否则 APPS、TACO、LiveCodeBench 代码生成不可用；完整数据集支持需装 opencompass[full]，API 评测需装 opencompass[api]。

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

最近发版

5 个版本0.5.5

全部 release

0.5.5最新正式版2026-11-06 · 2026-11-06 08:08 UTC
OpenCompass v0.5.5 Release Notes

🌟 Highlights

✨ Integrated More Benchmarks: Integrated new benchmarks spanning physics, reasoning, long-context, alignment, hallucination, maritime, biomedical, translation and multimodal evaluation, including CritPt, General365, AA-OmniScience v1.0 and v1.1, AA-LCR v1.1, HLE-Diamond, arena-hard-v2, ReasonZoo, LG-Bench, CHHallu-Src, GrandPhysics, PromptCBLUE Life Science, MaritimeBench, WMT19, LogiQA and ethics, plus ChartQAPro, MicroVQA, SFE and SciDocBench from VLMEvalKit (#2610, #2626, #2658, #2713, #2362, #2256, #2690, #2525, #2118, #2073, #2018, #1503, #516, #495, #2665).

✨ Documentation Refactor: Restructured the Chinese and English documentation into a new layout (get started, user guides, evaluation, execution, extension, tools and FAQ), rewrote the README, installation and quick start guides, and added configs for the latest models (#2666).

✨ Refactored CI: Refactored the mock API tests, fixed the PR stage CI, and extended the PR stage CI to cover Python 3.9 and 3.12 (#2698, #2670, #2715).

✨ Dropped Python 3.8 and Fixed Python 3.9 Compatibility: Removed Python 3.8 support and fixed the PEP 604 annotations that broke imports on Python 3.9 (#2715, #2674).

🚀 New Features

🔧 Added new benchmark configurations: CritPt, General365, AA-OmniScience, AA-LCR v1.1, HLE-Diamond, arena-hard-v2, ReasonZoo, LG-Bench, CHHallu-Src v1, GrandPhysics, PromptCBLUE Life Science, MaritimeBench, WMT19, LogiQA and ethics, with AA-LCR, AA-OmniScience (v1.1) and HLE-Diamond configs and the prompt template updated (#2610, #2626, #2658, #2713, #2362, #2256, #2690, #2525, #2118, #2073, #2018, #1503, #516, #495).

🔧 Added ChartQAPro, MicroVQA, SFE and SciDocBench datasets through VLMEvalKit (#2665).

🔧 Added retry and verbose options to the TurboMind API for get_ppl and get_ll (#2650, #2652).

🐛 Bug Fixes

🔧 Fixed reasoning parsing in the OpenAI, OpenAISDK and OpenAISDKStreaming models, and surfaced inference failures in OpenICLInferConcurrentTask (#2653, #2651).

🔧 Fixed dataset loading and downloads for AA-LCR, MBPP-CN, C-Eval (ModelScope and HF generation for causal LMs) and SciReasoner 1.5 (with a new llm_extract config), and fixed MATH JudgeLLM config imports (#2630, #2695, #2416, #2640, #2662).

🔧 Fixed VLMEvalKit empty predictions and JSONL resume with Unicode separators (#2691).

⚙ Enhancements and Refactors

⚙ Refactored the OpenCompass documentation (#2666).

⚙ Refactored the mock API tests and fixed the PR stage CI (#2698, #2670).

⚙ Dropped Python 3.8 support, added Python 3.9 and 3.12 to the PR stage CI, and fixed PEP 604 annotations that broke imports on Python 3.9 (#2715, #2674).

⚙ Fixed VLM installation dependencies and requirement declarations (#2679, #2676).

🎉 Welcome New Contributors

A warm welcome and special thanks to our newest contributors who made this release possible:
- @alleyxd made their first contribution in #2684
- @DeliWang made their first contribution in #2416
- @avalyset made their first contribution in #2618
- @BeingIsA made their first contribution in #2253
- @K-zhy made their first contribution in #2018
- @Xiao-Youth made their first contribution in #2118
- @JimmyWang0417 made their first contribution in #2669
- @ktwu01 made their first contribution in #2177
- @mkzung made their first contribution in #2594
- @Owen-Qin made their first contribution in #2216
- @SashaMIT made their first contribution in #2661
- @ShikangPang made their first contribution in #2187
- @DewidPig made their first contribution in #1677
- @WatchTree-19 made their first contribution in #2523
- @YL1N made their first contribution in #2585
- @JensenDong made their first contribution in #1811
- @YueshenZ made their first contribution in #2662
- @HYZ17 made their first contribution in #2218
- @ZhangXiangyi0523 made their first contribution in #2690
- @zhangzefeng92
0.5.42026-08-26 · 2026-08-26 06:41 UTC
查看更新说明
OpenCompass v0.5.4 Release Notes

🌟 Highlights

✨ Multimodal Evaluation with VLMEvalKit: OpenCompass now integrates VLMEvalKit for native multimodal dataset loading, OpenAI-compatible inference, and official VLMEvalKit evaluation metrics (#2586).

✨ Multi-Round Evaluation: GenInferencer now supports multi-round inference, accompanied by the new Multi-IF benchmark for evaluating multi-turn instruction following (#2557).

✨ Expanded Model API Ecosystem: Added LiteLLM as an AI gateway backend and extended TurboMind API capabilities with PPL evaluation support (#2441, #2510).

✨ Broader Benchmark Coverage: Added new evaluations spanning finance, long-context understanding, alignment, information extraction, and code reasoning, including BuySideFinBench, ELBench, Inverse IFEval, AA-LCR, CRUXEval, and updated LongBench/MRCR V2 configurations (#2446, #2495, #2514, #2558).

🚀 New Features

🔧 Integrated VLMEvalKit datasets and evaluators into the OpenCompass workflow, with MMBench and MMMU-Pro evaluation examples (#2586).

🔧 Added multi-round inference to GenInferencer and introduced Multi-IF dataset support (#2557).

🔧 Added LiteLLM AI Gateway model support and the TurboMind API /get_ppl endpoint (#2441, #2510).

🔧 Added and expanded benchmark configurations for CRUXEval, LongBench v1, Qasper, MRCR v2, Fake-Alignment, and more (#2572, #2592, #2549, #2553).

🐛 Bug Fixes

🔧 Strengthened evaluation correctness by rejecting deceptive return values in code evaluators and preventing invalid SIQA generations from being scored as correct (#2565, #2450).

🔧 Fixed generation behavior across HuggingFace and vLLM, including max_out_len=None, default torch dtype loading, accelerator generation config conversion, and chat-template BOS handling (#2459, #1969, #2519, #2554).

🔧 Improved evaluator reliability for BBH free-form answers, LiveCodeBench memory limits, Wikitext PPL without references, and official Fake-Alignment prompts (#2548, #2538, #2532, #2567).

🔧 Fixed dataset loading, local path resolution, download mappings, and prompts across AGIEval, BoolQ, GSM8K, C-Eval, SVAMP, TACO, and other benchmark configurations (#2497, #2498, #2540, #2546).

⚙ Enhancements and Refactors

⚙ Improved multi-round observability by fixing per-turn statistics in dump-res-length and repeat detection for multi-round conversations (#2569).

⚙ Improved model API robustness for Qwen response extraction, DeepSeek retry timeouts, Claude custom headers, and TurboMind proxy compatibility (#2521, #2530, #2445, #2529).

⚙ Improved data and task handling with safer SizePartitioner cost calculation and corrected TopK retriever metadata collation (#2438, #2556).

⚙ Refactored CI workflows, pinned third-party CI actions, and removed misleading error logs from successful RJob submissions (#2570, #2533, #2588).

📦 Compatibility and Packaging

📦 Made OpenCC optional to restore Windows installation compatibility (#2512).

📦 Pinned the mmengine-lite runtime dependency for more reproducible installations (#2531).

📦 Resolved the Dingo/Pillow dependency conflict and fixed Dingo evaluation behavior (#2547, #2607).

🎉 Welcome New Contributors

A warm welcome and special thanks to our newest contributors who made this release possible:
- @Amber0614 made their first contribution in (#2495).
- @ziyangwang007 made their first contribution in (#2459).
- @cindy90 made their first contribution in (#2446).
- @irexyc made their first contribution in (#2510).
- @Hibbert133 made their first contribution in (#2426).
- @RheagalFire made their first contribution in (#2441).
- @octo-patch made their first contribution in (#2438).
- @HXC-Cloud-lab made their first contribution in (#2549).
- @kobihikri made their first contribution in (#2533).
- @thecaptain789 made their first contribution in (#2397).
- @SongYippee made their first contri
0.5.32026-06-29 · 2026-06-29 03:17 UTC
查看更新说明
OpenCompass v0.5.3 Release Notes

🌟 Highlights

✨ 🧪 Extensive New Benchmarks and Dataset Support: Added support for a wide range of new benchmarks and datasets, including AIME2026, HMMT Feb 2026, SimpleQA-Verified, AdvancedIF, HLE-Verified, MRCR-V2, Molecular-IQ, SciReasoner 1.5, MP20, PerspectiveGap, ZebraLogic, ArxivRollBench, CL-Bench, and more.

✨ 🧩 RawPromptTemplate Support: Introduced RawPromptTemplate and expanded its support across dataset configs, OpenAISDKStreaming, ChatML datasets, retrievers, and multiple benchmark configurations.

✨ 🤖 New Model & API Support: Added support for the OpenAI Responses API, Gemini SDK API and Claude SDK thinking content.

✨ 🛠️ Infrastructure & Evaluation Enhancements: Improved concurrent inference, repeat analysis, summarizer configuration, judge post-processing, CI coverage, and evaluation robustness.

🚀 New Features

🔧 Added support for new datasets and benchmarks, including AIME2026 and HMMT Feb 2026 (#2404), Molecular-IQ (#2431), SimpleQA-Verified (#2436), AdvancedIF (#2461), HLE-Verified (#2454), MRCR-V2 (#2467), S2-ToM-G Bench (#2476), SciReasoner 1.5 (#2477, #2479), MP20 (#2482), PerspectiveGap (#2484), ZebraLogic (#2464), and ArxivRollBench (#2458).

🔧 Introduced RawPromptTemplate and new dataset configs (#2407).

🔧 Added RawPromptTemplate support for OpenAISDKStreaming and ChatMLDatasets (#2414).

🔧 Added SciReasoner config and support for retrievers using RawPromptTemplate (#2422).

🔧 Added repeat configs for HMMT2025 and UGD_hard (#2425).

🔧 Added new subjective dataset configs and fixed AdvancedIF related configs (#2466).

🔧 Added RawPromptTemplate configs for AlignBench v1.1 (#2472), RULER, NeedleBench, LongBench v2, and BABILong (#2474).

🔧 Added support for repeat analysis (#2455).

🔧 Added OpenAI Responses API model support (#2481).

🔧 Added Gemini SDK API model support (#2494).

🔧 Added support for Claude SDK thinking content (#2487).

🔧 Added CL Bench support (#2483).

🐛 Bug Fixes

🔧 Fixed IFEval path issue (#2406).

🔧 Fixed tag matching in generic_llmjudge_postprocess (#2417).

🔧 Fixed issues in generic.py (#2419).

🔧 Fixed ChatML and dataset utility issues (#2421).

🔧 Fixed OlympiadBench compatibility with RawPromptTemplate (#2423).

🔧 Fixed ellipsis extraction in bio_data (#2429).

🔧 Fixed Molecular-IQ PR test issues (#2432).

🔧 Added multi-round support from api_meta_template (#2435).

🔧 Fixed result metrics in SimpleQA (#2439).

🔧 Fixed no-prediction cases in SciReasoner biology instruction evaluation (#2456).

🔧 Fixed hub version issue (#2457).

🔧 Fixed SimpleQA prompt (#2463).

🔧 Added unsafe pattern handling for MathEvaluator (#2468).

🔧 Fixed MTBench101, WildBench, and AdvancedIF issues (#2470).

🔧 Added subprocess timeout to MATHVerifyEvaluator to prevent parse and verify hangs (#2478).

🔧 Resolved dependency issues for Python 3.12 (#2480).

🔧 Restored LongBench v2 answer prompt punctuation (#2452).

🔧 Fixed MP20 metrics (#2488).

🔧 Prioritized Responses reasoning content (#2490).

⚙ Enhancements and Refactors

⚙ Evaluation and Runtime Improvements:
- Supported concurrent inference across tasks (#2403).
- Reduced RJOB sleep time (#2411).
- Added new template for CompassAcademic (#2447).
- Expanded config options of the summarizer group (#2448).
- Added --no-progress parameter for repeat analysis to suppress detailed logs (#2473).
- Added prediction postprocessor support for judge models in GenericLLMEvaluator (#2491).
⚙ RawPromptTemplate Documentation and Compatibility:
- Added guide for RawPromptTemplate (#2420).
- Improved RawPromptTemplate coverage across benchmark and datas
0.5.22026-02-14 · 2026-02-14 03:46 UTC
查看更新说明
OpenCompass v0.5.2 Release Notes

🌟 Highlights

✨ 🧪 Extensive New Benchmarks Support: We have introduced comprehensive support for Scientific and General Benchmarks, including SciReasoner, Biology Instructions, Mol Instructions, CMPhysBench, IFBench, LCB-pro, etc. ✨ 🤖 New Model & API Support: Added support for Intern-S1-Pro and TeleChat API evaluation examples. ✨ 🛠️ Infrastructure & Enhancements: Fixed bugs, improved evaluation pipelines and updated CI.

🚀 New Features

🔧 Introduced support for HMMT2025 (#2305), AMO-Bench (#2305), IMO-Bench (#2305), ATLAS (#2297), OpenSWI (#2312), CMPhysBench (#2313), Biology Instructions (#2314), Mol Instructions (#2326), ARC_AGI_2 (#2330), IFBench (#2354), SciReasoner (#2360), PI-LLM (#2283), ProcessBench (#2274), and LCB_pro (#2361). 🔧 Supported monitoring of multi-dimensional evaluation metrics, including output length, logprobs, and finish reasons (#2351). 🔧 Added support for Intern-S1-Pro evaluation examples (#2394). 🔧 Added support for TeleChat API inference (#2371). 🔧 Added LLM-judge-based config for C-Eval (#2398).

🐛 Bug Fixes

🔧 Fixed OpenAISDKStreaming regarding output completeness and related issues (#2367, #2389, #2399). 🔧 Removed Pyext in runtime requirement (#2306). 🔧 Fixed pattern match in Smolinstruct (#2384). 🔧 Fixed buffer-related error in the LiveCodeBench evaluation (#2393).

⚙ Enhancements and Refactors

⚙ Infrastructure Refactors:
- Updated LCBench (#2166).
- Added headers as input param in BigCodeBench (#2302).
- Updated rjob with metadata name (#2316).
- Parametrized timeout in OpenAISDK (#2352).
- Added meta logger in OpenICLInferTask (#2383).
⚙ CI/CD Improvements:
- Refactored dailytest (#2308).
- Added CI for new datasets (#2358, #2369).
- Changed github runner (#2373).
- Added uni-test (#2390).
🎉 Welcome New Contributors

A warm welcome and special thanks to our newest contributors who made this release possible:
- @zhuangziGiantfish made their first contribution in (#2283).
- @xgao922 made their first contribution in (#2307).
- @Jensen246 made their first contribution in (#2310).
- @ccx06 made their first contribution in (#2371).
Full Changelog: https://github.com/open-compass/opencompass/compare/0.5.1.post1...0.5.2

Thank you for using OpenCompass! These updates empower deeper insights and more reliable evaluations. Keep exploring, and stay tuned for future innovations! 🌟
0.5.1.post12025-10-17 · 2025-10-17 10:17 UTC
查看更新说明
OpenCompass v0.5.1 Release Notes

🌟 Highlights

✨ A New Method to quickly Integrate and Evaluate Your Datasets: Added a fast dataset integration and evaluation method based on ChatML format, simplifying the previously complex dataset integration process. ✨ New Datasets: Integrated new benchmarks including SeedBench and BeyondAIME. ✨ Infrastructure & Enhancements: Fixed several bugs and updated CI.

🚀 New Features

🔧 Introduced a new approach for dataset integration and evaluation based on ChatML Template with evaluation examples (#2277).
🔧 Added SeedBench dataset (#2020).
🔧 Added BeyondAIME dataset (#2192).

🐛 Bug Fixes

🔧 Fixed Module Registers(#2262, #2266) 🔧 Fixed duplicate engine config update in TurboMindModelwithChatTemplate (#2276) 🔧 Fixed torchrun to avoid unexpected PATH environment (#2269) 🔧 Fixed the None return value case in cascade evaluator (#2211)

⚙ Enhancements and Refactors

⚙ Infrastructure Refactors:
- Update rjob.py and subjective_eval.py (#2263)
⚙ CI/CD Improvements:
- Updated testcase (#2257)
- Updated pr_test (#2281)
- Fixed pr_test installation (#2290)
🎉 Welcome New Contributors

A warm welcome and special thanks to our newest contributors who made this release possible:
- @ChenZiHong-Gavin for their first contribution in (#2020)
- @jibuzixin for their first contribution in (#2211)
- @JYLiAo32 for their first contribution in (#2276)
- @ShoupingShan for their first contribution in (#2262)
Full Changelog: https://github.com/open-compass/opencompass/compare/0.5.0...0.5.1.post1

Thank you for using OpenCompass! These updates empower deeper insights and more reliable evaluations. Keep exploring, and stay tuned for future innovations! 🌟

用户点评

登录用户的真实使用记录，可按版本和硬件筛选

共 0 条

还没有人点评。第一个说出真实体验的人最有价值。

README 原文

未翻译

在 GitHub 打开

English | 简体中文

[!IMPORTANT]

Star Us, You will receive all release notifications from GitHub without any delay ~ ⭐️

🧭 Welcome

to OpenCompass!

Just like a compass guides us on our journey, OpenCompass will guide you through the complex landscape of evaluating large language models. With its powerful algorithms and intuitive interface, OpenCompass makes it easy to assess the quality and effectiveness of your NLP models.

🚩🚩🚩 Explore opportunities at OpenCompass! We're currently hiring full-time researchers/engineers and interns. If you're passionate about LLM and OpenCompass, don't hesitate to reach out to us via email. We'd love to hear from you!

🔥🔥🔥 We are delighted to announce that the OpenCompass has been recommended by the Meta AI, click Get Started of Llama for more information.

✨ Introduction

OpenCompass is a one-stop platform for large model evaluation. It supports a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, and more, and integrates over 100 datasets covering knowledge, reasoning, coding, science, language, long context, safety, and other capability dimensions. OpenCompass also integrates VLMEvalKit, enabling text-only and vision-language datasets to be evaluated within a unified workflow.

OpenCompass provides a complete workflow spanning dataset and model configuration, inference, evaluation, and result summarization. It supports local open-source models and API models, as well as multiple inference backends such as HuggingFace, LMDeploy, and vLLM. The platform supports zero-shot, few-shot, chain-of-thought, rule-based, and LLM-as-judge evaluation, while task partitioning, concurrent execution, and distributed execution accommodate evaluation workloads of different scales.

OpenCompass is designed to be open, reproducible, and extensible. Users can flexibly integrate new models, datasets, evaluators, inference backends, and task scheduling systems. The accompanying CompassHub provides benchmark navigation, while CompassRank presents model evaluation leaderboards.

🚀 What's New

[2026.09.30] We release AgentCompass, an extensible open-source evaluation infrastructure for LLM/VLM agents. It decouples models, benchmarks, harnesses, and environments, and supports 30+ public benchmarks and 10+ agent harnesses such as Claude Code, Codex, and OpenHands. AgentCompass has been accepted to the EMNLP 2026 Demo Track. Check out the technical report and documentation for details! 🔥🔥🔥
[2026.09.21] OpenCompass has updated the recommended API configurations for flagship models from leading providers and comprehensively restructured its Chinese and English documentation, further improving the model integration and evaluation experience. See the model configurations and OpenCompass documentation for details! 🔥🔥🔥
[2026.08.25] OpenCompass now integrates with VLMEvalKit, enabling native multimodal dataset loading, inference through OpenAI-compatible APIs, and evaluation with official VLMEvalKit metrics. Check out the MMBench example and MMMU-Pro example for details! 🔥🔥🔥
[2026.07.28] OpenCompass has expanded its API model ecosystem with support for the OpenAI Responses API and LiteLLM AI Gateway, while updating the Gemini and Anthropic integrations to their latest SDK interfaces. Check out the OpenAI Responses API implementation, LiteLLM AI Gateway implementation, Gemini SDK implementation, and Anthropic SDK implementation for details!
[2026.07.27] OpenCompass now supports multi-round inference in GenInferencer and adds support for the Multi-IF dataset to evaluate multi-turn instruction-following capabilities. Check out the Multi-IF evaluation configuration for details! 🔥🔥🔥
[2026.05.25] OpenCompass now provides repeat analysis tools for detecting repetitive content and looping model outputs in current evaluation tasks or existing evaluation results. Check out the repeat analysis tool for details!
[2026.03.20] OpenCompass now supports concurrent inference across tasks together with evaluation watching, enabling completed inference tasks to be monitored and subsequent evaluations to be triggered in a coordinated pipeline. Parallel inferencers, task monitoring, and heartbeat mechanisms further improve large-scale evaluation efficiency. Check out the concurrent inference implementation and evaluation watcher implementation for details!
[2026.03.17] OpenCompass introduces RawPromptTemplate, allowing original benchmark prompts and structured conversations to be passed to models without unintended formatting transformations. It supports API models, ChatML datasets, and appending additional prompt content on the model side. Check out the RawPromptTemplate guide for details!
[2026.02.05] OpenCompass now supports Intern-S1-Pro related general and scientific evaluation benchmarks. Please check Example for Evaluating Intern-S1-Pro and Model Card for more details! 🔥🔥🔥
[2025.12.08] OpenCompass now supports evaluation for SciReasoner. Please check Example for Evaluating SciReasoner and Project GitHub Repo for more details! 🔥🔥🔥
[2025.07.26] OpenCompass now supports Intern-S1 related general and scientific evaluation benchmarks. Please refer to the Intern-S1 model configuration for details! 🔥🔥🔥
[2025.04.01] OpenCompass now supports CascadeEvaluator, allowing multiple evaluators to work in sequence and enabling custom evaluation pipelines for more complex scenarios. Check out the documentation for details! 🔥🔥🔥
[2025.03.11] OpenCompass now supports SuperGPQA, covering knowledge evaluation across 285 graduate-level disciplines. Give it a try! 🔥🔥🔥
[2025.02.28] OpenCompass now supports the DeepSeek-R1 model series. Check out the DeepSeek-R1 model configuration for more details! 🔥🔥🔥
[2025.02.15] We have added two practical evaluation tools: GenericLLMEvaluator for LLM-as-judge evaluation and MATHVerifyEvaluator for mathematical reasoning evaluation. Check out the LLM Judge and Mathematical Evaluation documentation for more details! 🔥🔥🔥

🛠️ Installation

Below are the steps for quick installation and dataset preparation.

💻 Environment Setup

We highly recommend using conda to manage your Python environment. OpenCompass supports Python 3.12 for regular and full installations. If your evaluation depends on code execution datasets backed by pyext, use Python 3.10 instead: pyext==0.7 is skipped on Python >=3.11 because it relies on inspect.getargspec, which was removed in Python 3.11. Without pyext, APPS (apps, apps_mini), TACO, and LiveCodeBench Code Generation are unavailable.

Create your virtual environment

conda create --name opencompass python=3.12 -y
conda activate opencompass

Install OpenCompass via pip

# Supports most datasets and models
pip install -U opencompass

# Full installation (supports more datasets)
# pip install "opencompass[full]"

# Model inference backends. Since these backends often have conflicting dependencies,
# we recommend managing them in separate virtual environments.
# pip install "opencompass[lmdeploy]"
# pip install "opencompass[vllm]"

# API evaluation (for example, OpenAI and Qwen)
# pip install "opencompass[api]"

# Multimodal evaluation
# pip install "opencompass[vlm]"

Install OpenCompass from source

To use the latest OpenCompass features, you can also build it from source:

git clone https://github.com/open-compass/opencompass opencompass
cd opencompass
pip install -e .
# pip install -e ".[full]"
# pip install -e ".[vllm]"

After installation, read the Quick Start to learn how to run an evaluation task.

For more tutorials, see our full documentation.

📖 Dataset Support

The OpenCompass documentation provides a statistical list of all datasets supported by the platform.

You can quickly find the dataset you need by sorting, filtering, and searching the list.

See the dataset statistics section of the dataset documentation for details.

📖 Model Support

📊 Leaderboard

We will continue to provide detailed leaderboards for open-source and API models. See the OpenCompass Leaderboard. To participate in an evaluation, send the model repository URL or a standard API endpoint via email.

👷‍♂️ Contributing

We appreciate all contributions to improving OpenCompass. Please refer to the contributing guideline for the best practice.

🤝 Acknowledgements

Some code in this project is cited and modified from OpenICL.

Some datasets and prompt implementations are modified from chain-of-thought-hub and instruct-eval.

🖊️ Citation

@misc{2023opencompass,
    title={OpenCompass: A Universal Evaluation Platform for Foundation Models},
    author={OpenCompass Contributors},
    howpublished = {\url{https://github.com/open-compass/opencompass}},
    year={2023}
}

怎么上手

项目说明

核心能力

容易踩的坑

最近发版

OpenCompass v0.5.5 Release Notes

🌟 Highlights

🚀 New Features

🐛 Bug Fixes

⚙ Enhancements and Refactors

🎉 Welcome New Contributors

OpenCompass v0.5.4 Release Notes

🌟 Highlights

🚀 New Features

🐛 Bug Fixes

⚙ Enhancements and Refactors

📦 Compatibility and Packaging

🎉 Welcome New Contributors

OpenCompass v0.5.3 Release Notes

🌟 Highlights

🚀 New Features

🐛 Bug Fixes

⚙ Enhancements and Refactors

OpenCompass v0.5.2 Release Notes

🌟 Highlights

🚀 New Features

🐛 Bug Fixes

⚙ Enhancements and Refactors

🎉 Welcome New Contributors

OpenCompass v0.5.1 Release Notes

🌟 Highlights

🚀 New Features

🐛 Bug Fixes

⚙ Enhancements and Refactors

🎉 Welcome New Contributors

用户点评

README 原文

🧭 Welcome

✨ Introduction

🚀 What's New

🛠️ Installation

💻 Environment Setup

Create your virtual environment

Install OpenCompass via pip

Install OpenCompass from source

📖 Dataset Support

📖 Model Support

📊 Leaderboard

👷‍♂️ Contributing

🤝 Acknowledgements

🖊️ Citation

相关项目