登录

helm

stanford-crfmApache-2.0活跃维护

斯坦福 CRFM 出品的 Python 评测框架，对基础模型和大语言模型做全面、可复现、透明的基准测试，方便横向对比模型能力。

Holistic Evaluation of Language Models (HELM) is an open source Python framework created by the Center for Research on Foundation Models (CRFM) at Stanford for holistic, reproducible and transparent evaluation of foundation models, including large language models (LLMs) and multimodal models.

v0.5.16发布于 2026-04-30 05:05 UTC2,967star435fork同步于 2026-11-07 07:10 UTC
模型评测基准测试大语言模型可复现评估多模态评测
对比 GitHub 官网

怎么上手

由 README 提炼，执行前请确认
原文 
pip install crfm-helm
helm-run --run-entries mmlu:subject=philosophy,model=openai/gpt2 --suite my-suite --max-eval-instances 10
helm-summarize --suite my-suite
helm-server --suite my-suite
硬件要求按 README 说明

README 未说明显存或 GPU 要求；快速开始示例用 openai/gpt2 通过 API 调用，未提及本地推理的硬件条件

下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼
没人工点评？写下你的经验

斯坦福 CRFM 出品的语言模型整体评测框架，统一跑基准、算指标、看榜单

斯坦福 CRFM 出品的 Python 评测框架，对基础模型和大语言模型做全面、可复现、透明的基准测试，方便横向对比模型能力。

核心能力

  • 用统一格式组织 MMLU-Pro、GPQA、IFEval 等基准
  • 通过统一接口调用 OpenAI、Claude、Gemini 等模型
  • 除准确率外还度量效率、偏见、毒性等维度
  • 提供 Web UI 逐条查看 prompt 与模型回复
  • 提供 Web 榜单跨模型跨基准对比结果

容易踩的坑

项目已于 2026 年 6 月 1 日进入维护模式，新功能可能不再合入；helm-server 默认监听 8000 端口，需浏览器访问 http://localhost:8000/；调用 OpenAI 等厂商模型需自行准备对应 API 凭证，README 未展开说明

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

最近发版

5 个版本v0.5.16
全部 release
  1. v0.5.16最新正式版2026-04-30 · 2026-04-30 05:05 UTC

    Breaking Changes

    • meta/llama-3.1-405b-instruct-turbo will no longer be used as a LLM-as-judge in Omni-MATH, WildBench, HarmBench, XSTest, and Anthropic Red Team scenarios due to deprecation by Together AI (#4233)

    Models

    • Deprecate meta/llama-3.1-405b-instruct-turbo (#4233)

    Scenarios

    • Release Arabic Legal scenario (#4226, #4236, #4237)
    • Release Arabic Content Generation scenario (#4224, #4229, #4236, #4237)
    • Migrate Unitxt integration to use new Unitxt API (#4234)
    • Remove meta/llama-3.1-405b-instruct-turbo from annotators (#4233)
    • Fix AutobencherSafetyScenario parsing the file path string as JSON (#4206)

    Contributors

    Thank you to the following contributors for your work on this HELM release!

    • @Chessing234
    • @yifanmai
  2. v0.5.152026-04-23 · 2026-04-23 01:07 UTC
    查看更新说明

    Breaking Changes

    • The ArabicMMLU scenario will now include the context field in the inputs; previously, the context field was incorrect omitted, which made some instances unsolvable (#4218)

    Models

    • Add GPT-5.4, GPT-5.4 mini, and GPT-5.4 nano (#4145)
    • Add Llama 4 Maverick on Vertex AI Llama 4 API Service (#4150)
    • Add Qwen3.5 models on Together (#4152)
    • Lazily load model in HuggingFaceClient (#4172)
    • Add Gemini 3 Flash and Gemini 3.1 Flash-Lite (#4204)
    • Work around Pydantic validation error in OpenAIResponseClient (#4223)

    Scenarios

    • Fix _apply_output_mapping_pattern returning wrong match results (#4192)
    • Release Arabic Finance scenario (#4220)
    • Fix missing context in ArabicMMLU scenario (#4218)

    Frontend

    • Add link to per_instance_stats.json on frontend (#4166)

    Framework

    • Suppress duplicate warnings from truncate_sequence (#4151)
    • Default suite to "default" in helm-run and helm-summarize (#4155)
    • Don't import PyTorch when registering run spec functions (#4221)

    Contributors

    Thank you to the following contributors for your work on this HELM release!

    • @atulydvv
    • @Chessing234
    • @yifanmai
  3. v0.5.142026-03-27 · 2026-03-27 00:24 UTC
    查看更新说明

    Breaking Changes

    • Previously, responses from models on the OpenAI, Anthropic Claude, OpenRouter, NVIDIA NIM, vLLM, and xAI Grok APIs were tokenized locally locally. These responses will no longer be tokenized. (#4108, #4113)
    • The MadinahQA scenario will now include the context field in the inputs for the reading comprehension task (#4127)

    Models

    • Add Mistral Large 3, Mistral Medium 3.1, Mistral Small 3.2, and Ministral 3 (#4098)
    • Add GPT-5.4 (#4099)
    • Switch llama-4-scout-17b-16e-instruct to use Vertex AI instead of Together (#4104)
    • Add the ability to use models from certain providers without manually configuring model_deployments.yaml (#4100, #4103, #4117):
      • The following providers are supported:
        • OpenAI (#4107)
        • Anthropic (#4109)
        • OpenRouter (#4114)
        • Mistral (#4118)
        • Writer (#4119)
        • xAI (#4120)
        • Cohere (#4130)
        • Google (#4131)
        • Hugging Face Hub (from v0.5.12)
        • HuggingFace Inference Providers (from v0.5.13)
        • Together (from v0.5.12)
        • LiteLLM (from v0.5.12)
      • Note: Together models will now use the chat API rather than the text completions API by default (#4106)
    • Remove tokenization from AnthropicMessagesClient (#4108)
    • Remove tokenization from OpenAIClient (#4113)

    Scenarios

    • Fix MadinahQA scenario to include Context field for reading comprehension (#4127)

    Framework

    • Remove the human-evaluation optional dependency (#4101)
    • Install dependencies for documentation on Read the Docs with uv (#4140)

    Contributors

    Thank you to the following contributors for your work on this HELM release!

    • @aaabulkhair
    • @yifanmai
  4. v0.5.132026-02-25 · 2026-02-25 18:24 UTC
    查看更新说明

    Breaking Changes

    • Previously, in HuggingFaceClient, Hugging Face models were loaded with trust_remote_code=True by default. This was a security risk as it permitted executing arbitrary code from the model repositories. This will no longer be done. If needed, trust_remote_code must be explicitly set in the args field in model_deployments.yaml for Hugging Face models. (#4054)
    • Previously, responses from models on the Mistral AI API were tokenized locally locally. These responses will no longer be tokenized. (#4084)

    Models

    • Do not set trust_remote_code=True by default in HuggingFaceClient (#4054)
    • Add Grok 4 Fast and Grok 4.1 Fast (#4071)
    • Add GPT-5.2 (#4075)
    • Add Claude Sonnet 4.6 and Opus 4.6 (#4083)
    • Add Gemini 3.1 Pro (#4076)
    • Explicitly set pad_token_id for Hugging Face generations (#4087)
    • Support models on HuggingFace Inference Providers (#4086)

    Scenarios

    • Add links to the cited papers in the schema for HEIM (#4063, #4074)

    Framework

    • Add --export-path arg to helm-server for exporting a static website (#4047)
    • Use Python standard library for downloading files (#4052)
    • Use Python standard library for decompressing archives (#4051)
    • Add validation for schema YAML files (#4049)
    • Add optional dependencies for summarization metrics (#4067)

    Frontend

    • Render whitespace in adapter values in frontend using CSS pre-wrap (#4062)
    • Render whitespace in model responses in proxy server frontend using CSS pre-wrap (#4085)

    Contributors

    Thank you to the following contributors for your work on this HELM release!

    • @HetanshWaghela
    • @yifanmai
  5. v0.5.122026-01-28 · 2026-01-28 00:35 UTC
    查看更新说明

    Models

    • Add Kimi K2 Instruct 0905 (#3955)
    • Add DeepSeek v3.1 (#3956)
    • Add Cohere Labs Command A (#3957)
    • Allow using Together models without configuration (#3980)
    • Allow using LiteLLM models without configuration (#3981, #3982)
    • Allow using Hugging Face Hub models without configuration (#3983)
    • Remove tokenizer from OpenAIResponseClient (#4020)

    Scenarios

    • Update schema, model metadata and scenario metadata for HELM Arabic (#3959)
    • Set subject to all by default for MMLU-Pro (#3975)
    • Add missing language entry for MMLU-Winogrande-Afr (#3973)
    • Fix scenario description in ThaiExam (#3972)
    • Support newer versions of MedCalc-Bench (#3921)
    • Fix broken BoolQ file download URLs (#4039)

    Framework

    • Allow omitting the tokenizer and disabling request truncation (#3974)
    • Support plugins that use --plugin or project.entry-points.helm (#3916)

    Contributors

    Thank you to the following contributors for your work on this HELM release!

    • @chakravarthik27
    • @Erotemic
    • @frogcat
    • @MiguelAFH
    • @teetone
    • @yifanmai

用户点评

登录用户的真实使用记录，可按版本和硬件筛选
共 0 条

    还没有人点评。第一个说出真实体验的人最有价值。

    README 原文

    未翻译
    在 GitHub 打开

    Holistic Evaluation of Language Models (HELM)

    Note: HELM entered maintenance mode on June 1, 2026. For more information, refer to the Maintenance Mode Policy.

    Holistic Evaluation of Language Models (HELM) is an open source Python framework created by the Center for Research on Foundation Models (CRFM) at Stanford for holistic, reproducible and transparent evaluation of foundation models, including large language models (LLMs) and multimodal models. This framework includes the following features:

    • Datasets and benchmarks in a standardized format (e.g. MMLU-Pro, GPQA, IFEval, WildBench)
    • Models from various providers accessible through a unified interface (e.g. OpenAI models, Anthropic Claude, Google Gemini)
    • Metrics for measuring various aspects beyond accuracy (e.g. efficiency, bias, toxicity)
    • Web UI for inspecting individual prompts and responses
    • Web leaderboard for comparing results across models and benchmarks

    Documentation

    Please refer to the documentation on Read the Docs for instructions on how to install and run HELM.

    Quick Start

    Install the package from PyPI:

    pip install crfm-helm

    Run the following in your shell:

    # Run benchmark
helm-run --run-entries mmlu:subject=philosophy,model=openai/gpt2 --suite my-suite --max-eval-instances 10

# Summarize benchmark results
helm-summarize --suite my-suite

# Start a web server to display benchmark results
helm-server --suite my-suite

    Then go to http://localhost:8000/ in your browser.

    Leaderboards

    We maintain official leaderboards with results from evaluating recent models on notable benchmarks using this framework. Our current flagship leaderboards are:

    We also maintain leaderboards for a diverse range of domains (e.g. medicine, finance) and aspects (e.g. multi-linguality, world knowledge, regulation compliance). Refer to the HELM website for a full list of leaderboards.

    Papers

    The HELM framework was used in the following papers for evaluating models.

    The HELM framework can be used to reproduce the published model evaluation results from these papers. To get started, refer to the documentation links above for the corresponding paper, or the main Reproducing Leaderboards documentation.

    Citation

    If you use this software in your research, please cite the Holistic Evaluation of Language Models paper as below.

    @article{
liang2023holistic,
title={Holistic Evaluation of Language Models},
author={Percy Liang and Rishi Bommasani and Tony Lee and Dimitris Tsipras and Dilara Soylu and Michihiro Yasunaga and Yian Zhang and Deepak Narayanan and Yuhuai Wu and Ananya Kumar and Benjamin Newman and Binhang Yuan and Bobby Yan and Ce Zhang and Christian Alexander Cosgrove and Christopher D Manning and Christopher Re and Diana Acosta-Navas and Drew Arad Hudson and Eric Zelikman and Esin Durmus and Faisal Ladhak and Frieda Rong and Hongyu Ren and Huaxiu Yao and Jue WANG and Keshav Santhanam and Laurel Orr and Lucia Zheng and Mert Yuksekgonul and Mirac Suzgun and Nathan Kim and Neel Guha and Niladri S. Chatterji and Omar Khattab and Peter Henderson and Qian Huang and Ryan Andrew Chi and Sang Michael Xie and Shibani Santurkar and Surya Ganguli and Tatsunori Hashimoto and Thomas Icard and Tianyi Zhang and Vishrav Chaudhary and William Wang and Xuechen Li and Yifan Mai and Yuhui Zhang and Yuta Koreeda},
journal={Transactions on Machine Learning Research},
issn={2835-8856},
year={2023},
url={https://openreview.net/forum?id=iO4LZibEqW},
note={Featured Certification, Expert Certification}
}

    相关项目

    同分类里 star 最高的项目
    对比全部 →
    promptfoo
    promptfoo·TypeScript

    用声明式配置对提示词、Agent 和 RAG 做自动化评测与红队渗透测试，支持多模型横向对比，可接入命令行和 CI/CD。

    提示词评测+4模型评测
    27033+922.7k0.125.12026-11-10 22:09 UTC
    opik
    comet-ml·Python

    面向 LLM 应用、RAG 和 Agent 工作流的可观测与评测平台，提供全链路追踪、自动评估和生产级监控看板，帮团队定位问题并持续优化效果。

    LLM 可观测+4模型评测
    23315+631.9k2.2.1432026-11-10 21:38 UTC
    deepeval
    confident-ai·Python

    一个 Python 的 LLM 评测框架，提供开箱即用的评估指标，帮你在开发和上线前系统检验大模型输出质量。

    LLM评测+3模型评测
    19416+592.2ksetup-v0.1.02026-11-10 13:05 UTC
    ragas
    vibrantlabsai·Python

    面向 LLM 应用的自动化评测工具，用合成测试集和多种指标量化 RAG 与生成效果，帮你在上线前发现质量问题。

    LLM评测+4模型评测
    16284+271.8kv0.4.32026-02-24 07:47 UTC