登录

ragas

vibrantlabsaiApache-2.0更新放缓

面向 LLM 应用的自动化评测工具，用合成测试集和多种指标量化 RAG 与生成效果，帮你在上线前发现质量问题。

Supercharge Your LLM Application Evaluations 🚀

v0.4.3发布于 2026-01-13 17:47 UTC16,284star +27 / 7 天1,807fork同步于 2026-11-10 23:27 UTC
LLM评测RAG评估合成测试集评估指标LLMOps
对比 GitHub 官网evaluationllmllmops

怎么上手

由 README 提炼，执行前请确认
原文 
pip install ragas
硬件要求按 README 说明

README 未说明

下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼
没人工点评？写下你的经验

Ragas 是评估和优化 LLM 应用的工具包，用客观指标和自动生成测试集替代主观人工评估

面向 LLM 应用的自动化评测工具，用合成测试集和多种指标量化 RAG 与生成效果，帮你在上线前发现质量问题。

核心能力

  • 提供基于 LLM 与传统方法的客观评估指标
  • 自动生成覆盖多场景的测试数据集
  • 内置 DiscreteMetric 等预置指标，可自定义评估维度
  • 与 LangChain 及主流可观测性工具无缝集成
  • 用生产数据构建反馈回路持续优化应用

容易踩的坑

示例默认使用 OpenAI，需先设置 OPENAI_API_KEY 环境变量；默认会收集匿名使用数据，可设 RAGAS_DO_NOT_TRACK=true 关闭；quickstart 目前仅提供 rag_eval 模板，其余模板尚未上线

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

最近发版

5 个版本v0.4.3
全部 release
  1. v0.4.3最新正式版2026-01-13 · 2026-01-13 17:47 UTC

    What's Changed

    Full Changelog: https://github.com/vibrantlabsai/ragas/compare/v0.4.2...v0.4.3

  2. v0.4.22025-12-23 · 2025-12-23 17:13 UTC
    查看更新说明

    What's Changed

    New Contributors

  3. v0.4.12025-12-10 · 2025-12-10 16:28 UTC
    查看更新说明

    What's Changed

    New Contributors

    Full Changelog: https://github.com/vibrantlabsai/ragas/compare/v0.4.0...v0.4.1

  4. v0.4.02025-12-03 · 2025-12-03 16:22 UTC
    查看更新说明

    What's Changed

    New Contributors

    Full Changelog: https://github.com/vibrantlabsai/ragas/compare

  5. v0.3.92025-11-11 · 2025-11-11 17:24 UTC
    查看更新说明

    What's Changed

    New Contributors

    Full Changelog: https://github.com/explodinggradients/ragas/compare/v0.3.8...v0.3.9

用户点评

登录用户的真实使用记录，可按版本和硬件筛选
共 0 条

    还没有人点评。第一个说出真实体验的人最有价值。

    README 原文

    未翻译
    在 GitHub 打开

    Objective metrics, intelligent test generation, and data-driven insights for LLM apps

    Ragas is your ultimate toolkit for evaluating and optimizing Large Language Model (LLM) applications. Say goodbye to time-consuming, subjective assessments and hello to data-driven, efficient evaluation workflows. Don't have a test dataset ready? We also do production-aligned test set generation.

    Key Features

    • 🎯 Objective Metrics: Evaluate your LLM applications with precision using both LLM-based and traditional metrics.
    • 🧪 Test Data Generation: Automatically create comprehensive test datasets covering a wide range of scenarios.
    • 🔗 Seamless Integrations: Works flawlessly with popular LLM frameworks like LangChain and major observability tools.
    • 📊 Build feedback loops: Leverage production data to continually improve your LLM applications.

    :shield: Installation

    Pypi:

    pip install ragas

    Alternatively, from source:

    pip install git+https://github.com/vibrantlabsai/ragas

    :fire: Quickstart

    Clone a Complete Example Project

    The fastest way to get started is to use the ragas quickstart command:

    # List available templates
ragas quickstart

# Create a RAG evaluation project
ragas quickstart rag_eval

# Specify where you want to create it.
ragas quickstart rag_eval -o ./my-project

    Available templates:

    • rag_eval - Evaluate RAG systems

    Coming Soon:

    • agent_evals - Evaluate AI agents
    • benchmark_llm - Benchmark and compare LLMs
    • prompt_evals - Evaluate prompt variations
    • workflow_eval - Evaluate complex workflows

    Evaluate your LLM App

    ragas comes with pre-built metrics for common evaluation tasks. For example, Aspect Critique evaluates any aspect of your output using DiscreteMetric:

    import asyncio
from openai import AsyncOpenAI
from ragas.metrics import DiscreteMetric
from ragas.llms import llm_factory

# Setup your LLM
client = AsyncOpenAI()
llm = llm_factory("gpt-4o", client=client)

# Create a custom aspect evaluator
metric = DiscreteMetric(
    name="summary_accuracy",
    allowed_values=["accurate", "inaccurate"],
    prompt="""Evaluate if the summary is accurate and captures key information.

Response: {response}

Answer with only 'accurate' or 'inaccurate'."""
)

# Score your application's output
async def main():
    score = await metric.ascore(
        llm=llm,
        response="The summary of the text is..."
    )
    print(f"Score: {score.value}")  # 'accurate' or 'inaccurate'
    print(f"Reason: {score.reason}")


if __name__ == "__main__":
    asyncio.run(main())

    Note: Make sure your OPENAI_API_KEY environment variable is set.

    Find the complete Quickstart Guide

    Want help in improving your AI application using evals?

    In the past 2 years, we have seen and helped improve many AI applications using evals. If you want help with improving and scaling up your AI application using evals.

    🔗 Book a slot or drop us a line: [email protected].

    🫂 Community

    If you want to get more involved with Ragas, check out our discord server. It's a fun community where we geek out about LLM, Retrieval, Production issues, and more.

    Contributors

    +----------------------------------------------------------------------------+
|     +----------------------------------------------------------------+     |
|     | Developers: Those who built with `ragas`.                      |     |
|     | (You have `import ragas` somewhere in your project)            |     |
|     |     +----------------------------------------------------+     |     |
|     |     | Contributors: Those who make `ragas` better.       |     |     |
|     |     | (You make PR to this repo)                         |     |     |
|     |     +----------------------------------------------------+     |     |
|     +----------------------------------------------------------------+     |
+----------------------------------------------------------------------------+

    We welcome contributions from the community! Whether it's bug fixes, feature additions, or documentation improvements, your input is valuable.

    1. Fork the repository
    2. Create your feature branch (git checkout -b feature/AmazingFeature)
    3. Commit your changes (git commit -m 'Add some AmazingFeature')
    4. Push to the branch (git push origin feature/AmazingFeature)
    5. Open a Pull Request

    🔍 Open Analytics

    At Ragas, we believe in transparency. We collect minimal, anonymized usage data to improve our product and guide our development efforts.

    ✅ No personal or company-identifying information

    ✅ Open-source data collection code

    ✅ Publicly available aggregated data

    To opt-out, set the RAGAS_DO_NOT_TRACK environment variable to true.

    Cite Us

    @misc{ragas2024,
  author       = {VibrantLabs},
  title        = {Ragas: Supercharge Your LLM Application Evaluations},
  year         = {2024},
  howpublished = {\url{https://github.com/vibrantlabsai/ragas}},
}

    相关项目

    同分类里 star 最高的项目
    对比全部 →
    promptfoo
    promptfoo·TypeScript

    用声明式配置对提示词、Agent 和 RAG 做自动化评测与红队渗透测试，支持多模型横向对比，可接入命令行和 CI/CD。

    提示词评测+4模型评测
    27033+922.7k0.125.12026-11-10 22:09 UTC
    opik
    comet-ml·Python

    面向 LLM 应用、RAG 和 Agent 工作流的可观测与评测平台，提供全链路追踪、自动评估和生产级监控看板，帮团队定位问题并持续优化效果。

    LLM 可观测+4模型评测
    23315+631.9k2.2.1432026-11-10 21:38 UTC
    deepeval
    confident-ai·Python

    一个 Python 的 LLM 评测框架，提供开箱即用的评估指标，帮你在开发和上线前系统检验大模型输出质量。

    LLM评测+3模型评测
    19416+592.2ksetup-v0.1.02026-11-10 13:05 UTC
    lm-evaluation-harness
    EleutherAI·Python

    统一评测语言模型少样本能力的框架，支持多种任务和模型，方便横向对比不同模型的性能表现。

    模型评测+4模型评测
    14444+203.8kv0.4.132026-11-04 14:34 UTC