ClawBench

Quick start · Results · Docs · Citation · 中文 · WebsiteBench

ClawBench is an open-source benchmark, published in Findings of EMNLP 2026, that evaluates AI browser agents on everyday online tasks — booking travel, ordering food, applying for jobs, managing email — across live websites. It records five layers of evidence and supports request-based scoring and trajectory evaluation against human references. In the historical V1 paper evaluation, the strongest evaluated agent completed about 1 in 3 tasks.

V1: 152 tasks · 143 live sites  |  V2: 129 tasks · 63 live sites  |  281 tasks across 163 live websites · 15 life categories

Built by NAIL Group · Sister project: WebsiteBench — Evaluation of agents' ability in reverse-engineering websites · Runs on any Chrome.

Featured in 37 curated lists and #3 Paper of the Day on Hugging Face: docs/featured.md.

What are you looking for?

I want to…	Start here
Run my model	Quick start
See scores	Live leaderboard · Results
Browse the tasks	Task explorer · Datasets
Reproduce a score	Reproduction workflow · Trace cookbook
Understand the benchmark	Paper · FAQ · How ClawBench compares
Change the framework or add tasks	Source layout · Contributing

All guides and integrations: Documentation index.

Quick start

Prerequisites: Python 3.11+, uv, and a running Docker or Podman engine. See platform setup.

Installed package

Install clawbench-eval from PyPI:

uv tool install clawbench-eval
clawbench

Launch from the directory where you want your configuration and results. The interactive TUI guides you through adding a model and selecting a task. Configure the default judge, deepseek-v4-pro, as well; the model setup guide explains judge selection and how to run without scoring.

Source checkout

git clone https://github.com/TIGER-AI-Lab/ClawBench.git
cd ClawBench
cp models/models.example.yaml models/models.yaml
# Edit models/models.yaml with your model and judge API settings.
uv run clawbench

./run.sh is a shortcut for the same TUI. For a non-interactive run from the checkout:

uv run clawbench-run test-cases/v1/886-entertainment-hobbies-experience-topgolf your-provider-model-id

Replace your-provider-model-id with a configured model key. The first run builds container images; the local browser's printed noVNC URL lets you watch the agent. Recordings and metadata land under test-output/.

Continue with the getting started guide for batch and human runs, harness selection, and remote browser options. The CLI reference covers non-interactive commands. Coding agents can start with AGENTS.md.

How it works

   You pick a task            ClawBench spins up           Agent drives the         Interceptor captures
   from V1 or V2              an isolated Docker           browser: navigates,      every action across
   everyday scenarios         container + Chromium         fills forms, clicks      all 5 layers of data

   ┌──────────────┐           ┌──────────────┐           ┌──────────────┐           ┌──────────────┐
   │  "Book a pet │    ──►    │   Container  │    ──►    │   AI Agent   │    ──►    │   5 layers   │
   │   sitter on  │           │  + Chromium  │           │  browses the │           │  intercepted │
   │   Rover"     │           │  + Agent     │           │   live site  │           │  & recorded  │
   └──────────────┘           └──────────────┘           └──────────────┘           └──────────────┘

Live websites · isolated containers · request interception · five-layer recording.

┌─────────────────────────────────────────────────┐
│  Container (Docker / Podman)                    │
│                                                 │
│  ┌──────────┐  CDP Fetch/Runtime/Page events    │
│  │ Chromium ├─────────────────────────────┐     │
│  │ :9222 CDP│                             │     │
│  └──────────┘                             │     │
│                                           │     │
│  ┌──────────┐            ┌────────────────▼─┐   │
│  │  Xvfb    │◄──ffmpeg──►│  FastAPI Server  │   │
│  │ :99      │  x11grab   │  :7878           │   │
│  └──────────┘            └──────────────────┘   │
│                                  │              │
│                          ┌───────▼─────────┐    │
│                          │     /data       │    │
│                          │  actions.jsonl  │    │
│                          │  requests.jsonl │    │
│                          │  screenshots/   │    │
│                          │  recording.mp4  │    │
│                          └─────────────────┘    │
└─────────────────────────────────────────────────┘

Datasets

ClawBench ships three Hugging Face datasets — task definitions plus full execution traces for V1 and V2. All open, downloadable in one command.

Dataset	What's in it	Get it
NAIL-Group/ClawBench (mirrored at TIGER-Lab/ClawBench)	Task definitions, rubrics, and metadata for V1 and V2 — what to attempt and how it's judged.	hf download --repo-type dataset NAIL-Group/ClawBench
NAIL-Group/ClawBenchV1Trace	One directory per V1 model run: recording.mp4, requests.jsonl, actions.jsonl, agent-messages.jsonl, interception.json, run-meta.json.	hf download --repo-type dataset NAIL-Group/ClawBenchV1Trace
TIGER-Lab/ClawBenchV2Trace	Same 5-layer bundle for V2 runs. Rolling — new models added as they're evaluated.	hf download --repo-type dataset TIGER-Lab/ClawBenchV2Trace

Build with the traces: See the Trace Cookbook for download recipes, data schemas, and research directions.

The trace datasets are large; use hf download --include "<pattern>" to pull a single model or a single task.

Live leaderboard: claw-bench.com/leaderboard (V2 default, two-stage scoring — interception + LLM judge). Full scoring formula in eval/scoring.md. Add your run: PR to leaderboard/results.csv.

News

• [2026.11.05] — Check out our new project: WebsiteBench — Evaluation of agents' ability in reverse-engineering websites.
• [2026.10.10] — ClawBench has surpassed 1,000 GitHub stars! Thank you to everyone who has starred, used, and contributed to the benchmark.
• [2026.08.20] — Our paper has been accepted to EMNLP 2026 Findings.

Earlier updates are in docs/news.md; the full change history is in CHANGELOG.md.

Awesome Works using ClawBench

Authors from Google DeepMind, Stanford, UC Berkeley, Google, Microsoft Research, Harvard, ETH Zürich, Oxford, Northwestern, ByteDance Seed, and HKUST build on ClawBench — and Li Auto's Mach-Mind-4-Flash technical report evaluates on it.

• Google DeepMind, University of Oxford & Columbia University, The Recipe for Intelligence in Natural and Artificial Systems (DOI)
• Stanford, UC Berkeley, Microsoft Research & UCSB, Auditing Agent Harness Safety (Code, Project)
• Google, Agentic Coding Needs Proactivity, Not Just Autonomy (Google Research Blog)
• Harvard Kempner Institute, Massachusetts General Hospital & CUHK, NeuroClaw Technical Report (Code, Project)
• ETH Zürich & Handshake AI Research, Verifying Agents in Rubric-Graded Environments (Code, Workshop)
• University of Oxford, NUS & Peking University, OpenClaw Research: A Systematic Survey of Large Language Model Agents in Open Deployment (Project, Resources)
• Northwestern University, A Survey of the OpenClaw Ecosystem: From Platform Extensibility to Constraint Design (Project & Resources)
• UC Davis & UT Dallas, Toward Trustworthy Computer-Use Agents: Risk Propagation, Evaluation Gaps, and Human Governance (Code & Project, Resources)

• ByteDance Seed & HKUST, Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context (Models)
• Tencent Hunyuan & Fudan University, TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training
• Unipat AI, VibeSearchBench: Benchmarking Long-horizon Proactive Search in the Wild (Code, Project)
• Tsinghua University & CUHK, WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation (Code, Project)
• NUS, HKUST, Tsinghua University & Peking University, Towards Long-Horizon Agents: A Survey (Project, Resources)
• HKU MMLab, UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks (Code, Project)
• Tsinghua University & SJTU, MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktop (Code, Project)
• Peking University & CUHK, π-Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows (Code, Project)
• SJTU, AcademiClaw: When Students Set Challenges for AI Agents (Code, Project)

If we missed your work, please open an issue or submit a pull request.

Results

V2 · Hermes harness · 8 models · deepseek-v4-pro judge, lenient and strict rubrics · live at claw-bench.com

Rank	Model	Harness	Intercepted	Reward (lenient)	Reward (strict)	Pass / Total
1	claude-opus-4-7	hermes	54.6%	44.6%	24.6%	58 / 130
2	gpt-5.5	hermes	45.4%	35.4%	18.5%	46 / 130
3	glm-5.1	hermes	48.5%	34.6%	17.7%	45 / 130
4	deepseek-v4-pro	hermes	43.9%	33.9%	12.3%	44 / 130
5	openrouter-owl-alpha	hermes	14.6%	0.0%	0.0%	0 / 130
6	z-ai/glm-4.5-air:free	hermes	4.6%	2.3%	0.8%	3 / 130
7	deepseek-v4-flash:free	hermes	3.1%	2.3%	0.0%	3 / 129
8	minimax-m2.5:free	hermes	2.3%	1.5%	0.0%	2 / 130

Intercepted = final HTTP request matched the task's URL/method (Stage 1, deterministic). Reward (lenient) = additionally judged by deepseek/deepseek-v4-pro to fulfill the instruction under the "no contradiction → match" rubric (Stage 2). Reward (strict) = same judge, strict rubric ("ambiguous → mismatch"). Ranked by Intercepted; Reward as tiebreak. Totals reflect the corpus size at run time.

Paper results · V1 · trajectory evaluation against human references · Findings of EMNLP 2026

Rank	Model	Overall
1	Claude Sonnet 4.6	33.3%
2	GLM-5	24.2%
3	Gemini 3 Flash	19.0%
4	Claude Haiku 4.5	18.3%
5	GPT-5.4	6.5%
6	Gemini 3.1 Flash Lite	3.3%

More results are in docs/results.md: the per-category breakdown of the paper results, V2 under OpenClaw, a V1 run under the Hermes harness, and the V1 task categories. Codex and Claude Code runs on V2, and the V1 OpenClaw aggregate, are still in flight — they land on the live leaderboard first.

Reproduce the leaderboard

Two independent runs of the same model under the same judge reproduce Intercepted and Reward within ±2 pp on V2. To check a published row against our traces:

clawbench-reproduce --model deepseek-v4-flash --tolerance 2.0

Re-running the agent, re-judging published traces, and the pass criterion are in docs/reproduce.md.

ClawBench-Lite

New here? Run this first. test-cases/v1-lite/ is a 20-task curated subset of V1, selected for household-name sites, real-world relevance, difficulty, and category diversity. It matches the 20-tasks-per-source convention of browser-use/benchmark and gives you a credible signal at a fraction of the full-benchmark cost.

Tier distribution: flagship 9 / core 8 / wildcard 3 — spanning daily life (OpenTable, DoorDash, Instacart, TaskRabbit), entertainment (Eventbrite, Goodreads, Fandango), creation (Asana, Mailchimp, Squarespace), travel (Airbnb), education (LeetCode), dev-tech (GitHub), academia (Overleaf), personal management (1Password), and more. Use the separate human-reference evaluation workflow to evaluate Lite trajectories under eval/agentic_eval.md; selecting the suite does not switch the native request judge to trajectory evaluation.

Run it with --cases-suite v1-lite, or inspect the link-backed task files in test-cases/v1-lite/.

Example walkthrough

The task walkthrough follows a V1 task from its JSON specification through the recorded browser session and human-reference assessment. It also explains why a placeholder interception rule needs trajectory evaluation.

Evaluation

The native runner scores intercepted requests with an LLM judge by default: a pass requires both an interception and a matching judge verdict. --no-judge reports interception-only results; clawbench-rescore can judge saved requests later. See the CLI reference.

The original V1 evaluation compares complete agent and human-reference trajectories under eval/agentic_eval.md. This is a separate workflow described in the evaluation guide. Compare scores using the same corpus revision, harness, and rubric.

To share rescored results, see the OpenEval / EvalPort export instructions.

CLI

./run.sh                                                                   # interactive TUI
uv run clawbench-run <case-dir> <model>                                    # one task
uv run clawbench-run <case-dir> --human                                    # human reference run
uv run clawbench-batch --models <model> --cases-suite v2 --all-cases       # a whole corpus

Command overview and commonly used flags: CLI reference. Run any command with --help for its complete option list.

V1 tasks are in test-cases/v1/ (152 tasks), V2 in test-cases/v2/ (129), Lite in test-cases/v1-lite/ (20), and converted Claw-Eval tasks in test-cases/claw-eval/ (19). All suites use test-cases/task.schema.json. For test case authoring, see CONTRIBUTING.md; for output structure and evaluation guidance, see eval/README.md.

How ClawBench compares

ClawBench's niche: live consumer websites, everyday tasks, end-to-end recording. If you want a controlled sandbox or replayed traces, benchmarks such as WebArena and Mind2Web are excellent. If you want to know whether your agent can actually order food or book a flight today, this is the benchmark for that. The side-by-side table with WebArena, GAIA, SWE-bench, Mind2Web and five other benchmarks is in docs/comparison.md.

FAQ

See the FAQ for task coverage, account setup, recording artifacts, scoring, and live-site limitations. For installation and model configuration, use Getting started.

Contributing

We welcome contributions -- especially new test cases. If you've ever ordered groceries, booked an appointment, or filed a form online, you already know how to write one. Most PRs are a single JSON file and land in under a day.

Quick wins:

• Add a new test case (~30 min, no container expertise needed)
• Add a new category of 10+ tasks → co-author invitation on the next paper revision
• Submit a new model to the public leaderboard
• Browse good first issues

See CONTRIBUTING.md for the full guide and contributor recognition policy.

Community

• Questions and bugs: GitHub issues, the fastest route to a maintainer.
• 中文社区：微信群，研究者、开发者、贡献者交流。
• Dataset and leaderboard: Hugging Face discussions, next to the data.

Citation

If you use ClawBench in your research, please cite:

@inproceedings{zhang2026clawbench,
  title={Clawbench: Can ai agents complete everyday online tasks?},
  author={Zhang, Yuxuan and Wang, Yubo and Zhu, Yipeng and Du, Penghui and Miao, Junwen and Lu, Xuan and Li, Zhuofeng and Qu, Xingwei and Guo, Zhengkang and Shen, Yuanzhe and others},
  booktitle={Findings of the Association for Computational Linguistics: EMNLP 2026},
  pages={9386--9408},
  year={2026}
}

Contact

For questions, suggestions, bug reports, or feature requests, please open a GitHub issue.

Core Contributors

Advisors

Support ClawBench

If ClawBench is useful for your research or product work, the single most helpful thing you can do is star the repo — it surfaces the benchmark to other AI-agent researchers and helps us justify continued dataset curation.

Open to contributions — new test cases, bug fixes, or evaluation submissions for a model we haven't scored yet. See CONTRIBUTING.md.

Star History

License & Acknowledgments

Apache 2.0 -- see LICENSE.

The converted Claw-Eval suite in test-cases/claw-eval/ is derived from claw-eval/claw-eval and the claw-eval/Claw-Eval dataset, which are released under the MIT License. Third-party package notices are in NOTICE.

Built with OpenClaw, opencode, Claude Code, the Claude in Chrome extension, OpenAI Codex CLI, browser-use, claw-code, Hermes Agent, Pi with pi-browser-harness, and WebBrain (selectable harnesses), Microsoft Playwright MCP (browser control bridge for the opencode, claude-code, codex, and claw-code harnesses), LiteLLM (API translation proxy for the claude-code, claude-code-chrome-extension, codex, browser-use, claw-code, and pi harnesses), noVNC (MPL 2.0), and websockify (LGPL 3.0).