OSWorld-V2

[!IMPORTANT] New benchmark release: osworld-v2.1 is now available and recommended.

TL;DR: Upgrade from OSWorld 2.0 to OSWorld 2.1

1. Save local changes and update the main checkout:

git fetch origin tag osworld-v2.1
git switch --detach osworld-v2.1

2. Update the host environment:

uv sync --frozen

3. Download matching task files (xlangai/[email protected]):

uv run scripts/tools/download_osworld_v2_tasks.py \
  --benchmark-release osworld-v2.1

4. Download complete assets (xlangai/[email protected]) into a fresh directory:

uv run scripts/tools/download_osworld_v2_assets.py \
  --benchmark-release osworld-v2.1 \
  --target-dir cache/osworld_v2_assets_v2.1

export OSWORLD_FILE_BASE_URL="$(pwd)/cache/osworld_v2_assets_v2.1"
export OSWORLD_BENCHMARK_RELEASE="osworld-v2.1"

5. Public assets (xlangai/[email protected]) are pinned by the code. No separate download is needed; update custom mirrors to the same tag.

6. Update and redeploy Task-Web/[email protected]. Run in the website checkout:

git fetch origin tag osworld-v2.1
git switch --detach osworld-v2.1
git submodule sync --recursive
git submodule update --init --recursive

Then set the website suffix on the evaluation host:

export WEBSITE_HOST_SUFFIX="<your-v2.1-host-suffix>"

7. Use the matching provider images:

• Docker: uses xlangai/[email protected] and a pinned runtime digest. Stop the old VM and move existing docker_vm_data to a backup location before the first run so the matching image is downloaded.
• AWS: reuse ami-01017272139e01feb in us-east-1 (1920x1080). No new AMI is needed.

8. Update your runner's asset path, website suffix and credentials, then run a small evaluation. See the evaluation guide and version matrix.

The following is the original announcement for the 2026.08.08 release. For the current recommended release, use the osworld-v2.1 instructions in this README.

[!IMPORTANT] New benchmark release: osworld-v2-2026.08.08 is now available.

TL;DR: Migrate from osworld-v2-2026.06.24 to osworld-v2-2026.08.08

1. Download the latest Python task files (xlangai/[email protected]):

uv run scripts/tools/download_osworld_v2_tasks.py \
  --benchmark-release osworld-v2-2026.08.08

2. Download the latest task assets (xlangai/[email protected]):

uv run scripts/tools/download_osworld_v2_assets.py \
  --benchmark-release osworld-v2-2026.08.08 \
  --target-dir cache/osworld_v2_assets \
  --clean

export OSWORLD_FILE_BASE_URL="$(pwd)/cache/osworld_v2_assets"

3. Update the OSWorld-V2 checkout to xlang-ai/[email protected]:

4. Update self-hosted websites to Task-Web/[email protected]:

When using the team-hosted websites, change the website suffix:

export WEBSITE_HOST_SUFFIX="site.hku.icu"

5. Update the host environment. The uv dependency files changed and now require imageio-ffmpeg>=0.6.0:

uv sync --frozen
# Or install only the new package in an existing host environment:
uv pip install "imageio-ffmpeg>=0.6.0"

The provider VM images are unchanged between the 0624 and 0808 releases. See Benchmark Releases for the complete version list.

[!WARNING] Development version: The main branch contains the latest development code. For reproducible benchmark evaluation, use a supported release instead. Recommended: osworld-v2-2026.08.08. See Benchmark Releases for details.

📢 Updates

• 2026-09-16: We released osworld-v2.1, a new bug-fix release with updated agents, evaluation workflows, and pinned task, asset, website, and provider image references.
• 2026-09-08: We no longer host the osworld-v2-2026.06.24 websites at web.hku.icu. To use this version, self-host Task-Web/[email protected].
• 2026-08-08: We published the osworld-v2-2026.08.08 benchmark release with updated code, task files, task assets, and mocked websites.
• 2026-06-26: We released our paper, environment and benchmark, project page, and the osworld-v2-2026.06.24 benchmark release.

Benchmark Versions

Version	Status	Hosted website	Provider images
osworld-v2.1	Active (recommended)	Self-host osworld-v2.1	Images pinned by osworld-v2.1
osworld-v2-2026.08.08	Active	site.hku.icu	Unchanged from 2026.06.24
osworld-v2-2026.06.24	Active	Not hosted; self-host v2026.06.24	Images pinned by 2026.06.24

See Benchmark Releases for the complete version matrix and pinned component versions.

💾 Setup Evaluation Environment

[!WARNING] Release version control: For a reproducible evaluation, use the OSWorld-V2 code, Hugging Face Python task files, Hugging Face task assets, and mocked website deployment from the same release.

The setup instructions below use the recommended release, osworld-v2.1:

• OSWorld-V2 code: xlang-ai/[email protected]
• Python task files: xlangai/[email protected]
• Task assets: xlangai/[email protected]
• Public runtime assets: xlangai/[email protected] (separate pinned component)
• Mocked websites: self-host Task-Web/[email protected]
• Provider VM images: Docker Ubuntu artifact pinned by commit and SHA256; AWS AMI pinned by region and image ID

Do not mix releases or replace a pinned tag with main or latest. The osworld-v2.1 checkout includes its manifest and both download scripts; no bootstrap checkout is needed.

Migrating from OSWorld 1.0

If you are coming from xlang-ai/OSWorld, see Migrating from OSWorld 1.0 to OSWorld 2.1 for the full checklist covering dependency setup, task conversion, provider reuse, AWS notes, mocked websites, GitLab, agent migration, and result comparability.

Setup with Agent

Clone this repository:

git clone --branch osworld-v2.1 https://github.com/xlang-ai/OSWorld-V2
cd OSWorld-V2

We provide a skill called setup-osworld. You can use your favorite agent to help you set up with a single prompt!

Sample prompt:

Use $setup-osworld --version 2.1 to provision this OSWorld 2.1 checkout after clone. Use benchmark release osworld-v2.1 for all release-controlled components. Ask me first which supported provider and optional services I want, create or verify the required infrastructure where possible, ask before any cloud spend, DNS, SSH, or secret step, then report what is configured versus blocked and give me the final export commands I need to run OSWorld.

Setup Manually

Package Setup

git clone --branch osworld-v2.1 https://github.com/xlang-ai/OSWorld-V2
cd OSWorld-V2
uv sync --frozen

The project metadata requires Python >= 3.12 through pyproject.toml. Optional full dependencies for heavier agents or OCR/model stacks (only for v1 tasks) can be installed with:

uv sync --extra full

Environment Provider Setup

OSWorld 2.1 can run with different environment providers depending on where you test:

• Docker for Linux servers, especially hosts with KVM support.
• AWS for large-scale parallel evaluation or training infrastructure.

Select the matching provider in runner commands with --provider_name. The environment provider layer also includes VMware, Azure, GCP, Aliyun, and Volcengine code paths. Currently we only provide images for docker and AWS provider, if you want to use other providers, see migrating from 1.0 to update 1.0 image to 2.1.

For AWS, make sure the client VM security group allows the V2 task service ports 3000 and 8000 in addition to the standard OSWorld backend and control ports documented in Provider Setup.

Mocked Website and GitLab Setup

Some OSWorld 2.1 tasks use mocked websites and GitLab.

Self-host Task-Web/[email protected] and set your deployment’s host suffix:

export WEBSITE_HOST_SUFFIX="<your-v2.1-host-suffix>"

The osworld-v2-2026.06.24 websites are no longer hosted at web.hku.icu; self-host Task-Web/[email protected] to use this version.

You can also self-host the mocked websites by following Task-Web/[email protected], then setting WEBSITE_HOST_SUFFIX to your own host suffix.

For GitLab, you must self-host because GitLab-backed tasks require a private token, and exposing a shared hosted token creates a security risk. Follow Task-Web/gitlab, then set the matching environment variables before running tasks:

export GITLAB_URL="<your-gitlab-url>"
export GITLAB_PRIVATE_TOKEN="<your-private-token>"

Download Task Classes and Assets

The official OSWorld 2.1 Python task classes are distributed through the gated Hugging Face dataset xlangai/osworld_v2_tasks, not directly in this public GitHub checkout. This reduces benchmark leakage and helps prevent evaluated agents from finding task answers, setup logic, or evaluator details online while executing a task.

To download them:

1. Visit https://huggingface.co/datasets/xlangai/osworld_v2_tasks while logged in to Hugging Face and accept the gated access request, your request will automatically be approved.
2. Visit https://huggingface.co/datasets/xlangai/osworld_v2_assets_gated and accept its gated access request.
3. Log in locally:

uvx --from huggingface_hub hf auth login

4. Download or update the task classes for the current release:

uv run scripts/tools/download_osworld_v2_tasks.py --benchmark-release osworld-v2.1

Official benchmark release manifests live in benchmark_releases/. See Benchmark Releases for how releases pin task, asset, website, code, and provider-image versions for comparable runs.

The script writes files to evaluation_examples/task_class, overwrites existing task_*.py files when re-run, and removes stale local task files by default. Use --dry-run to preview changes or --keep-stale to keep local task files that are no longer present in the gated dataset.

See task class download notes for details.

The complete asset snapshots are distributed only through the gated xlangai/osworld_v2_assets_gated dataset. The public xlangai/osworld_v2_assets dataset is reserved for files that browser-facing task state must access without Hugging Face authentication; it is not a complete asset snapshot. All users should download the gated snapshot locally before running tasks rather than relying on either repository for online asset resolution.

Official V2 task classes may reference large setup files, ground-truth files, and other task assets through desktop_env.file_source.asset(...). Before running tasks, download the asset snapshot pinned by the selected benchmark release and point OSWORLD_FILE_BASE_URL at the local directory:

uv run scripts/tools/download_osworld_v2_assets.py \
  --benchmark-release osworld-v2.1 \
  --target-dir cache/osworld_v2_assets \
  --clean

export OSWORLD_FILE_BASE_URL="$(pwd)/cache/osworld_v2_assets"

Use --revision <branch-tag-or-commit> if you need a specific asset version without a benchmark release manifest. Use --allow-pattern 'task_027/**' for task-scoped downloads during local debugging.

For an existing checkout, save local changes and switch before installing or downloading release inputs:

git fetch origin tag osworld-v2.1
git switch --detach osworld-v2.1
uv sync --frozen
git describe --tags --exact-match
# Expected: osworld-v2.1

Keep OSWORLD_FILE_BASE_URL set while evaluating so the release does not fall back to the public runtime-only asset snapshot, which is not a complete asset source.

Proxy Configuration

Some tasks may require proxy settings depending on website defenses and network location. See Proxy Guideline.

• Impact of Missing Configuration: If these settings are missing, corresponding tasks may fail and lower the evaluation score.

🧪 Evaluation

Run Sample

For a concrete multi-environment Claude run, start from scripts/bash/run_multienv_claude.sh. Before executing it, fill in every environment variable required by your provider, model endpoint, and selected tasks. At minimum, check the AWS variables, model API key environment variable, OSWorld client password, website host suffix, and any GitLab credentials needed by the task set.

The sample script exports the required variables near the top, then calls scripts/python/run_multienv_claude.py through uv. Adjust options such as --result_dir, --test_all_meta_path, --num_envs, --provider_name, and model settings for your run. Before running evaluations, ensure the matching task assets have been downloaded and set OSWORLD_FILE_BASE_URL to their local directory; do not rely on online asset resolution.

# Edit scripts/bash/run_multienv_claude.sh first:
# - AWS_REGION, AWS_SUBNET_ID, AWS_SECURITY_GROUP_ID
# - AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
# - ANTHROPIC_API_KEY, or the API key named by OSWORLD_EVAL_MODEL_API_KEY_ENV
# - OSWORLD_CLIENT_PASSWORD
# - WEBSITE_HOST_SUFFIX
# - OSWORLD_FILE_BASE_URL, preferably an absolute local asset directory
# - GITLAB_URL and GITLAB_PRIVATE_TOKEN, if the selected tasks use GitLab

bash scripts/bash/run_multienv_claude.sh

Qwen models served through an OpenAI-compatible endpoint can use the modular mm_agents.qwen implementation and its dedicated runner:

export OPENAI_BASE_URL=http://127.0.0.1:8000/v1
export OPENAI_API_KEY=dummy

uv run python scripts/python/run_multienv_qwen.py \
  --smoke_only \
  --model your-served-model \
  --base_url "$OPENAI_BASE_URL"

Manual Task Examination

Use scripts/python/manual_examine.py when you want to inspect a specific benchmark task by hand, verify task behavior, debug evaluator metrics, or record a short manual trajectory with screenshots and videos.

uv run python scripts/python/manual_examine.py \
  --headless \
  --provider_name aws \
  --observation_type screenshot \
  --result_dir ./results_human_examine \
  --test_config_base_dir evaluation_examples \
  --domain tasks \
  --eval_version v2 \
  --example_id 146 \
  --max_steps 3

Local Evaluation

Correctly implement the agent interface and import your customized version in the run.py (for single-threaded execution) or scripts/python/run_multienv.py / scripts/python/run_multienv_xxx.py (for parallel execution) file. For implementation references, you can follow the current Claude or GPT implementations, such as scripts/python/run_multienv_claude.py and scripts/python/run_multienv_gpt_response_api.py.

If you are migrating an agent from OSWorld 1.0, you can use the migrate-osworld-agent skill to help port and test the agent. Sample migration prompt:

Use $migrate-osworld-agent to migrate my OSWorld 1.0 agent into this OSWorld 2.1 checkout. Read the upstream agent <agent file path> and the local Claude/GPT runner patterns, add the adapted agent under mm_agents/, create matching multi-env Python and bash entrypoints, then run the fast checks and one small smoke test without staging unrelated local files.

Afterward, you can execute a command similar to the one in the previous section to run the benchmark on your agent.

Public Evaluation

If you want your results to be verified and displayed on the verified leaderboard, you need to schedule a meeting with us (current maintainer: [email protected], [email protected]) to run your agent code on our side and have us report the results. You need to upload and allow us to disclose your agent implementation under the OSWorld framework (you may choose not to expose your model API to the public), along with a report that allows the public to understand what's happening behind the scenes. Alternatively, if you are from a trusted institution, you can share your monitoring data and trajectories with us. Please carefully follow the OSWorld 2.1 Public Evaluation Guide to get results.

❓ FAQ

What is the username and password for the virtual machines?

For all providers in OSWorld 2.1, the Ubuntu account credentials are user / osworld-public-evaluation.

If you modify credentials, pass the correct client_password to DesktopEnv and to agents that require it. Some setup and task actions require sudo privileges inside the VM.

How do I configure a proxy for the VM?

If you are behind a restricted network, or if some tasks are blocked by website defenses, see Proxy Guideline. The public evaluation guide also describes a pre-configured proxy option.

Where are task files stored?

OSWorld 2.1 task classes are downloaded into evaluation_examples/task_class with scripts/tools/download_osworld_v2_tasks.py. Complete initial files, ground-truth files, and other task assets are stored in the gated xlangai/osworld_v2_assets_gated dataset. Accept access to that dataset, download the matching snapshot with scripts/tools/download_osworld_v2_assets.py, and set OSWORLD_FILE_BASE_URL to the local directory before running tasks. The public xlangai/osworld_v2_assets dataset contains only assets that task state must access anonymously and is not a complete download source.

📚 Citation

If you find this environment useful, please consider citing OSWorld 2.0:

@misc{yuan26osworld2,
      title={OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks}, 
      author={Mengqi Yuan and Zilong Zhou and Xinzhuang Xiong and Weiming Wu and Jiayang Sun and Jiamin Song and Kaiqian Cui and Bowen Wang and Haoyuan Wu and Yitong Li and Dunjie Lu and Haikong Lu and Qi Zhen and Xinyuan Wang and Jiaqi Deng and Yuhao Yang and Cheng Chen and Boyuan Zheng and Alex Su and Xiao Yu and Hao Zou and Saaket Agashe and Xing Han Lu and Manpreet Kaur and Zhengyang Qi and Vincent Sunn Chen and Frederic Sala and Dayiheng Liu and Junyang Lin and Zhou Yu and Yu Su and Siva Reddy and Xin Eric Wang and Peng Qi and Tianbao Xie and Tao Yu},
      year={2026},
      eprint={2606.29537},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2606.29537}, 
}

Acknowledgement for OSWorld 2.0

We thank Cheng Chang, Dawn Song, Delin Chen, Ke Xu, Qiyue Xu, Ruiling Xu, Shengwei Wang, Yanzhuo Lin, Yimo Cai, Yiyong Sun, and Yutong Yao for their helpful feedback and for contributing materials to this benchmark. We thank Snorkel AI, our research & data partner, for their support of this work. We gratefully acknowledge support from the Google Research gift fund.