midscene

Write a test in natural language

Midscene.js brings AI vision to E2E testing. Describe UI steps and expected results in natural language, then use the visual GUI Agent to carry them out and verify what users see. Organize tests with Midscene Test, or integrate the Agent APIs into your existing testing framework.

YAML test cases with Midscene Test

After creating a Midscene Test project and configuring a model and the Web runtime, write a case like this. Replace the example URL with your app's address:

beforeEach:
  - gotoUrl: https://your-shop.example

cases:
  - name: Find headphones under $100
    steps:
      - aiAct: Search for headphones and filter the results to prices under $100
      - aiWaitFor: The filtered search results are visible
      - aiAssert: There are products in the search results, and each costs less than $100

JavaScript in your existing tests

You can also use the Agent directly in your existing tests. Configure a model and open your app in a Playwright page, then describe the same steps in code:

import { PlaywrightAgent } from '@midscene/web/playwright';

const agent = new PlaywrightAgent(page);

await agent.aiAct('Search for headphones and filter the results to prices under $100');
await agent.aiWaitFor('The filtered search results are visible');
await agent.aiAssert('There are products in the search results, and each costs less than $100');

For mobile and desktop apps, use the platform Agent with Vitest. Open the generated HTML report to inspect screenshots, element locations, the AI decision process, and action and assertion results.

👁️ See, act, and verify like a human

E2E testing verifies complete workflows from the user's perspective, so Midscene tests the way a user does: observe the screen, act on what you see, and verify the visible result.

Locate and act from screenshots

Midscene locates elements by their appearance and position, then clicks, types, or scrolls to carry out your instructions. You can target icon-only buttons, custom controls, <canvas>, and elements inside cross-origin iframes without maintaining selectors or adding semantic annotations.

Verify what users see

Describe the expected appearance in natural language to check colors, selection highlights, layout, and visual feedback — including content drawn on <canvas> or displayed in native apps.

await agent.aiAssert('The selected plan has a blue border and a checkmark');
await agent.aiAssert('The error message is visible below the email field');

🌐 One API, across platforms

Visual localization does not depend on the UI rendering stack, so the same Agent APIs work on every platform:

Platform	Integration
Web	Playwright, Puppeteer, or desktop Chrome
Android	Devices and emulators through ADB
iOS	Devices and simulators through WebDriverAgent
HarmonyOS	HarmonyOS NEXT through HDC
Desktop	macOS, Windows, and Linux, with remote RDP support
Custom interfaces	Provide screenshots and action capabilities

📊 Models, cost, and benchmarks

Choose models for your needs and budget

Screenshot-based UI actions avoid sending large DOM trees to the model. In the reported AppControlBench run, Midscene with Doubao Seed 2.1 Turbo passed 58 of 60 tasks with $0.59 in total model cost. The report provides per-task costs and comparisons across models.

Midscene supports GPT, Qwen, Doubao Seed, DeepSeek, Gemini, Kimi, and more, including open-source options you can self-host. See Model Selection Guide to choose a model and Model Configuration to set it up.

Benchmarks

Benchmark	Pass@1	Model used in the reported run
AndroidWorld	93.1%	Gemini-3.5-Flash
MobileWorld	78.6%	Gemini-3.6-Flash
AppControlBench	96.7%	Doubao Seed 2.1 Turbo

Each report includes the run configuration and task results; see AndroidWorld's report for its environment and validator adjustments. Explore cross-platform showcases, from web form validation to mobile workflows and community-built in-vehicle testing.

🧩 Midscene Test: a natural-language E2E testing framework

Midscene Test (@midscene/test) is a natural-language E2E testing framework. Describe actions and expectations in YAML, and keep logic that needs precise control in TypeScript.

Test cases maintained by people and AI

• Describe test cases in YAML. Combine UI actions, assertions, and business operations to express the flow and expected results.
• Reuse business capabilities through TypeScript Nodes. Wrap data preparation, API calls, and cleanup in Nodes that different cases can share. A refund test can prepare an order through an API, request a refund through the UI, and verify the result in one workflow.
• Generate a Node Spec for developers and AI agents. Export registered Nodes and their inputs as Markdown, so both can write cases without reading TypeScript.

Test framework capabilities

• Lifecycle hooks and cleanup: setup, fileSetup, beforeAll, and beforeEach with matching cleanup hooks; cleanup still runs after timeouts or Ctrl+C.
• Retries and timeouts: case retries, step timeouts, continue-on-error, and bail.
• Concurrency and isolation: maxConcurrentFiles, with fileSetup giving each file its own page, Agent, or device.
• Step-level reports: screenshots, inputs, outputs, and Agent replay for every step, #runner-step=last-error deep links, and summary.json for scripts and AI agents.

Start with Create and use test projects, then learn how to Develop custom Nodes and control lifecycle and concurrency.

Reports, Test results, caches and logs are saved in midscene_run/. See artifact directories for configuration and CI archiving.

🚀 Get started

• Write YAML test cases — create a Midscene Test project.
• Add visual capabilities to existing tests — Playwright for web apps, Vitest for mobile and desktop apps.
• Try it before writing tests — the Chrome extension or a Playground for mobile or desktop.
• Run standalone automation scripts — the JavaScript SDK.
• Let your AI agent operate the UI — Midscene Skills.

📄 Resources

• Documentation: https://midscenejs.com
• Sample projects: midscene-example
• API reference: https://midscenejs.com/reference/#common

🤝 Community

• Discord
• Follow us on X
• Lark Group (飞书交流群)

🌟 Awesome Midscene

Community projects that extend Midscene.js capabilities:

• midscene-ios - iOS Mirror automation support for Midscene
• midscene-pc - PC operation device for Windows, macOS, and Linux
• midscene-pc-docker - Docker image with Midscene-PC server pre-installed
• Midscene-Python - Python SDK for Midscene automation
• midscene-java by @Master-Frank - Java SDK for Midscene automation
• midscene-java by @alstafeev - Java SDK for Midscene automation

📝 Credits

We would like to thank the following projects:

• Rsbuild and Rslib for the build tools.
• UI-TARS for the open-source agent model UI-TARS.
• Qwen-VL for the open-source multimodal model Qwen-VL.
• scrcpy and yume-chan for browser-based Android device control.
• appium-adb for its JavaScript bridge to ADB.
• appium-webdriveragent for controlling XCTest from JavaScript.
• YADB for improving text input performance.
• libnut-core for cross-platform native keyboard and mouse control.
• Puppeteer for browser automation and control.
• Playwright for browser automation, control, and testing.

📖 Citation

If you use Midscene.js in your research or project, please cite:

@software{Midscene.js,
  author = {Xiao Zhou, Tao Yu, YiBing Lin},
  title = {Midscene.js: GUI Agent for E2E Testing.},
  year = {2025},
  publisher = {GitHub},
  url = {https://github.com/web-infra-dev/midscene}
}

⭐ Star Midscene

If Midscene is useful to you, please give it a star.

📝 License

Midscene.js is MIT licensed.