AutoCua

pip install AutoCua

https://github.com/user-attachments/assets/f02ac3e9-fe6c-42f8-a685-88001d8d178e

See everything AutoCua can do → agent_capabilities.md

	Windows	Linux	macOS
Computer use — any desktop app, with real mouse and keyboard input	✓	✓	✓
Browser skills — Chrome, Firefox, Edge, Brave and Opera	✓	✓	✓
Shell use — a coding agent in a real terminal	✓	✓	✓
Web use — your own Chrome, driven through the AutoCua extension	✓	✓	✓
Mobile use — an iOS Simulator, or your iPhone or iPad, over USB, or Wi-Fi in the app	—	—	✓
Mobile use — an Android emulator, or your Android phone, over USB, or Wi-Fi in the app	✓	✓	✓

---

Web agent: fast mode

6.1 seconds for a flight search. Agent mode: web.

https://github.com/user-attachments/assets/8cc797d5-5556-40a9-bbfc-5d6aa8880fb8

Web agent: quality mode

Under a minute for three tasks in one run: an iPhone 18 Pro into the Apple bag, a case into the Amazon basket, and a Google Sheet of Meta's five-year financials. Agent mode: web.

https://github.com/user-attachments/assets/82bf6163-55c9-4cc4-95e7-9c6466f2a25d

iOS agent: iPhone use

A paired iPhone, driven over the USB cable, or Wi-Fi in the app. Agent mode: mobile use, iOS.

https://github.com/user-attachments/assets/3c22eb45-d939-4742-9053-22ecc75b416f

---

Pick a surface

One string decides what the agent drives. Everything else stays the same.

from AutoCua.agent_launcher import run_agent

run_agent(mode="web use", provider="openrouter", model="gemini-3.8-flash",
          task="Open wikipedia.org and summarise today's featured article.")

mode	What it drives	Where it runs
"computer use"	The desktop. Clicks, typing, hotkeys, apps, AppleScript	macOS, Windows or Linux, auto detected
"computer use, daisy chain"	The same desktop, plus the phone sub-agents when a phone is connected: android_agent, and ios_agent on a Mac. Plain "computer use" lists them offline	macOS, Windows or Linux, auto detected
"shell use"	A coding agent, straight into a real terminal	macOS, Windows or Linux
"web use"	Your own Chrome through the AutoCua extension; headless runs on a debugging port	Any host with Google Chrome installed
"mobile use, ios"	An iOS Simulator by default, or your paired iPhone or iPad over the cable	macOS host
"mobile use, android"	An Android emulator by default, or your Android phone over the cable	macOS, Windows or Linux host

From a script a phone always goes over the cable. Wi-Fi is the desktop app's: see iPhone and iPad and Android. A phone needs device="hardware"; the default, device="simulation", is the simulator or the emulator.

On Linux, computer use needs a GNOME desktop (tested on Wayland), and computer use and shell use both need a few distro packages that pip cannot install. See Linux, from pip.

---

Why it holds up on real software

No pixel guessing. The model never sees a coordinate. It picks a numbered element from a view that fuses the operating system's accessibility tree with OCR, and a number it was never shown is rejected before anything moves.

No debugging port on your desktop. Input goes in at the operating system's own input layer. Mouse events posted at kCGHIDEventTap on macOS, pywinauto's real input path on Windows, and a kernel driver when Windows UAC blocks everything else. An ordinary application sees the same events it would get from your hand on the mouse.

No debugging port on your browser either. A normal "web use" run drives your own Chrome through the AutoCua extension, which Chrome loads at launch with no store install. Chrome shows its own "AutoCua started debugging this browser" bar while the agent works, and its Cancel button hands the browser back to you. A debugging port is opened only for headless runs, on a browser of the agent's own. The page scanner never calls Runtime.enable, and the numbered boxes are painted into the screenshot, not into your DOM.

No context wall. A separate compression agent watches the transcript and splices a handoff summary into it mid-run, so a long task finishes instead of dying at the context limit. In the app you watch the memory ring's count fall.

No telemetry. AutoCua sends nothing anywhere except to the model provider you chose, and to Telegram if you connect it yourself.

---

Contents

• Quick start
• How it sees a screen
• How it acts on a screen
• The agent team
• The browser agent
• iPhone and iPad
• Android
• Windows, UAC and the kernel driver
• Automation testing and workflows
• Running tasks in parallel
• The desktop app
• Memory, vault, chats and skills
• Remote control from Telegram
• Providers and models
• Requirements and setup
• Limits and safety
• Project layout
• Roadmap
• Licence, credits and citation

Quick start

pip install AutoCua

On Linux, install a few distro packages first: see Linux, from pip.

Put the four lines from Pick a surface in a main.py, then:

python main.py

That is the whole API.

What the wheel brings, and what it cannot

Three of the native pieces are fetched when you say yes to them on the terminal, and one, the desktop app's Electron, is offered the same way; the Rust agents come prebuilt in the wheel. WebDriverAgent and Google Chrome are never downloaded: you put them in place yourself, and AutoCua tells you how when one is missing. Two cannot: a Windows kernel driver, which pip is structurally unable to install, and on Linux the desktop's accessibility and input bindings, which come from your distro.

	How it reaches you
Browser agent — Rust	Ships prebuilt in the Windows x64, Apple Silicon macOS and Linux x86_64 wheels. On any other platform pip builds it from the source distribution, which needs Rust and a C compiler.
WebDriverAgent — iOS	Never downloaded. It lives in AutoCua_data/WebDriverAgent. While it is missing, iOS automation is paused, and the terminal and the app show the one command that puts it there, with the full path: git clone --depth 1 --branch v15.1.1 https://github.com/appium/WebDriverAgent.git "<your AutoCua_data>/WebDriverAgent". Needs full Xcode and git.
adb — Android	Downloaded when you say yes on the terminal: by the setup scripts, by python main.py once before its window opens, or by python -m AutoCua.android_connector.adb fetch (the window itself cannot ask): Google's platform-tools, unless an adb is already on your PATH, from Google and pinned by size and SHA-256, never shipped in this repo. Nothing is installed on the phone.
Android emulator — Google's	Downloaded on the first emulator run, after you accept Google's SDK terms on the terminal: Google's emulator and the Android version it needs, about 2.5 GB, into a per-user folder. An Android Studio SDK that already has both is used instead. Needs the computer's hypervisor (below).
Android agent — Rust	Ships prebuilt in the same three wheels as the browser agent. On any other platform the source distribution carries its sources, and it builds on the first Android run, about a minute, once, which needs Rust and a C compiler.
Electron — the desktop app's Chromium window	Downloaded only if you say yes. The first ui=True launch from a terminal asks (120 to 160 MB, once) on macOS 13+, Windows 10+ and Linux, except stock Ubuntu 23.10+ (below); without it the app opens in pywebview. python -m AutoCua.utils.electron fetches it any time. No Node.js needed.
Interception — Windows UAC	Not installed by pip or by the setup script. Add it yourself if you want it, below.
AT-SPI, GTK and the desktop portal — Linux	Not installed by pip. One apt line, once — below.

Answering UAC prompts on Windows

Windows' user-mode SendInput cannot deliver input to the UAC secure desktop, so an elevation prompt can only be answered from kernel mode. That is the one thing the Interception driver does here.

pip install cannot set it up, and no packaging trick changes that: a wheel is unpacked and never executed, pip does not run elevated, and the driver binds its input slots only when it loads at boot. Without it every mode still works — you simply lose UAC handling, and a task that hits an elevation prompt stops there.

AutoCua does not download or install it. To enable it, do this yourself, once, in a terminal opened as Administrator:

1. Download the release from its author, github.com/oblitum/Interception/releases (AutoCua is tested with v1.0.1). Check it before running anything: certutil -hashfile Interception.zip SHA256 must print the SHA-256 listed in THIRD_PARTY_NOTICES.md. Then unzip it.

2. From inside the unzipped folder, install it: "command line installer\install-interception.exe" /install

3. From the checkout's top folder, bind it to your built-in keyboard and mouse only, so external keyboards are never filtered:

   powershell -ExecutionPolicy Bypass -File AutoCua\windows\controller\tool\interception_toggle.ps1 -Action bind
   

   It prints one line. If that line does not say "ok":true, do not reboot: run the same command with -Action unbind, which removes every filter, and fix the cause first (most often a terminal not opened as Administrator). A pip install has the same script inside the installed AutoCua package.

4. Reboot. The driver binds its input slots only when it loads at boot.

It is never bundled because it is dual-licensed LGPL-3.0 non-commercial / paid commercial. Shipping it inside the wheel would make this project its distributor to every person who runs pip install, which that licence does not allow for free. You receive it from its author. See THIRD_PARTY_NOTICES.md.

Linux, from pip

The Linux wheel is built for x86_64 and needs a GNOME desktop (tested on Ubuntu with Wayland; X11 is not tested yet). The agent reads and drives the desktop through AT-SPI, GTK and the XDG RemoteDesktop portal. Those bindings come from your distro, not from pip, so install them once:

sudo apt install python3-gi python3-cairo python3-gi-cairo python3-venv gir1.2-atspi-2.0 gir1.2-gtk-3.0 \
                 gir1.2-webkit2-4.1 at-spi2-core xdg-desktop-portal xdg-desktop-portal-gnome \
                 gir1.2-gstreamer-1.0 gstreamer1.0-pipewire gstreamer1.0-plugins-base \
                 libglib2.0-bin libgtk-3-bin xdg-utils

On KDE, swap xdg-desktop-portal-gnome for xdg-desktop-portal-kde. Fedora, Arch and openSUSE ship the same packages under their own names; setup_linux.sh in the repository lists them.

Optional, for faster recordings: also install your distro's ffmpeg. Each run's recording is then compressed on the GPU instead of the CPU, because the ffmpeg pip brings on Linux has no GPU encoders:

sudo apt install ffmpeg            # Debian, Ubuntu
sudo pacman -S --needed ffmpeg     # Arch

Skip it on Fedora and openSUSE: their own ffmpeg builds leave out codecs the app needs (RPM Fusion and Packman have full ones). Without it, recordings are still compressed, on the CPU.

Ubuntu and Debian refuse pip install into the system Python, so create a virtualenv first, inside the folder that holds your main.py and .env so the agent finds that .env. Build it on your distro's own python3 (3.10 to 3.14): AutoCua loads the system PyGObject from inside the venv only for the same 3.x version, so a venv on another version stops with "cannot import gi".

cd my-agent                         # the folder with your main.py and .env
python3 -m venv .venv
source .venv/bin/activate
pip install AutoCua
python main.py                      # the main.py from Quick start

The browser and Android agents are prebuilt in the wheel, so you need no Rust or C compiler. On the first computer-use run GNOME asks for screen-share consent: turn Allow Remote Interaction on before you click Share, because GNOME leaves it off. Your choice is saved in AutoCua_data/. The first scan also downloads the PP-OCRv6 models that read text the desktop does not publish.

A few things worth knowing on the first run

• Add ui=True to open the desktop app instead of running the task in the terminal. The first time, it asks in the terminal whether to download Electron for its Chromium window (y, n for no and don't ask again, or Enter for not now); without it the same app opens in pywebview (The desktop app).
• Put your provider key in a .env file, or paste it into Settings in the app.
• Model names come from model_list.txt and are case sensitive. A name that is not on the list is forwarded verbatim and comes back as a 404.
• Everything the agent keeps, chats, keys, your own skills and browser profiles, lands in an AutoCua_data/ folder created where you run Python from.

From a checkout

git clone https://github.com/ComputerUseAgent/AutoCua.git
cd AutoCua

bash setup_mac.sh            # macOS
setup_windows.bat            # Windows

cp .env.example .env         # add your provider key
python main.py

main.py is the whole configuration surface: one run_agent(...) call. Edit it in place:

run_agent(
    mode="computer use",
    provider="anthropic",
    model="claude-sonnet-5",           # names come from model_list.txt
    task="check the version of macOS.",
)

Every optional flag goes straight into that call (ui=True, speed="fast", …). All of them, ui, save_conversation, speed, external_terminal, headless, browser_port, browser_profile, extra_tasks, device, ios_version, android_version, sim_device, app_build, api_key and os, are documented in agent_capabilities.md.

---

How it sees a screen

Every step the agent gets an element tree and a matching annotated image, then reasons over both.

macOS

1. Find the real front window. Window z-order comes from CGWindowListCopyWindowInfo, and the tree is walked per process from AXUIElementCreateApplication. The front app, any dialog owner, the Dock, Finder's desktop and menu-bar status-item owners are each walked separately, so a sheet on top of a window does not hide what owns it.
2. Wake the app up. AXEnhancedUserInterface is set when it is not already on, so apps publish their full tree, then the scanner waits 0.3 s for it to settle.
3. OCR whatever the tree missed. Apple Vision runs at the accurate level with a confidence floor of 0.4, on worker threads, while the accessibility walk is still running. Recognising at 1x logical resolution instead of 2x took a 2940 px capture from 1.17 s to 0.78 s with no loss in hit rate.
4. Drop what cannot be clicked. Every element is re-tested against the real window stack with a 20x20 grid of sample points. Anything under 1 percent visible is thrown away, so the model is never handed a number that sits behind another window.
5. Draw the numbers. The capture is downscaled first to fit inside full HD (1920 x 1080, aspect kept, never upscaled), then 13 px magenta labels with a 2 px black rim are stamped on from cached tiles, and the frame is encoded as a 4:4:4 JPEG at quality 85. That chroma choice is deliberate. Default subsampling smears thin magenta digits into mush, and this is a tenth of PNG's bytes.

Windows

The same contract, built out of what Windows offers.

One UI Automation cached round trip pulls 18 properties for the whole subtree, then four threads run in parallel: WinRT OCR, a taskbar walk, a Win32 backend scan and a raw UIA scan of the Start menu. The results are merged and deduped at 5 px, with OCR lines nested into the deepest container that does not already hold a labelled element there. Chromium browsers are launched with --force-renderer-accessibility so they publish their full tree, and page loads are gated by polling the toolbar reload button until it stops saying "Stop".

Clicks aim at the pixel centroid of the element's own content rather than the middle of its box, so a wide button with a label on the left still gets hit where the label is.

Both platforms

A step that only digests a tool result, the turn after a web lookup or after the coder agent reports back, deliberately skips the scan entirely. No screenshot, no tree, just the payload and an instruction to pull the findings into the scratchpad. Scanning a screen that nobody looked at costs tokens and buys nothing.

---

How it acts on a screen

Nothing in the desktop input path opens a debugging port, sets an automation flag or injects JavaScript. Input goes in at the operating system's own input layer, so an ordinary application sees the same events it would get from your hand on the mouse.

macOS

Mouse events are posted at kCGHIDEventTap, the same tap a physical mouse feeds, from a private event source, after a genuine cursor warp with CGWarpMouseCursorPosition.

• A partially visible element tries the accessibility AXPress action first, then falls back to a synthetic click. A fully hidden element is refused with "scroll first" instead of being clicked blind.
• Multi-line text is put on the clipboard and pasted, then the previous clipboard is restored, because code editors auto-indent typed newlines and corrupt the text.
• Drags are 20 interpolated move events at 15 ms, not a teleport.
• applescript is a first-class tool, not a bash escape hatch. The model supplies a complete tell application block and the runtime handles launching, foregrounding and a 30 second cap.

Typing and hotkeys go through pyautogui and pynput rather than the private event source, so they are ordinary synthetic input.

Windows

Element clicks use pywinauto.click_input(), which is a real SendInput. When that fails because User Interface Privilege Isolation blocked it, the click is retried through the Interception kernel-mode driver. Typing switches to the driver when the front application is Windows Security. See Windows, UAC and the kernel driver.

A watcher for permission dialogs

macOS asks for consent the first time an app does something new, and a modal dialog will deadlock a run. AutoCua fingerprints those dialogs structurally, any window or sheet carrying a "Don't Allow" or "Deny" button, and clicks the affirmative. It runs on a one second loop around every AppleScript call and every shell command, and if a dialog is present but cannot be clicked, the command fails after 8 seconds with a permission-specific error instead of hanging forever.

Browser windows on the desktop

When a browser is in front, the agent gets browser rules injected into its prompt and the runtime changes behaviour:

• It reuses the browser you already have open, with your profile, cookies and sessions. The rules are emphatic that launching a second instance lands in a signed-out profile.
• macOS waits for a real AXWebArea before scanning, with a 1.5 second grace period and a 15 second load budget. No web area means it is not a web window, so scan it as it is. Firefox publishes no load property, so it falls through to a children-ready check.
• Known destinations are opened in one command with the query already in the URL, rather than opening a site to click its search box.

Tools the desktop agent has

	macOS	Windows
Click	left_click (1, 2 or 3 clicks), right_click	same
Type	input, typewrite	same
Navigate	scroll, hotkey, screenshot, drag_drop	scroll, hotkey, screenshot
System	open_app, shell (zsh), applescript	open_app, shell (PowerShell)
Delegate	sub_agent (coder_agent, browser_agent, ios_agent, android_agent), agent_wait, web	sub_agent (coder_agent, browser_agent, android_agent), agent_wait, web
State	todo_list, update_todo, scratchpad, wait, done	same

19 tools on macOS, 17 on Windows. The names are a frozen allow-list. Anything else the model invents comes back as an error result and is never executed.

---

The agent team

Behind one task sits a hierarchy of agents, each in its own process.

The parent agent owns the screen and decides what a task needs. When it needs real code work it delegates and can carry on, or block until the result comes back.

The coder agent has 17 tools: two notes tools, insight and next_goal, then shell, read, grep, glob, write, replace, web, planning_mode, plan, todo_list, update_todo, wait, scratchpad, subagent and exit. Its operating procedure is EXPLORE, PLAN, EXECUTE, VERIFY, and it is written into the prompt with teeth:

• Exploration is delegated by rule. Workflow subagents explore, and the coder is told to "build first-time understanding from the reports, not from reading the codebase yourself."
• The plan is a structured Markdown document with real path:line anchors, not a restatement of the request.
• Every change with logic in it needs a throwaway test under ./.AutoCua_verify/ that is actually run, whose output is recorded, and which is deleted before exit.
• replace verifies the old block before writing and re-verifies after, so a stale line number fails with "mismatch at line X" instead of corrupting your file.

Subagents are read-only helpers with eight tools (insight, next_goal, shell, read, grep, glob, scratchpad, exit) and no recursion. Each runs in one of two modes: workflow maps the code (which file holds what at which lines, the callers, the dependencies) and critic judges an edit and tries to prove it wrong. The restriction on tool names is structural: the eight names are the allow-list a subagent's calls are checked against, so a hallucinated write comes back as an error result and never runs. Several subagents fire from one coder step, run in parallel, and the loop blocks until all of them report back with findings anchored to exact path:line. A subagent that fails still returns its partial scratchpad, and one that crashes comes back as an error with its last stderr lines, so the parent never hangs. To be precise about the boundary: the tool names are enforced, and the "read-only commands only" rule on its shell tool is held by the prompt.

Subagents exist to keep the coder's context small. Every tool result is budgeted too: 200 lines or 15,000 characters, with shell keeping a 50-line tail, read capped at 2,000 lines and grep at 8 MB, so one wide search or noisy build log can never blow up the conversation.

Run either directly. They start in their own workspace, so point them at an absolute path when the work is in an existing project:

python -m AutoCua.coder          --task "refactor the auth module in /Users/me/projects/api" --provider anthropic --model claude-sonnet-5
python -m AutoCua.coder.subagent --task "where is _validate_token defined in /Users/me/projects/api and who calls it?" --provider anthropic --model claude-sonnet-5 --mode workflow --result /tmp/scout.json

---

The browser agent

"web use" is a different program from the rest of the framework. The whole browser agent, the agent loop, the page scanner and the controller, is one Rust crate compiled into a single Python extension module and imported like any other. Its LLM calls go through the same Python providers every other mode uses, in AutoCua/llm_provider/.

A checkout builds it on first import with cargo build --release and rebuilds only when a .rs file actually changed. The pip wheel ships it already compiled. One binary covers every CPython from 3.10 up.

Two ways into Chrome. Every run with a window goes through the AutoCua extension, a parallel run included: Chrome is started with the flags that load it, or taken as it is when it is already running with it, and the agents drive your own browser, logged in as you are, with no debugging port open; each agent is on a line of its own to the extension. A headless run uses the DevTools Protocol on a Chrome of the agent's own. Every tool works the same way in both. Chrome is never downloaded: when it is missing, the terminal says so at each web run, the app shows a popup when it starts, and you install it from google.com/chrome.

Raw CDP, hand written

No Playwright, no Selenium, no Puppeteer, no chromedriver. None of them appear anywhere in the repository. In the debugging-port mode there is one WebSocket to Chrome plus hand-written HTTP for tab management, one session per tab and a child session per cross-origin iframe, attached in flatten mode up to six levels deep. In extension mode the same scanner runs as a content script of the extension, in an isolated world the page cannot see, and the tools send their input through chrome.debugger as the same trusted CDP events.

The property that matters:

The debugging-port scanner runs no JavaScript in the page, and Runtime.enable is never called in either mode.

In the debugging-port mode the page model is two CDP calls per frame, DOMSnapshot.captureSnapshot plus Accessibility.getFullAXTree; in extension mode it is one injection into every frame of the tab, cross-origin frames included, reading the same facts from the DOM. Clicks and keystrokes go through CDP's trusted input path with a rising click count. Even the numbered boxes the model sees are painted into the JPEG in Rust with a hand-rolled 3x5 bitmap font, never into the DOM.

Two honest exceptions, both deliberate:

• run_script and scrape are the two doors for JavaScript in the page, and they are split by what they are for: run_script runs what the model wrote, to DO something the ordinary tools cannot reach; scrape runs a reader we wrote, which takes no selectors from the model and only reads. Both are raced against a 10 second in-page timer and serialised in page, run_script is parse-probed with new Function and capped at 30,000 characters, and both are screened by the same guardrails_check.rs: web pages only, theft and cross-site sends refused before anything runs, credential-shaped values taken out of the result. Both descriptions tell the model to use them sparingly, because automation blockers can notice them.
• A scrape reads the screen the page is on plus the next two below it: three viewports at most, whole records only, in one chunk of about 8k tokens. The reader finds the page's records itself, by shape and never by site: the block in the page's main content (a sidebar or a menu only when nothing else repeats), a custom element that repeats and carries its data in attributes, or the repeated block on plain HTML, with nesting or indentation read as thread depth; it falls back to the text in reading order on a page without records, so an article or a table reads too. A thread comes back as the post, then its comments with each reply nested under the comment it answers; a list as its items with title, url, score, counts, price, phone or address where the page shows them. After the call the agent is in scraping mode until it calls exit_scrape_mode: no scan, one clean picture per screen read, and a schema of six tools: index (what the agent keeps from the chunk: a one-line title, what is still pending, and the value as JSON in a shape of its own), more (the next three screens, scrolled to from where the last read ended; nothing is read twice), exit_scrape_mode, scratchpad, update_todo and skills. Every index is stored as JSON under AutoCua_data/scrapes/<run>/<n>.json with its number and the time. On exit the mode's steps leave the history and the scrape result becomes one line per index; <scraped_index> lists them on every step, and index_read brings an index back in full. The rules the model follows are in AutoCua/default_skills/web/skills/scraping.md.
• The "agent is driving" overlay, an edge glow and the agent's own cursor, is injected into the page (its main world in the debugging-port mode, the extension's own world in extension mode), which makes it detectable by a page that looks for it. That is a product trade-off in favour of you being able to see what the agent is doing.

The page model

• Settling counts in-flight requests rather than sleeping. 150 ms of quiet, capped at 3 seconds.
• Occlusion by paint order demotes elements behind cookie banners and modals.
• Noise filtering collapses the wrappers and SVGs inside a button that would otherwise stack four boxes on the same pixels.
• [1] is always the page itself. Numbering follows document order and stops at 300 elements.
• Per-host overrides are shipped, for example YouTube turns off pointer-cursor detection and caps at 200 elements.
• Modal dialogs are shown to the model as a step of their own, with their text, for it to answer. A tab whose page froze for 5 seconds or crashed is closed and its page opened again in its place. The Chrome Web Store is off limits: the tools refuse to open it or act on it. Navigation failures, end of history and stale ids all come back as recoverable tool errors rather than silent successes.

Staying logged in

A run with a window is in your own Chrome, so the agent is logged in wherever you are. Headless runs use a persistent profile keyed by name, so cookies, localStorage and logins survive between runs there too. As the source puts it, arriving already logged in matters far more than any fingerprint tuning, because a login wall is where a web agent usually stops.

18 tools: new_tab, switch_tab, close_tab, update_tab, navigate_tab, click, hold_click, input, keyboard, run_script, scroll, wait, scratchpad, todo_list, update_todo, skills, index_read, done, plus dialog on a step a popup holds. Scraping mode sends exactly six: index, more, exit_scrape_mode, scratchpad, update_todo, skills. A parallel agent pinned to one tab gets 15, with the three tab-lifecycle tools removed from its registry and refused at the router as a backstop.

---

iPhone and iPad

The same loop that drives your Mac drives a phone. Read the accessibility tree, annotate a screenshot, tap, swipe and type through WebDriverAgent. The model is never told which target it is on, because both paths end at the same endpoints.

	device="simulation" (default)	device="hardware"
Runs on	An iOS Simulator on your Mac	Your paired iPhone or iPad
Driven by	xcrun simctl plus one unsigned xcodebuild test	pymobiledevice3, USB forward plus XCUITest launch
Needs	Full Xcode. No Apple ID, no signing, no pairing	One-time pairing and Team ID signing
iOS version	ios_version="26.5", or the newest your Xcode can build for	Whatever the device runs
Your own app build	app_build="MyApp-Simulator.ipa", installed before the task starts	Not supported, the run stops with an error
Best for	Everyday runs, testing, pinning a specific iOS version	Real apps with your logins, camera, cellular

run_agent(
    mode="mobile use, ios",
    provider="anthropic", model="claude-sonnet-5",
    device="simulation",        # or "hardware"
    ios_version=None,           # e.g. "26.5"; None picks the newest usable runtime
    sim_device="iphone",        # "iphone", "ipad", or an exact simulator name
    app_build=None,             # simulation only: a .ipa/.app in AutoCua_data/app_build/
    task="Open Settings, turn on Dark Mode, and confirm it on the home screen.",
)

AutoCua boots the simulator, starts WebDriverAgent, runs the task, then shuts everything down when the agent finishes, whether it succeeded, errored or you pressed Ctrl+C. The first run compiles WebDriverAgent once. After that it starts in well under a minute.

Testing your own app. app_build works with device="simulation" only. Put a simulator build (.ipa or .app) in AutoCua_data/app_build/ and pass its file name. It is installed the moment the simulator has booted, at the same time as WebDriverAgent starts, and the agent only begins once both are done. No app, no run: a missing file, a failed install or device="hardware" stops the run with a clear error before the agent starts. Details in agent_capabilities.md.

Three details worth knowing.

What simctl can boot is not the same as what xcodebuild can build for. Simulator runtimes are system wide and outlive Xcode upgrades, so AutoCua probes xcodebuild -showdestinations up front and filters to what is actually usable, instead of dying minutes into a run. If a parallel run needs one more device than you have, it creates one.

The element scan asks WebDriverAgent to skip its visible and accessible attributes, which it computes by hit-testing every element. On a 364-element home screen that takes the request from 3.60 s to 0.39 s, about 82 percent of the time. Visibility is recomputed from the element's centre, which is the exact point the tap will land.

There is a video_player tool because DRM players black out screenshots. When vision goes dark it drives play, pause, close and streaming checks from the accessibility tree, and proves "streaming" by watching the progress label advance across a four second window.

Over Wi-Fi, in the app. Mobile use and Computer use have a Wireless / Wired toggle. Wired, the default, is the cable, as in the table; Wireless you pick on purpose. It needs nothing beyond the pairing: Xcode already reaches a paired iPhone on the same Wi-Fi, so it builds and starts WebDriverAgent there, with the same xcodebuild run Settings makes, which also signs it again when a free profile has expired. The app then talks to it at the phone's Wi-Fi address. pymobiledevice3, the cable's transport, would need root or an extra setting on the phone for Wi-Fi. A start takes about 15 to 35 s, against about 5 s over the cable. With several paired iPhones on the Wi-Fi it takes an unlocked one first, the newest of those, and says so at once when they are all locked. In Computer use, Daisy chain, the same toggle says over what ios_agent looks for the iPhone at the start of each request: the cable only on Wired, the Wi-Fi only on Wireless; This PC looks for no phone. A run from a script (run_agent with "computer use, daisy chain") always uses the cable.

15 tools: open_app, click, ground_click, ground_scroll, input, scroll, wait, shell, web, vault, video_player, todo_list, update_todo, scratchpad, done. ground_click taps and ground_scroll swipes from a raw point of the screenshot (x and y on a 0 to 1000 scale), for a target that has no number.

---

Android

The same loop drives an Android emulator, or a real Android phone, plugged in by USB or, from the app, over Wi-Fi once you connected it in Settings. Nothing is installed on the phone. While AutoCua uses it, one helper runs there, in a single app_process started over adb. It is AutoCua's own Helper.java: it reads the accessibility tree through UiAutomation and keeps that connection open, so there is none of uiautomator dump's one-second wait for a still screen, and it takes each screenshot through the same connection, compressed to JPEG on the phone, so a few hundred KB cross the cable instead of 10 MB of pixels. The model gets the iPhone's view, the same numbered elements and annotated screenshot, and taps, swipes and keys go through Android's own input tool.

	device="simulation" (default)	device="hardware"
Runs on	Google's Android emulator on your computer	Your Android phone
Driven by	The emulator, on a virtual device AutoCua writes itself: no Java, no Android Studio	adb, over USB or, from the app, Wi-Fi (below)
Needs	The computer's hypervisor: built into macOS, Windows Hypervisor Platform on Windows, KVM on Linux	USB debugging on, and Allow tapped
Android version	android_version="16", from 11 to 17, or the newest one already here	Whatever the phone runs
Phone or tablet	sim_device="phone" (default) or "tablet"	Whatever it is
Your own app build	app_build="MyApp.apk", installed before the task starts	Not supported, the run stops with an error
Best for	Testing your app, any Android version, parallel tasks	Your own apps and logins, Play Store, camera, cellular

run_agent(
    mode="mobile use, android",
    provider="anthropic", model="claude-sonnet-5",
    device="simulation",        # or "hardware" for your phone
    android_version=None,       # e.g. "15"; None picks the newest one already here
    sim_device="phone",         # "phone" or "tablet"
    app_build=None,             # simulation only: an .apk in AutoCua_data/app_build/
    task="Open Settings and tell me the Android version.",
)

The emulator. Every run is a fresh phone: Google's apps, your .apk if you gave one, and nothing from the last run, the way a tester wants it. The first run of an Android version does its one first boot (15 to 30 s on Apple Silicon) and saves that moment; every run after starts a read-only copy from it in about 5 s (12 s when several start at once) and stops in a second. The emulator (about 420 MB) and the Android it needs (1.2 to 2.1 GB, 4 to 6 GB unpacked) are Google's, fetched from Google's SDK repository the first time they are needed, pinned by size and SHA-1, into a per-user folder: ~/Library/Caches/AutoCua/android-sdk, %LOCALAPPDATA%\AutoCua\android-sdk or ~/.cache/AutoCua/android-sdk. Before that first download AutoCua asks you once, on the terminal, to accept Google's Android SDK terms, as Android Studio does; nothing of Google's is in this repository or the wheel. python -m AutoCua.android_connector.emulator does all of that ahead of time, with a check of what the computer has. An Android Studio SDK that already has the emulator and the image is used as it is. The images are Google APIs ones: Google's apps, no Play Store. Android 10 is not offered yet (not tried with the helper's screenshots). The emulator draws no window of its own: the run shows its screen in AutoCua's own window, the one iOS Simulator runs use, one device in the middle or several side by side. Details in agent_capabilities.md, Part 10.

A real phone goes over USB or, from the app, over Wi-Fi:

	USB	Wi-Fi
Needs	USB debugging on, and Allow tapped with "Always allow from this computer" ticked	The phone connected once over USB in Settings, Connect Device, Android. Then Android 11 or newer, Wireless debugging on, and the same Wi-Fi as your computer. No pairing code: the phone takes the key it already trusts from USB
Which phone	The one plugged in, or the one Settings connected	Only phones you connected in the app's Settings, found by their USB serial in the name they announce on the network
Where	Everywhere, a script included	The desktop app only, when you pick Wireless on the Wireless / Wired toggle, in Mobile use or in Computer use's Daisy chain (there for android_agent)
Best for	Everything	A phone on your desk

With device="hardware" a script uses USB only. With several phones plugged in it takes the newest one you connected in Settings, or AutoCua_ANDROID_SERIAL names the one to use; an emulator is never taken for a phone. In the app, Mobile use follows its Wireless / Wired toggle: Wired, the default, is USB only, and Wireless, picked on purpose, reaches the phone over Wi-Fi only, also with its cable in. Over Wi-Fi the phone must report the serial it was saved with once it connects, or it is dropped again. Android is tested on a Mac host with Android 10 and Android 16 phones so far, and the emulator with Android 11 to 17 on an Apple Silicon Mac.

Four details worth knowing.

Screens an app marks secure (banking apps, some password screens, Chrome's incognito tabs) come out black in the screenshot; the tree still lists their elements.

Android's "visible to user" ignores views drawn on top, so the scanner hit-tests each element against the tappable ones drawn over it, from the tree's drawing order. Keys hidden under a calculator's number pad are listed without a number, with the element that covers them, so they cannot be tapped by mistake.

Wireless debugging turns itself off when the phone changes Wi-Fi or access point, and often after a restart, and its port changes every time. Turn it on again, or plug the phone in and pick Wired. On a Mac, the app that starts adb needs Local Network access, under Privacy and Security; without it a Wi-Fi connect fails with "No route to host", and AutoCua says where to allow it.

Typing goes through input text, which takes ASCII only for now. The vault fills any text, through the helper.

16 tools: open_app, click, ground_click, ground_scroll, input, scroll, back, wait, shell, web, vault, video_player, todo_list, update_todo, scratchpad, done. back is the system Back button; ground_click and ground_scroll work on a raw point of the screenshot, as on the iPhone. video_player tells streaming from paused by watching the picture move. Unlike the iPhone's, it does not work on a DRM player, which is black in the phone's screenshots.

In computer use, the main agent can hand phone work to an android_agent sub-agent, on macOS, Windows and Linux, the way it hands iPhone work to ios_agent on a Mac. That is Computer use, Daisy chain in the app, or "computer use, daisy chain" from a script; This PC (plain "computer use") lists the phone sub-agents offline. It shows up once adb is on the computer: after one Android run or the Settings check, or with an adb on your PATH. From a script it uses the cable, never Wi-Fi; in the app it looks for the phone over what the Wireless / Wired toggle says at the start of each request: the cable only on Wired, the default, or the Wi-Fi only on Wireless, a phone connected once in Settings with Wireless debugging on. In the app, Settings, Connect Device, Android checks a phone once and remembers it, and Mobile use, Android in the chat box starts it for the run.

---

Windows, UAC and the kernel driver

Windows UAC runs on a separate secure desktop. User-mode SendInput cannot reach it and screen capture fails there too. Most automation tools simply hang at an elevation prompt.

AutoCua turns that failure into the detector, then answers from kernel space.

1. Detect. The full-screen grab fails, so the scanner reports that UAC is up.
2. Ask the model. That step ships with no screenshot and no element tree. Just a one-shot prompt: "A Windows UAC prompt is blocking the screen. Based on your previous actions, do you want to allow this?" The agent answers alt+y or alt+n. Elevation is never automatic. It is a decision the model has to make in context.
3. Inject. Those two combinations are intercepted and sent as raw scancodes through the Interception kernel-mode input filter driver, which the secure desktop does accept.

The driver is also the fallback whenever User Interface Privilege Isolation blocks ordinary input, for example when driving Windows Security.

How it is installed. You install it yourself, from its author (steps). AutoCua never downloads it, and setup_windows.bat only reminds you that it exists. The bind step puts the driver on the built-in keyboard and mouse only, at the device level. That detail matters. The driver has ten keyboard slots that are never freed, so the installer's own class-wide binding burns one on every wireless-keyboard reconnect and eventually leaves keyboards dead until reboot. Device-level binding takes exactly one slot at boot and never filters another keyboard.

Without the driver everything still works except three things: answering a UAC prompt, a click that User Interface Privilege Isolation blocked, and typing into Windows Security.

Licensing. Interception is dual licensed. LGPL v3.0 for non-commercial use, and commercial use requires a separate paid licence from its author. AutoCua's MIT licence does not and cannot grant it. See THIRD_PARTY_NOTICES.md.

macOS has no UAC and AutoCua does not elevate on it. The analogue is TCC, handled entirely in user space by the permission wizard and the consent-dialog watcher. There is no sudo anywhere in the agent paths.

---

Automation testing and workflows

A task here is a sentence, not a selector. Nothing in it names an XPath, a CSS class or a coordinate, so the redesign that breaks a selector-based suite does not break the task. The agent finds the button again because it reads the screen the way a person does.

What that buys you.

Need	How AutoCua covers it
End-to-end across surfaces	One harness covers a desktop app, a website, an iPhone and an Android phone. Change one string to move between them
A device or browser matrix	extra_tasks=[...] runs N tasks at once. One Chrome with a tab each, or one simulator or emulator each
Unattended runs	headless=True for Chrome. Desktop, iOS and Android emulator runs still need a logged-in graphical session
An audit trail	save_conversation=True writes exactly what the model saw and answered at every step
A result you can branch on	run_agent returns a status dict, and every parallel child writes its own result.json
Credentials in a test	On the web, iOS and Android the vault fills the secret. The value never enters the model's context
Site or app specific rules	Skills are Markdown files loaded only when that site or app is in front
Long flows	Memory compression keeps a long checkout or onboarding flow inside the context window

A smoke test you can run today.

from AutoCua.agent_launcher import run_agent

result = run_agent(
    mode="web use", provider="anthropic", model="claude-sonnet-5", headless=True,
    save_conversation=True,
    task="""
    Go to staging.example.com, sign in as [email protected], add any item to the basket,
    go to checkout and stop at the payment step. Report the basket total and whether the
    payment form rendered. Do not submit a payment.
    """,
)

assert result["status"] == "success", result["message"]

run_agent returns {"status", "message"}, where status is success when the agent called done, error when the loop died, and incomplete when it was stopped or ran out of steps. That is the hook you branch on.

What you get back to read afterwards. With save_conversation=True, conversation/ holds the exact payload sent to the model at every step and raw_reasoning/ holds its raw reply, so a failed run is readable rather than guessed at. A parallel run additionally writes result.json per child and exits 0 when it returned, 1 on a crash and 130 on Ctrl+C.

A run with save_conversation=True clears those folders when it starts, and a run without it leaves them alone. Copy out anything you need to keep before you run with the flag again.

One design decision matters more than any of this for reliability: the agent is told that a tool result reports that a tool ran, never that the screen changed. Every step judges the previous step's expected outcome against the new screenshot before it does anything else.

A cross-surface matrix in one call.

run_agent(
    mode="web use", provider="openrouter", model="gemini-3.8-flash", headless=True,
    task="Sign up for a new account on staging.example.com and confirm the welcome screen.",
    extra_tasks=[
        "Reset the password for [email protected] and confirm the email prompt appears.",
        "Open the pricing page and check every plan card shows a price and a CTA button.",
    ],
)

Each task gets its own tab, its own agent and its own working directory under ./parallel/task_N/, with a result.json holding status and message. Live output is prefixed per task, and one Ctrl+C stops all of them.

Everyday workflow automation is the same machinery pointed at your own machine:

task = "Open Numbers, put last month's totals from ~/Downloads/report.csv into a new sheet, chart them, and save it to the Desktop as monthly.numbers"
task = "Go through my Gmail inbox, find every invoice from this month, and save the PDFs into ~/Documents/Invoices"
task = "Check the three competitor pricing pages in my bookmarks and tell me what changed since the notes in ~/Desktop/pricing.md"

Be honest about what this is

AutoCua is a language model driving real software. Two runs of the same task can take different routes. It is a strong fit for exploratory testing, smoke flows, repetitive back-office work and anything where writing selectors costs more than the test is worth. It is not a replacement for a deterministic unit or integration suite, and it has no assertion framework of its own. Use it where a human tester would otherwise be clicking.

---

Running tasks in parallel

Pass extra_tasks and every task, including the first, runs at the same time in its own child process.

run_agent(
    mode="web use", provider="anthropic", model="claude-sonnet-5", headless=True,
    task="find the cheapest flight to Tokyo next month",
    extra_tasks=[
        "summarise today's top Hacker News thread",
        "check my GitHub notifications",
    ],
)

	"web use"	"mobile use, ios" with device="simulation"	"mobile use, android" with device="simulation"
Each task gets	Its own agent pinned to its own tab	Its own simulator and its own WebDriverAgent port	Its own copy of the emulator, read-only
Shared	Your own Chrome, through the extension (headless: one browser of the agent's own on a debugging port); its front tab shows a cover page while the agents work in tabs behind it	Nothing. A phone screen cannot be split	The saved first boot, which every copy starts from and none changes
Isolation	A 13-tool single-tab registry. It cannot touch another agent's tab	Own port (8100, 8101, ...), own scratchpad, own build directory	Own emulator (emulator-5554, emulator-5556, ...), own scratchpad
Ceiling	Tabs and RAM	Simulators, and it creates one when it needs another	16 emulators, and RAM: 2 to 4 GB each

Backgrounded tabs are told they are still focused, so pages that gate on focus keep behaving while another agent is in front. Output is prefixed per task, one Ctrl+C stops everything, and results land in ./parallel/task_N/. On iOS every simulator the run booted is shut down at the end, whether it succeeded, failed or was interrupted, and on Android every emulator copy. With app_build, every simulator or emulator installs your app at the same time, each as soon as it has booted. Emulator copies show side by side in the same window as the iOS simulators do.

./parallel/ is deleted and recreated on every parallel run. Copy out anything you want to keep.

Parallel mode is for run_agent. The desktop app runs one agent at a time.

---

The desktop app

ui=True opens a native window, 1140 by 700, on a Flask server at AutoCua's own port (27321, or a free one for that run when another program holds it). The window is Chromium, through Electron, the same on every OS. Electron is not in the wheel, and nothing is downloaded without asking: on macOS 13 or newer, Windows 10 or newer, and 64-bit Linux, the first launch from a terminal asks whether to fetch it, and says how big it is and where it goes (a 120 to 160 MB download from Electron's own GitHub releases, the same zip npm install electron fetches, checked against a pinned SHA-256, and 300 to 390 MB on disk). y puts it in a per-user cache (~/Library/Caches/AutoCua/electron, %LOCALAPPDATA%\AutoCua\electron, ~/.cache/AutoCua/electron); no Node.js is needed. n is remembered and the question is not asked again; Enter, Ctrl+C, or no answer within two minutes means not now. With no terminal (an IDE's run button) or a hidden one, nothing is asked. python -m AutoCua.utils.electron fetches it any time (--force fetches it again). Deleting the cache folder removes it. ELECTRON_MIRROR and ELECTRON_CUSTOM_DIR fetch from a mirror, as they do for npm, and the question names it. A checkout that ran npm install in AutoCua/desktop (Node.js 22.12 or newer) uses that copy and is never asked.

Without Electron, or with AUTOCUA_UI=webview, the app opens in pywebview, with the OS title bar tinted to match the page so the whole surface reads as one. So does an Electron that cannot run here (macOS older than 13) or that fails before the app is up (a missing system library, a policy that blocks it). While the Chromium window is open, a second launch just brings it to the front.

On Linux, Chromium's sandbox needs a user namespace it can use, or its setuid helper, chrome-sandbox next to the executable, owned by root. Ubuntu 24.04 and newer (23.10 as an opt-in) take every capability away in the user namespaces of a program with no AppArmor profile (kernel.apparmor_restrict_unprivileged_userns=1), and a download made as you cannot make the helper root's, so there the Chromium window starts without the sandbox (--no-sandbox, for this window only) and the launch says so. The pywebview window has no sandbox there either: WebKitGTK's is off in it. To turn the sandbox on, make the helper root's, as Chrome's and Cursor's packages do, with the two commands the launch prints:

sudo chown root:root ~/.cache/AutoCua/electron/v<version>-linux-<arch>/chrome-sandbox
sudo chmod 4755 ~/.cache/AutoCua/electron/v<version>-linux-<arch>/chrome-sandbox

An update that brings a new Electron needs them again. An AppArmor profile that allows user namespaces for this Electron is not the way: it runs any app folder it is given, so the profile would let every program on the machine around Ubuntu's restriction, and everything the agent runs would inherit it.

+--------------+----------------------------------------------+
|  AutoCua     |  live agent screenshot  |  tracking progress |
|              |  (what the agent sees)  |  (scratchpad notes)|
|  + New chat  +-------------------------+--------------------+
|              |  tool-response chain    |  live TODO list    |
|  chat 1      |  "N tools used"         |  (agent's plan)    |
|  chat 2      +-------------------------+--------------------+
|  chat 3      |   Agent Notes  or  Skills   (big centre)     |
|              +----------------------------------------------+
|  settings    |   composer  [fast] [mode] [model] [skills]   |
+--------------+----------------------------------------------+

Element	What it does
Live screenshot	The screen the agent just captured, updated every step
Tracking progress	Each scratchpad entry streams in as a bullet on a connecting line
Tool-response chain	Every tool call drawn as an animated canvas icon, so you can see why it did something
Live TODO	The agent's own plan, tailed from its file as it edits it, frozen with crosses if you stop it
Agent Notes	The final write-up, shown when a run ends, completed or stopped
Skills	Browse, preview, add, edit and delete domain knowledge files, live, for desktop and iOS
Memory ring	Context gauge beside the Skills icon, with the live token count inside. It breathes while compression runs and its count falls when the handoff lands
Fast / Quality	Leaner prompt and fewer tokens per step, or full reasoning, which is the default
Mode picker	Computer use (This PC, or Daisy chain to reach the phones: picking it pairs them at once, the chat box held until each one answers, then a line such as iPhone online · Android offline), Mobile use, Shell use
Wireless / Wired	Mobile use and Computer use: the phone over the cable, the default, or over Wi-Fi when you pick Wireless; its label reads Mobile via: Wired or Mobile via: Wi-Fi. In Mobile use, while it pairs, a circle turns beside the picker and the chat box is locked; if pairing fails, the box keeps the reason and the circle turns into an arrow that pairs again. In Computer use it says over what android_agent and ios_agent look for the phone at each request, and can be flipped any time, also while a run goes; it shows in Daisy chain only, as This PC looks for no phone
Stop	Retires the run id instantly, so a still-running model call can never repaint your next chat

Chats are saved and resumable. R