computer-use-linux

computer-use-linux gives AI agents real, structured control of a local Linux desktop: semantic AT-SPI accessibility trees, bounded screenshots, compositor-aware window targeting, Wayland/X11 input synthesis, and single-round-trip post-action verification across GNOME, KDE Plasma/KWin, Hyprland, niri, COSMIC, i3, Sway, labwc and other wlroots compositors, and X11/EWMH, plus a click-through on-screen activity indicator on GNOME and on Wayland compositors with wlr-layer-shell.

Quick start

Pick the setup that matches your agent or shell workflow:

Host / workflow	Install command
Claude Code (plugin + skill)	claude plugin marketplace add agent-sh/computer-use-linux && claude plugin install computer-use-linux@computer-use-linux
Codex CLI (plugin + skill)	codex plugin marketplace add agent-sh/computer-use-linux && codex plugin add computer-use-linux@computer-use-linux
Pi Coding Agent (native lazy-loaded tools)	pi install npm:@agent-sh/computer-use-linux
Hermes Agent (CLI + skill + MCP server)	npm install -g @agent-sh/computer-use-linux && hermes skills tap add agent-sh/computer-use-linux && hermes skills install agent-sh/computer-use-linux/computer-use-linux && hermes mcp add computer-use-linux --command computer-use-linux --args mcp
CLI + any stdio MCP client (npm)	npm install -g @agent-sh/computer-use-linux
CLI + system setup from source	git clone https://github.com/agent-sh/computer-use-linux && cd computer-use-linux && ./install.sh
Rust crate (crates.io)	cargo install computer-use-linux

Verify desktop readiness from the shell (after installing the CLI on PATH via npm install -g @agent-sh/computer-use-linux, ./install.sh, or cargo install computer-use-linux):

computer-use-linux doctor | jq .readiness
computer-use-linux setup                      # enable AT-SPI accessibility bridge if needed
computer-use-linux setup-window-targeting     # install GNOME Wayland window-control extension if needed

Plugin-only or Pi-only installs: The Claude Code / Codex plugin launcher and pi install manage private cached binaries and do not add computer-use-linux to PATH. Either run the MCP doctor / setup_accessibility / setup_window_targeting tools (or computer_use_linux_doctor in Pi) directly from your agent, or also install the CLI above when you want shell subcommands on PATH.

The Rust crate is published as computer-use-linux and the npm package as @agent-sh/computer-use-linux. Prebuilt x86_64 and aarch64 Linux binaries ship with the latest release. They need glibc 2.28 or newer: RHEL 8 and its rebuilds, Debian 10, Ubuntu 20.04, and anything later. On an older system, build from source with cargo install computer-use-linux.

Need an isolated desktop where the agent never touches your real mouse, keyboard, or focused window? Use the sibling project agent-workspace-linux alongside this one.

What this is

computer-use-linux is a Rust MCP server and CLI for Linux desktop control. It ships three binaries: the main computer-use-linux server/CLI, the computer-use-linux-cosmic Wayland toplevel helper for COSMIC, and the computer-use-linux-indicator click-through activity overlay. Any MCP host (Claude Code, Codex CLI, Codex Desktop's Linux build, Pi Coding Agent, Hermes Agent, Claude Desktop, or your own client) can spawn it to inspect accessibility trees, list and focus windows, take screenshots, click, drag, scroll, type, and invoke semantic accessibility actions.

Most computer-use MCP servers are macOS-only (they lean on AppKit, AXUIElement, CGEvent). The few that target Linux either drive xdotool against an X11 root window or shell out to OCR over screenshots. It differs in five ways:

• Wayland actually works. On Wayland, pointer actions use /dev/uinput or org.freedesktop.portal.RemoteDesktop. For pointer actions ydotool only sends wheel events there, and a coordinate action with neither route is refused. Literal text prefers KDE clipboard integration on Plasma, wtype on compatible Wayland compositors, and portal keysyms on other Wayland sessions, including GNOME with ydotool installed. Portal text startup or conversion errors fail the call without replaying text through ydotool. Raw ydotool text accepts printable ASCII, tab and newline only, and its physical US key positions can produce different text under the active layout. Screenshots use the GNOME Shell DBus screenshot method when present, org.freedesktop.portal.Screenshot otherwise, then on a native X11 session a root-window GetImage (device pixels, no toolkit scaling), and finally spawn gnome-screenshot for background/systemd contexts where the DBus paths are denied.
• Window targeting is compositor-aware. The window registry tries GNOME Shell extension, GNOME Shell Introspect, COSMIC Wayland helper, KWin DBus scripting, Hyprland hyprctl, the niri IPC socket, i3 IPC, labwc's labwc.sock, wlr-foreign-toplevel-management, and generic X11/EWMH in order, then reports exactly which backend won or why each backend failed. Terminal windows are enriched with their controlling TTY and foreground process so agents can target "the terminal running pytest" directly. Every window_id fits JavaScript's exact integer range (at most 2^53-1), so JSON clients can pass it back unchanged; a larger value is rejected with a prompt to re-run list_windows.
• Semantic selectors first, pixel coordinates when needed. Tools like click, perform_action, and set_value accept role / name / text / states selectors or element_index values backed by AT-SPI, and every mutating tool can return a post-action screenshot and updated accessibility tree in the same round trip. Pixel coordinates remain available for rendering-only surfaces (canvas, games, X clients without ATK).
• Visible to humans, invisible to screenshots. While an agent acts, computer-use-linux-indicator renders a click-through overlay with a per-agent cursor glide, click ripple, keycaps/typing banner, screen-edge glow, and status pill, and hides automatically under a shared capture lock before any screenshot is read.
• One JSON readiness report. computer-use-linux doctor returns a structured document covering platform, portals, AT-SPI, windowing, input, and a readiness summary with explicit blockers and a recommended next step. MCP hosts can render or surface that to the user without parsing prose.

The crate was extracted from codex-desktop-linux (the Linux distribution of Codex Desktop), which bundles this backend for its native Computer Use feature. This standalone repo is the upstream.

MCP tools & features

Category	Tool	What it does
Diagnostics & setup	doctor	Single-shot JSON readiness report (platform, portals, accessibility, windowing, input, readiness summary, and capability map).
	setup_accessibility	Enables and verifies GNOME's org.gnome.desktop.interface toolkit-accessibility setting so toolkit apps expose AT-SPI trees.
	setup_window_targeting	Installs and enables the bundled GNOME Shell extension. GNOME Wayland needs it for exact window activation and move/resize; every GNOME session needs it for the on-screen indicator.
Discovery & observation	get_app_state	Combined screenshot + compacted AT-SPI accessibility tree with stable element_index handles. Scope with app_name_or_bundle_identifier or a window target (window_id, pid, app_id, wm_class, title).
	get_tree_page	Page through the latest cached tree by node offset, for trees too large to return inline when the host cannot read the spill file.
	screenshot	Bounded PNG or JPEG capture of the full desktop (full_screen: true or no target) or a raised and cropped target window (window_id, pid, app_id, wm_class, title, raise_window).
	list_windows	Compositor windows with window_id, title, app_id, wm_class, PID, focus state, client type (wayland/x11), bounds, and terminal TTY/process metadata.
	focused_window	The window currently holding keyboard focus.
	list_apps	Running desktop applications visible to the AT-SPI registry.
Input	click	Click by element_index, semantic selector (role/name/text/states), or desktop/window-relative coordinates (left/right/middle, click_count).
	drag	Desktop coordinate drag from (start_x, start_y) to (end_x, end_y).
	scroll	Page-based vertical/horizontal scroll on an element, window, or pixel coordinate.
	press_key	Single keys or chords (Ctrl+L, Alt+F4, Return), optionally focusing a target window or terminal first.
	type_text	Literal text entry, optionally targeted at a window or terminal (tty, terminal_pid, terminal_command, terminal_cwd).
Semantic actions	perform_action	Invoke any AT-SPI action exposed by an element (Press, Activate, Toggle, Jump, …); defaults to the primary action.
	set_value	Write directly to a settable accessibility element (text fields, sliders, spinners).
Window management	activate_window	Focus a window by window_id, pid, app_id, wm_class, title, or terminal selectors.
	move_window / resize_window	Reposition or resize a window in desktop coordinates (GNOME Shell extension and X11/EWMH backends).
Opt-in host tools	complete_interaction	Parameter-free desktop completion notification (notify-send); registered only when COMPUTER_USE_LINUX_NOTIFY_ON_COMPLETE=1.
	run_shell	Bounded non-login /bin/sh -c execution; registered only when COMPUTER_USE_LINUX_ENABLE_SHELL=1 (absent by default).

click, scroll, perform_action, and set_value take an optional app target (pid, window_id, app_id, wm_class, window_title). With one, an element_index or selector from another app's get_app_state snapshot is refused; call get_app_state for the target app first. When the window backend reports no pid for a target window (i3 without xprop, COSMIC, other wlroots compositors), the snapshot must have been taken for that window, or the call is refused. perform_action and set_value do not focus the target window. An element_identifier is refused when its owner on the accessibility bus is a different pid than the target; it is not checked when either pid is unknown.

Post-action observation in one round trip

All mutating tools (click, drag, scroll, press_key, type_text, perform_action, set_value, activate_window, move_window, and resize_window) accept optional post-action observation parameters so an action and its verification complete in a single MCP tool call:

• screenshot_after: true | "screen" | "window" | false: capture after the action is dispatched and the indicator overlay is hidden (true or "screen" for full desktop, "window" cropped to the action's target window or the focused window when untargeted)
• format, quality, scale, max_width, max_height, max_bytes: same screenshot payload controls as screenshot and get_app_state
• include_tree: true: include a fresh compacted AT-SPI accessibility tree and update cached element_index values

When post-action screenshot capture succeeds, the image is returned as the first MCP image content block alongside the JSON action summary and screenshot metadata. If the action succeeds (ok: true) and only the post-action screenshot or tree capture fails, ok remains true and the error is reported in screenshot_error / accessibility_error and appended to message.

Screenshot sizing & coordinate scaling

Screenshot payloads are size-bounded by default before they are returned to the MCP host: max 1920 px width/height and 2 MiB image bytes, with hard caps even when callers request more. Agents that need more detail can pass max_width, max_height, max_bytes, scale, format: "jpeg", or quality, preferably with a window target or crop. PNG remains the default; JPEG lets callers trade lossless pixels for a smaller payload before the byte cap forces further resizing. Returned screenshot metadata includes coordinate_width, coordinate_height, scale, format, and quality so callers can convert from a downscaled preview to desktop coordinate pixels.

Input & semantic targeting contracts

For a plain left click by element index or selector, the element's first AT-SPI action takes precedence over bounds when it is click, press, or toggle. Entry activate and slider jump must be requested explicitly with perform_action; they are never a substitute for a pointer click, even when bounds are missing. This avoids pointer conversion for GTK3 HiDPI extents and GTK4 zero-origin bounds when the element exposes such an action. The preference does not substitute an arbitrary action name for a coordinate click. Explicit x/y, right clicks, and double/multiple clicks retain pointer semantics. Re-check application state after either kind of activation.

On KDE Wayland, element clicks that need pointer dispatch match the cached AT-SPI frame to a coherent KWin window/output snapshot, then map the target separately to physical capture pixels (uinput) and logical desktop coordinates (portal). Client-local and decorated frames, actual decoration offsets, and nonzero or negative output origins are supported when the geometry and uniform output scale can be verified. Ambiguous windows, mixed or unknown scales, overlapping outputs, or stale pointer geometry fail before input. These clicks get no off-screen or monitor-gap warning, since KWin maps the point onto an output; the uinput route still reports a clamp.

Explicit x/y defaults to coordinate_space: "capture" (raw screenshot pixels). To click a global AT-SPI screen point, pass coordinate_space: "atspi" together with its element_index or semantic selector; KDE Wayland then applies the same native KWin decoration and scale conversion as an element-bounds click, including when the AT-SPI window frame is client-local. This mode requires both coordinates and element identity and cannot be combined with relative: true. Other calls keep their existing coordinate behavior.

For coordinate click or scroll with relative: true, select a target window and measure from its clipped screenshot crop origin. Divide preview x and y by the returned screenshot scale before passing them. These are not raw GDK surface or widget-local coordinates; decorations and clipping can change the origin. A missing window target is rejected.

X11 text fidelity: xdotool/XTEST can silently substitute symbols under non-US XKB layouts (reported with Brazilian br/ABNT2) or changing keymap state, even when type_text returns ok: true. Every successful xdotool text dispatch includes a layout warning. XKB layout replacement events, persistent mapping/group changes, and keyboard focus/grab transitions detected across a call return an error after dispatch, without replaying the text. Keyboard grabs can divert characters without changing the polled focused-window ID. If monitoring cannot start, xdotool still dispatches and the response explicitly warns that layout-change and keyboard interception detection were unavailable. PointerRoot focus is unsupported because pointer motion can redirect input without a detectable focus event. This also detects a setxkbmap layout changed and restored during one call. Temporary raw remaps or group switches restored within a call and dropped text with a stable map remain unverified. Re-observe the actual text before pressing Enter, submitting it, or treating it as delivered. Use include_tree: true for AT-SPI readback when the application exposes the text, or screenshot_after: true to inspect it. These observations do not automatically compare the field to the requested text. Do not retry blindly.

Targeted press_key/type_text results append focused-element feedback from AT-SPI (role, name, editable); type_text also warns when a concrete non-terminal focus is not editable. Terminals accept keyboard input without an EditableText interface; re-observe the contents to confirm input. Click/screenshot/input results warn when the target window or coordinate is partially or fully off-screen. A click, scroll or drag coordinate between monitors, where monitors of different sizes or offset rows leave part of the screenshot uncovered, gets a warning too: the server reads the monitor layout once per screenshot (from the portal session, KWin, the GNOME Shell extension, Hyprland, kscreen, Sway, COSMIC, niri or wlr-output-management, and from RandR on X11 only) and checks the point in screenshot pixels. Without a layout that fits the screenshot, only its outer rectangle is checked. The warning says what the route did. The portal sends nothing. X11 keeps the pointer on the screen, so a point off the screenshot lands at an edge. On X11 and the uinput pointer, a point between monitors lands at a monitor edge or on no visible pixel. A failed X11 click or scroll gets no warning, since nothing was sent. A failed drag gets none either, though it may have pressed and released before it failed. A drag checks its start and end points, and each warning names the endpoint. On the uinput pointer, a drag result also says where an off-screen endpoint was clamped to, as a click result does. Points along the path are not checked: the portal moves them onto the nearest shared monitor, and on X11 and uinput the drop goes to the end point. get_app_state returns a compact readiness block by default; pass verbose: true for the full diagnostics report. Tree nodes use a compact encoding: null, empty, zero and false fields are left out, containers do not carry the U+FFFC placeholder text Chromium gives them, and actions lists action names (pass one as perform_action's action). verbose: true returns the full encoding, with every field and action descriptions and keybindings. interactive_only: true returns only nodes with an action, editable text or a value, plus headings, status and live regions, windows and dialogs; containers are collapsed, so parent_index names the nearest kept ancestor. Its element_index values are the same as in the full tree. When title or window_id names one of several windows under one AT-SPI application root (Chromium puts every browser window under one), the tree covers that window plus the app's popups and dialogs, and drops its other named windows; message says so. It also reports tree_scoped (false when no app target narrowed the AT-SPI tree, with a warning in message) and accessibility_tree_truncated (true when the node, depth, read, or time budget stopped traversal with unread elements left). When a result would pass 32,000 bytes, the tree comes back as its leading nodes plus accessibility_tree_spill (path, node_count, bytes, inline_node_count). path is a private JSON file holding the whole tree with the same element_index values, in $XDG_RUNTIME_DIR/computer-use-linux/trees when that runtime directory is private, otherwise under ~/.cache (directories 0700, file 0600). Search the file instead of requesting the tree again; hosts that cannot read files page through the same tree with get_tree_page. The file lasts until the next snapshot or until the server exits; after a crash it is swept at the next start or after an hour.

An app_id may be a full desktop ID or its final dot-separated segment; matching is case-insensitive, ambiguous aliases are rejected, and an unresolved explicit app_id returns an empty scoped tree rather than the desktop.

set_value on editable text reads the full contents back after dispatch. verified: true means an exact match; mismatches and readback failures return ok: false without replaying the mutation. Password fields return verified: false because masked content cannot prove the write. Re-observe after any mismatch; the contents may have changed.

On X11, text input sends LF and CRLF line endings as Return, with one newline per CRLF pair.

On GNOME, a read-only keymap preflight rejects the whole string before input if a character is absent or requires a modifier level Mutter cannot synthesize. The background reader cannot establish the active layout group, so every configured group must support the text. This can reject text that works in one of several groups; use set_value on an editable field instead. The check requires libxkbcommon.so.0 at runtime and stops after a two-second keymap read deadline. It does not cover held modifiers or a keymap changed concurrently with typing.

On-screen indicator

While an agent acts, the server drives computer-use-linux-indicator, a click-through overlay that shows what it is doing: a software cursor tinted per agent that glides to each coordinate click, drag, or scroll target and ripples on click (AT-SPI actions such as perform_action, set_value, and element clicks show in the pill without moving it, since no pointer moves); keycaps for press_key and a typing line for type_text; a glow along every screen edge; and a status pill such as Claude is using your computer · click. Pointer actions wait 350 ms so the cursor lands before the real input. While screenshot or get_app_state captures the screen, the overlay draws nothing, so agents never see it: not their own, not one shown for another agent, and not one that another agent starts mid-capture. This holds with COMPUTER_USE_LINUX_INDICATOR=0 too, which only stops a server from showing its own actions. The capture waits for the compositor only when something was on screen. The agent name comes from the MCP client's clientInfo.

The server starts the indicator on the first action and it exits after ten idle minutes. It stays hidden between bursts of activity. Compositors with wlr-layer-shell use the shared-memory overlay. GNOME uses native Shell actors through the bundled extension. X11 sessions other than GNOME show no indicator. Run setup_window_targeting once, then log out and back in after installing or updating the extension. Both renderers use the same capture lock and acknowledgement protocol. Set COMPUTER_USE_LINUX_INDICATOR=0 to turn it off.

Screenshot processing uses one blocking worker per server. Captures enter one pipeline before reading pixels, so concurrent requests do not queue decoded full-screen images. A source can contain up to 134,217,728 pixels; serial processing bounds decoded-image memory while letting MCP answer other requests during resizing and encoding. Session discovery is reused after a successful probe and refreshed when session variables change. Failed discovery is retried.

Captures require the shared capture lock and, when an overlay is running, its acknowledgement that it has hidden. A missing or invalid acknowledgement, a stalled socket, or a failed lock returns a capture error. Concurrent captures keep separate holds; their acknowledgements are exchanged in order.

MCP safety contract

computer-use-linux is not a read-only data source. It can observe the local desktop and, when a mutating tool is called, can change real application state. The tools/list response includes MCP ToolAnnotations so hosts can surface this distinction before invocation:

Class	Tools	Contract
Read-only observation	doctor, list_apps, list_windows, focused_window, get_app_state, get_tree_page	readOnlyHint=true; may reveal app, window, accessibility, and screenshot contents. get_app_state may trigger the desktop screenshot portal prompt.
Local setup mutators	setup_accessibility, setup_window_targeting	readOnlyHint=false, destructiveHint=false, idempotentHint=true; modifies user desktop configuration by enabling accessibility or installing/enabling the GNOME window-targeting extension.
UI state mutators	activate_window, move_window, resize_window, scroll, screenshot	readOnlyHint=false, destructiveHint=false; changes focus, geometry, or scroll position in the live desktop, or raises a window to capture it.
Desktop action mutators	click, drag, press_key, type_text, perform_action, set_value	readOnlyHint=false, destructiveHint=true, openWorldHint=true; can trigger arbitrary actions in whatever local application is targeted.
Conditional notification	complete_interaction	Absent unless COMPUTER_USE_LINUX_NOTIFY_ON_COMPLETE=1; when enabled, readOnlyHint=false, destructiveHint=false, idempotentHint=false, openWorldHint=true. Sends a desktop notification.
Conditional host-code execution	run_shell	Absent unless COMPUTER_USE_LINUX_ENABLE_SHELL=1; when enabled, readOnlyHint=false, destructiveHint=true, idempotentHint=false, openWorldHint=true. Runs with the MCP server user's host permissions.

Annotations are safety hints, not an authorization system. MCP hosts should still ask the user before calls that could submit, delete, send, purchase, overwrite, or otherwise commit state.

run_shell is an explicit trust-boundary opt-in, not a restricted command runner. Enabling it grants an approved MCP call the same file and network authority as the user running the server. The tool clears the ambient environment and inherits only a small desktop/runtime allowlist (PATH, home/user/locale fields, display/session-bus fields); additional variables must be supplied in the visible call payload. Commands use a fixed non-login /bin/sh, an existing canonical working directory, a 30-second default / 120-second hard timeout, process-group cleanup, and stderr audit records keyed by the command SHA-256 rather than command text. Collected streams up to 8 MiB are returned with a 512 KiB per-stream response cap and truncation flag; exceeding 8 MiB on either stream fails the call without partial output. These controls bound accidental leakage and runaway work; they do not make arbitrary shell code safe.

The binary also exposes the same capabilities from the CLI for scripting and debugging:

computer-use-linux mcp                                  # stdio MCP server
computer-use-linux doctor                               # JSON readiness report
computer-use-linux setup                                # enable AT-SPI
computer-use-linux guard-accessibility                  # explicit foreground GNOME accessibility guard
computer-use-linux setup-window-targeting               # install GNOME Shell extension
computer-use-linux apps
computer-use-linux state [APP_NAME]
computer-use-linux screenshot                           # JSON screenshot summary
computer-use-linux windows

Support matrix

Validated manually on Ubuntu 25.10 (GNOME Shell 50.1, Wayland). niri window listing and exact focus were also validated on Ubuntu 26.10 (development), niri 26.04, with a single 2x output, through both CLI and direct IPC. Other compositor backends are implemented and covered by parser / contract tests, but real desktop behavior still depends on each session exposing its expected control API.

Desktop/session	Window backend	Notes
GNOME Wayland	GNOME Shell extension first, org.gnome.Shell.Introspect fallback	Full target. The extension provides exact window activation when GNOME blocks native introspection; Introspect can list windows and verify the current app; activating a window requires the extension.
GNOME X11	org.gnome.Shell.Introspect, then generic X11/EWMH	AT-SPI works; keyboard input prefers xdotool/XTEST so the live XKB layout resolves keys correctly. Scroll uses xdotool wheel buttons so GTK 3 does not drop the event after a pointer warp.
KDE Plasma / KWin	temporary KWin DBus scripting	Lists and focuses windows through Plasma 5 or 6 org.kde.KWin scripting APIs when the session bus exposes them. IDs are derived from the KWin window UUID, stay the same for a window across calls, and fit JavaScript integers.
Hyprland	hyprctl clients -j; focus through the Lua hl.dsp.focus dispatcher, then focuswindow	Requires hyprctl in the desktop session. With COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND=1, targeted keyboard input uses send_key_state and targeted capture uses grim -T, so neither focuses the window.
niri	niri msg with a direct JSON IPC fallback	Lists toplevels and focuses exact window IDs. When the host does not pass NIRI_SOCKET, the server reads it from its parent processes or the systemd user environment; without it, discovery requires an unambiguous socket matching WAYLAND_DISPLAY. The fallback works without the niri binary. Missing positions remain null; bounds are omitted if output scaling is unknown or mixed.
i3, Sway	i3-msg, or swaymsg when i3 is not installed; on i3, optional xprop for PID hydration	Lists and focuses windows over the active i3 or Sway IPC socket. On Sway this covers native Wayland and Xwayland windows, with pids from the tree, and bounds scaled to capture pixels when every output has the same scale (left out with mixed scales). The socket is the first of I3SOCK and SWAYSOCK that names a socket file, then the newest i3 socket, then the newest Sway socket in XDG_RUNTIME_DIR. X11 windows keep their X11 id; a native Wayland window on Sway gets 2^32 plus its container id, so the two never collide.
labwc with Selkies IPC	$XDG_RUNTIME_DIR/labwc.sock	Requires the Selkies IPC patch; stock labwc has no such socket. Lists windows and focuses unique app-id/title matches through wlr-foreign-toplevel-management. IDs include pid, app id and title, fit JavaScript integers, and change when the title changes. Ambiguous windows are rejected. Bounds are omitted because the IPC does not expose output scaling.
COSMIC Wayland	computer-use-linux-cosmic helper	Installed automatically by ./install.sh, cargo install, and npm. For custom/manual layouts, put the helper next to the main binary, on PATH, or point COMPUTER_USE_LINUX_COSMIC_HELPER at it. The on-screen indicator was validated here (three outputs, mixed fractional scaling).
Other wlroots compositors (stock labwc, phoc, river-classic)	zwlr_foreign_toplevel_manager_v1 on WAYLAND_DISPLAY	Lists toplevels with title, app id, and activated/minimized state, and focuses unique app-id/title matches. The protocol reports no pids or bounds. IDs derive from the app id and title, fit JavaScript integers, and change when the title changes; windows sharing both are rejected for exact focus. On Sway it answers only when the i3 backend does not: when neither i3-msg nor swaymsg is installed, the IPC socket cannot be reached, or it lists no windows. Compositors that do not advertise the protocol, such as Cage, are not supported.
Generic X11 / XFCE / other EWMH WMs	wmctrl plus xprop	Lists, focuses, moves, and resizes windows; keyboard input prefers xdotool/XTEST, and scroll uses xdotool wheel buttons. Window origins are read from the X server, since wmctrl -lG counts the frame offset twice.

If you run on a desktop not covered above, or a covered backend does not come up cleanly, please open an issue with the output of computer-use-linux doctor.

Install

./install.sh, cargo install, the npm wrapper, and the Claude/Codex plugin launcher all install computer-use-linux, computer-use-linux-cosmic (COSMIC Wayland helper), and computer-use-linux-indicator (on-screen activity overlay) together automatically. Only manual prebuilt-binary installs need you to download all three release assets.

Option A: ./install.sh from a clone

Installs system packages on Debian/Ubuntu, Fedora/RHEL-like, Arch-like, or Artix systems; installs Rust if needed; builds all three release binaries (computer-use-linux, computer-use-linux-cosmic, and computer-use-linux-indicator); installs them to ~/.local/bin; configures ydotoold as a systemd user service when available; enables GNOME AT-SPI settings when running under GNOME; and installs the bundled GNOME Shell extension on GNOME Wayland.

git clone https://github.com/agent-sh/computer-use-linux
cd computer-use-linux
./install.sh
# log out and back in if the GNOME extension was newly installed
computer-use-linux doctor | jq .readiness

ydotool is an optional fallback. On X11, the installer includes xdotool as the required keyboard backend. If ydotool is unavailable from the configured repositories, the installer continues, but doctor still requires a keyboard-capable RemoteDesktop portal or wtype (on compatible compositors) on Wayland, or xdotool on X11; direct uinput provides absolute pointer input only. On non-systemd hosts, automatic ydotoold service setup is skipped and the installer prints a command suitable for a per-user supervisor. For an unrecognized distro, pass --package-manager apt|dnf|pacman; --force-unknown-distro auto-selects only when exactly one of those managers is available.

Option B: cargo install (Rust binaries, no system setup)

Installs all three Rust binaries from crates.io. You still handle the system-level pieces yourself: AT-SPI, desktop portals, the optional ydotoold fallback, and the GNOME extension if you need the GNOME Wayland exact-focus backend.

cargo install computer-use-linux
computer-use-linux doctor

For unreleased changes from main, install directly from Git:

cargo install --git https://github.com/agent-sh/computer-use-linux

Then, as needed:

sudo apt install ydotool at-spi2-core         # ydotool 1.0.3+ when using this fallback
sudo apt install wtype                        # optional Unicode typing on wlroots/Hyprland Wayland
systemctl --user enable --now ydotool          # only when doctor selects ydotool (Ubuntu's unit name)
computer-use-linux setup                      # gsettings AT-SPI bridge
computer-use-linux setup-window-targeting     # GNOME Shell extension

Option C: npm wrapper (binary download)

Good for users who already have Node.js and want a no-Rust install. The npm package downloads and verifies the matching computer-use-linux, computer-use-linux-cosmic, and computer-use-linux-indicator binaries during install, or on first launch when npm skipped the install script (npm 12 does for scripts it was not told to allow). The wrapper wires COMPUTER_USE_LINUX_COSMIC_HELPER and COMPUTER_USE_LINUX_INDICATOR_BIN to the bundled helpers automatically.

npm install -g @agent-sh/computer-use-linux
computer-use-linux doctor

You will still need AT-SPI enabled (computer-use-linux setup) and one input backend reported ready by doctor. Start ydotoold only when using the ydotool fallback.

Option D: prebuilt binaries

Linux x86_64 / aarch64 builds are published with each tag. Each binary ships a .sha256 next to it.

• Latest release: https://github.com/agent-sh/computer-use-linux/releases/latest

target=x86_64-unknown-linux-gnu
base=https://github.com/agent-sh/computer-use-linux/releases/latest/download
for binary in computer-use-linux computer-use-linux-cosmic computer-use-linux-indicator; do
  asset="$binary-$target"
  curl -L -O "$base/$asset"
  curl -L -O "$base/$asset.sha256"
  sha256sum -c "$asset.sha256"
  install -D -m 0755 "$asset" "$HOME/.local/bin/$binary"
done

You will still need AT-SPI enabled (computer-use-linux setup) and one input backend reported ready by doctor (start ydotoold only when using the ydotool fallback).

Wire it into your MCP host

The binary speaks MCP over stdio through rmcp 3.5. It answers initialize with the client's protocol version when that is 2024-11-05, 2025-03-26, 2025-06-18 or 2025-11-25, and with 2025-11-25 for any other. Pass mcp as the only argument; everything else is configured through MCP tool calls.

Claude Code and Codex plugin

This repo is a plugin marketplace for both Claude Code and Codex. The plugin registers the MCP server and ships the computer-use-linux skill. On first start it downloads the release binaries pinned to the plugin version, checks each against its published .sha256, and caches them under ~/.cache/computer-use-linux/plugin/. It needs curl or wget, and nothing else is installed.

Claude Code:

claude plugin marketplace add agent-sh/computer-use-linux
claude plugin install computer-use-linux@computer-use-linux

Codex CLI:

codex plugin marketplace add agent-sh/computer-use-linux
codex plugin add computer-use-linux@computer-use-linux

Start a new session afterwards. To run your own build instead of the pinned release, set COMPUTER_USE_LINUX_BIN to its path in the host's environment. System setup (computer-use-linux setup, setup-window-targeting, ydotoold) still applies; ask the agent to run the doctor tool to see what is missing. Codex passes only a short allowlist of environment variables to MCP servers, so the Codex manifest forwards the desktop session variables, the server settings, and every variable the Pi extension passes, such as the GLib and loader paths gsettings reads. Codex Desktop's Linux build already bundles Computer Use; do not install both there.

Codex Desktop (Linux build)

codex-desktop-linux includes a native computer-use-linux feature that embeds and stages this backend inside the app's unified Computer Use plugin. Enable computer-use-linux via make setup-native (or in linux-features/features.json followed by make install-native), then toggle Settings → Computer use → Any App in the desktop app. Do not register a second standalone computer-use-linux MCP server inside Codex Desktop when the native feature is enabled.

Claude Code (CLI)

The plugin above is the simplest route. To wire a binary you installed yourself, use the claude mcp add command to register the binary as a stdio MCP server. Pick a scope:

• --scope user: available across all projects for your user.
• --scope project: written to .mcp.json at the project root for team sharing.
• --scope local (default): only the current project, stored in ~/.claude.json.

# User-wide install (recommended for desktop control)
claude mcp add --scope user computer-use-linux -- computer-use-linux mcp

# Verify the server is registered and reachable
claude mcp list

If computer-use-linux is not on PATH, pass the absolute path (e.g. ~/.local/bin/computer-use-linux). Inside a Claude Code session, run /mcp to confirm the tools are loaded.

Claude Desktop

Edit ~/.config/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "computer-use-linux": {
      "command": "computer-use-linux",
      "args": ["mcp"]
    }
  }
}

Restart Claude Desktop. The tools should appear in the tools list.

Pi Coding Agent

pi install npm:@agent-sh/computer-use-linux

Restart Pi or run /reload. The package exposes one small loader initially; the real tools keep their upstream schemas and are enabled only when Computer Use is needed:

Native tools require Pi 0.84.4 or newer (Node.js 22.19 or newer). The standalone npm CLI wrapper continues to support Node.js 18 or newer.

computer_use_linux_tools({ tools: ["doctor", "list_windows"] })
computer_use_linux_doctor({})
computer_use_linux_list_windows({})

You can also search by capability:

computer_use_linux_tools({ query: "observe a window and click a control" })

Codemode scripts receive { text, images, isError }. Each image is an MCP image block that can be passed directly to image():

const result = await tools.computer_use_linux_screenshot({});
if (result.isError) throw new Error(result.text);
text(result.text);
for (const block of result.images) image(block);

No separate MCP adapter or manual MCP configuration is required. When npm skipped the package's install script (npm 12 does for scripts it was not told to allow), the first real tool call downloads the binary. Pi starts one computer-use-linux process lazily on the first real tool call, reuses it for the session so accessibility snapshots remain valid, serializes desktop actions, and closes it on reload, session switch, or exit. See the Pi setup guide for migration from older adapter-based installs.

Hermes Agent

Install the companion Hermes skill so Hermes has the desktop-specific runbook:

hermes skills tap add agent-sh/computer-use-linux
hermes skills install agent-sh/computer-use-linux/computer-use-linux

The skill is optional but recommended for Hermes users. It teaches Hermes how to install, configure, verify, and call the Linux desktop MCP safely. It follows the same skills/<name>/SKILL.md tap layout used by Hermes community skills.

Then add the stdio MCP server:

hermes mcp add computer-use-linux --command computer-use-linux --args mcp
hermes mcp test computer-use-linux
hermes mcp configure computer-use-linux

configure opens Hermes' tool-selection UI for the server. The generated config holds command, args and enabled, plus tools.include when you pick a subset of tools. Hermes defaults to a 300-second tool-call timeout and a 60-second connect timeout; add timeout or connect_timeout under the server to change them.

mcp_servers:
  computer-use-linux:
    command: computer-use-linux
    args: ["mcp"]
    enabled: true

Subagents keep the parent's MCP tools by default (delegation.inherit_mcp_toolsets).

If you installed the binary somewhere that is not on PATH, pass the absolute path as --command.

Restart Hermes after editing the config. Hermes registers the tools as mcp_computer_use_linux_<tool> and creates the mcp-computer-use-linux runtime toolset.

You can verify both sides before asking Hermes to use the desktop:

computer-use-linux doctor | jq .readiness
hermes skills inspect agent-sh/computer-use-linux/computer-use-linux
hermes chat --toolsets mcp-computer-use-linux -q "List the current desktop windows."

For one-off installs without adding the tap first, Hermes also accepts hermes skills install agent-sh/computer-use-linux/skills/computer-use-linux.

Generic MCP client

Spawn the binary with ["mcp"] as the argv tail. It speaks MCP (JSON-RPC over stdio) through rmcp, with the protocol versions listed above; capability discovery happens through tools/list and the doctor tool. The server normally needs no MCP-specific configuration, but desktop runtime environment still matters (DBUS_SESSION_BUS_ADDRESS, XDG_RUNTIME_DIR, portals, AT-SPI, ydotoold, and optionally COMPUTER_USE_LINUX_COSMIC_HELPER).

Selkies containers (linuxserver.io)

linuxserver.io's Selkies desktop images (webtop, chrome, and others) can run the server inside the container, with the host's MCP client connecting over stdio through docker exec. Start the container with PELORUS=true so pixelflux serves screenshots and input. A small launcher joins the desktop session, because docker exec starts outside it. See the Selkies container guide.

First-run checklist

1. Run doctor.

   computer-use-linux doctor | jq .readiness
   

   Aim for can_register_mcp_tools, can_build_accessibility_tree, can_send_development_input, and can_query_windows all true, with an empty blockers array. Doctor lists detected screenshot routes in capabilities.screenshot, but does not take a screenshot or request consent. Detected routes are always screenshot_capture_status: "unverified" and can_capture_screenshots: false in a doctor report. get_app_state records the result of its requested raw capture as verified or failed; only verified sets can_capture_screenshots: true in that response. If no route is detected, the status is unavailable. An unverified capture is a warning, not an installation blocker.

2. If accessibility.at_spi_bus.ok = false, run computer-use-linux setup (or call the setup_accessibility MCP tool). This sets:

   • org.gnome.desktop.interface toolkit-accessibility true

   You may need to restart toolkit-using apps for the change to take effect.

3. If windowing.can_list_windows = false, inspect doctor.windowing.backends. On GNOME Wayland, run computer-use-linux setup-window-targeting (or call setup_window_targeting) to install the bundled [email protected] Shell extension, then log out and back in so GNOME Shell loads it. On KDE, Hyprland, niri, i3, Sway, COSMIC, labwc and other wlroots compositors, or generic X11, install or expose the matching compositor tool/helper shown in the backend details.

4. Verify screenshot capture. Call get_app_state or run computer-use-linux screenshot. The Screenshot portal may request consent for the caller's app ID. On GNOME, that dialog can be rejected when the app ID does not match the focused app. A detected portal interface alone does not prove capture permission. Inspect the full backend error and the desktop portal logs if capture fails; gnome-screenshot is an optional fallback when GNOME Shell or portal calls are denied.

5. If doctor selects ydotool, confirm ydotool 1.0.3+ and ydotoold are running.

   systemctl --user status ydotoold   # install.sh's unit; Ubuntu's package names it ydotool
   

   Its socket should appear at /run/user/$UID/.ydotool_socket.

Environment variables

Most setups need none of these: the server and installers detect backends and paths. They exist for overriding auto-detected paths and input backends.

Server runtime (set in the MCP host's environment):

Variable	Effect
COMPUTER_USE_LINUX_NOTIFY_ON_COMPLETE	Set exactly to 1 to expose the optional complete_interaction notification tool. Requires notify-send and a desktop notification service; disabled by default.
COMPUTER_USE_LINUX_COSMIC_HELPER	Path to the computer-use-linux-cosmic helper when it isn't next to the binary or on PATH.
COMPUTER_USE_LINUX_INDICATOR	Set to 0 (or false/off/no) to turn off the on-screen indicator. On by default.
COMPUTER_USE_LINUX_INDICATOR_BIN	Path to the computer-use-linux-indicator overlay when it isn't next to the binary or on PATH.
COMPUTER_USE_LINUX_INDICATOR_HIDE_TEXT	Set exactly to 1 to mask all typed text and keycaps in the indicator. set_value values are always masked.
COMPUTER_USE_LINUX_AGENT_NAME	Name shown in the indicator instead of the one derived from the MCP client's clientInfo.
CU_DISABLE_ABS_POINTER	Set exactly to 1 to disable the uinput absolute pointer. Wayland coordinate actions then need the RemoteDesktop portal; they refuse input rather than use ydotool's relative-motion approximation. Native X11 fallback is unchanged.
COMPUTER_USE_LINUX_FORCE_PORTAL_POINTER / …_KEYBOARD	Set exactly to 1 to prefer the RemoteDesktop portal on Wayland, skipping auto-detection. Pointer forcing skips the absolute uinput backend. Forced ydotool or XTEST keyboard overrides take precedence; KDE literal text keeps its clipboard route. Portal startup and text conversion failures return errors without replaying through ydotool.
COMPUTER_USE_LINUX_PORTAL_SCROLL_INVERT	Set to 1 to reverse vertical scrolling through the RemoteDesktop portal. Defaults to normal direction on both GNOME and KDE Plasma; horizontal direction is unchanged.
COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND	Set exactly to 1 on Hyprland to work on a window without focusing it. type_text and press_key calls that name a target window send keys to it through Hyprland's send_key_state dispatcher instead of focusing it and typing, and get_app_state and screenshot capture an unfocused target alone with grim -T instead of raising it or using the portal. The user's focus, cursor and workspace do not change, so the target can sit on a hidden workspace. Text is mapped for a US layout: other keymaps and characters outside printable ASCII, tab and newline are refused before any key is sent. Browser accelerators such as Ctrl+R are ignored by an unfocused Chromium. Unset, input focuses the target as before.
COMPUTER_USE_LINUX_PERSIST_REMOTE_DESKTOP	Set exactly to 1 to ask the RemoteDesktop portal to remember pointer and keyboard grants across processes. The first dialog still appears (on GNOME the remember box starts checked). Later processes reuse the single-use restore token from Start, stored mode 0600 under $XDG_STATE_HOME/computer-use-linux/ (or ~/.local/state/computer-use-linux/). Pointer and keyboard tokens are separate. Needs interface version 2. Unset, every new process is prompted. No effect when input is not using the portal.
COMPUTER_USE_LINUX_FORCE_YDOTOOL_POINTER / …_KEYBOARD	Set exactly to 1 to always route pointer / keyboard through ydotool, skipping the portal and KDE clipboard paths; pointer forcing also skips the absolute uinput backend and native-X11 xdotool clicks, scrolls and drags. Coordinate pointer actions on Wayland then refuse input because ydotool cannot position them accurately; scrolling at the current pointer remains available.
YDOTOOL_SOCKET	Explicit ydotool socket path. When unset, only an owner-only .ydotool_socket in the user's private runtime directory is accepted; the legacy shared /tmp/.ydotool_socket is never selected automatically. Setting this variable is an operator trust override for custom layouts.
COMPUTER_USE_LINUX_FORCE_XDOTOOL_KEYBOARD	Set exactly to 1 to prefer xdotool/XTEST keyboard input when DISPLAY is available. COMPUTER_USE_LINUX_FORCE_YDOTOOL_KEYBOARD=1 takes precedence.
COMPUTER_USE_LINUX_XDOTOOL_TYPE_DELAY_MS	Per-character delay for xdotool type in milliseconds (default 12). 0 is faster but can deliver characters out of order on some X servers.
COMPUTER_USE_LINUX_SCREENSHOT_BACKEND	Force a single screenshot backend, skipping the fallback chain. Accepts gnome-shell, portal, x11, gnome-screenshot, or pixelflux. x11 works only on a native X11 session. pixelflux needs PIXELFLUX_CU and PIXELFLUX_CU_TOKEN. Pin gnome-screenshot for background/systemd contexts where the GNOME Shell and portal DBus paths are denied.
PIXELFLUX_CU	Loopback port of a Selkies pixelflux computer-use server (5000 or 127.0.0.1:5000). With PIXELFLUX_CU_TOKEN, screenshots try pixelflux before the other backends, and click, scroll, drag, press_key, and type_text go there without a portal prompt. A closed port falls through. A server error does not. Anything other than loopback is ignored.
PIXELFLUX_CU_TOKEN	Bearer token for PIXELFLUX_CU. Never logged. Both variables must be set.
COMPUTER_USE_LINUX_ENABLE_SHELL	Set exactly to 1 before starting the MCP server to register the destructive run_shell tool. Unset by default. Do not enable for untrusted or unattended MCP hosts.

Build-time identity overrides (set while compiling a downstream embedded bundle): CUL_GNOME_EXTENSION_UUID, CUL_DBUS_SERVICE, and CUL_DBUS_OBJECT_PATH replace the default standalone GNOME Shell extension UUID and DBus endpoint in both the Rust probes and the generated extension files.

npm wrapper (set during npm install, or before running). The package's install script downloads the release binaries. When npm skipped it (npm 12 runs only install scripts it is told to allow), the wrapper runs the same download on first launch, with its output on stderr. A download that receives no data for 30 s fails:

Variable	Effect
COMPUTER_USE_LINUX_BIN	Run this binary instead of the bundled one. The CLI wrapper and the plugin launcher use the path as given and fail if it does not exist. The Pi extension uses it only when it is an absolute path to an executable.
COMPUTER_USE_LINUX_TIMEOUT_MS	Pi native tools only: per-call timeout in milliseconds (default 60000). Raise it for run_shell calls with timeout_seconds above 60.
COMPUTER_USE_LINUX_DOWNLOAD_BASE	Override the GitHub release base URL the installer downloads from (mirrors, air-gapped hosts).
COMPUTER_USE_LINUX_SKIP_DOWNLOAD=1	Skip the binary download entirely, at install and at first launch.
COMPUTER_USE_LINUX_LOCAL_BINARY / …_LOCAL_COSMIC_HELPER / …_LOCAL_INDICATOR	Install from a local build instead of downloading (used by CI and local testing).

Architecture

• Accessibility tree: atspi crate (tokio backend) talks to the AT-SPI registry on the user session bus. The tree is flattened to (role, name, text, states, bounds) tuples and indexed; element indices are stable for the duration of a get_app_state snapshot. A new snapshot is published only between input operations. click, scroll, perform_action, and set_value hold the input lock while they check and resolve an index, so a checked element cannot turn into another app's node mid-action.
• DBus where desktops expose it: zbus for portal calls (org.freedesktop.portal.Screenshot, …RemoteDesktop, …ScreenCast), GNOME Shell screenshots (org.gnome.Shell.Screenshot), the bundled GNOME extension's dev.avifenesh.ComputerUseLinux.WindowControl service, and temporary KWin scripting.
• MCP transport: rmcp with the transport-io feature; stdio framing, no network.
• Input fallback: on X11, keyboard input prefers xdotool/XTEST and falls back only when xdotool cannot launch. On Wayland, literal text uses wtype when it is installed on a compatible compositor (not GNOME, KDE or COSMIC), unless COMPUTER_USE_LINUX_FORCE_PORTAL_KEYBOARD=1; wtype supports Unicode through the virtual-keyboard protocol. Otherwise KDE uses its clipboard route and other Wayland sessions use portal keysyms, which never fall back to ydotool. ydotool 1.0.3+ with its ydotoold socket types only when the selected wtype is missing or cannot start, when KDE cannot read the Klipper clipboard, or when ydotool is forced. A wtype that started and failed is returned without replaying the text. press_key chords use the portal when ydotool is absent; on a session whose bus answers that it has no RemoteDesktop portal (wlroots compositors such as Sway, labwc, river or cage), they go through wtype instead. The route is chosen before any input is sent, so a failed or denied portal request is never replayed through wtype. install.sh can configure ydotoold; the setup command only enables the GNOME AT-SPI bridge.
• Hyprland background mode: with COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND=1, a targeted type_text or press_key is sent as hl.dsp.send_key_state down/up pairs for the target's address, written straight to Hyprland's control socket in one [[BATCH]] request per 64 key presses (never through hyprctl arguments, so typed text cannot be read from /proc/<pid>/cmdline), and the window is never focused. Hyprland answers each dispatch, so when every dispatch in a failed batch was answered the error states how many key presses were delivered (and whether a key was left down), and the caller can send only the remainder. An empty or incomplete reply, a timeout, or an ok after an error proves nothing about what ran, so the error gives the confirmed count and says the rest may have been delivered. The action indicator shows the input; it is masked unless an AT-SPI probe of the target app finds a non-secret editable field or a terminal. The dispatcher resolves a key name to its unshifted physical key and applies only the modifiers it is given, so text goes through a US-layout table (shifted symbols become SHIFT plus the base key); unmappable characters or a non-US main keymap refuse the whole call. An unfocused target is captured with grim -T <stableId> (ext-foreign-toplevel image capture). Hyprland stops sending frame callbacks to windows on hidden workspaces, so their buffer holds the last frame drawn on screen; each capture makes the client repaint, so a hidden-workspace target is captured every 150 ms until two consecutive frames match (at most 8 captures) and the result's source says whether it settled.
• Hyprland/Electron text: for classified nonsecret fields on Hyprland, wtype passes one literal argument after -- to avoid stdin keymap-refresh corruption. Argument text is visible in process listings; confirmed password fields are rejected. Terminals and ambiguous fields (with or without AT-SPI) use private stdin for at most 100 Unicode characters per call; split larger input into smaller calls or expose accessibility. The input probe has its own 1.5-second budget; the 250 ms indicator probe only masks text. Other compatible compositors retain their existing stdin path. NUL and text over 131,071 UTF-8 bytes are rejected before focus or indicators. Success confirms dispatch only; re-observe the field before retrying.
• Limited accessibility feedback: when exact window focus was verified before input but AT-SPI reports no focused element, the result describes the missing feedback without claiming the input was lost. It still does not verify the field contents. A complete scoped tree containing only application/window containers adds an Electron/Chromium --force-renderer-accessibility hint; a truncated tree instead retains its traversal-limit guidance.
• Focused containers: Chromium can mark a document web focused alongside its contenteditable editor. The focus probe keeps searching that document and reports the editor; a document that holds focus itself still gets a warning, which no longer claims the text was lost. An editor can carry the editable state without the EditableText interface (supports_editable_text: false): type_text can reach it, set_value cannot. Verify its text or a screenshot before submitting or retrying.
• Large embedded browser trees: an app-sc