OpenAdapt

OpenAdapt

Give your agent a program, not a mouse.

The open-source, verified execution layer for computer-use agents.

What it can do · Documentation · Quickstart · Live demo · Website · Get updates · Discord

A computer-use agent looks at the screen and chooses every click again, on every run. OpenAdapt lets your agent, or a person, do a task once in a web, Windows, macOS, Linux or remote-desktop app. It compiles that run into a program, and your agent calls the program as one tool. The program runs with no model calls and sends no screenshots anywhere. It confirms the right record before each save, reads the save back from the system of record after it, and returns a plain outcome. When anything doesn't match, it stops and tells your agent instead of guessing.

Measured on OpenEMR and openIMIS

One new-patient referral in a test copy of OpenEMR, the open-source EHR whose 7.0 release is ONC-certified, timed from a fresh browser through sign-in, the entry and the saved record read back (median):

Who does the entry	Time per referral	Saved correctly	Model cost
Agent alone (Claude Sonnet 5.5 computer use)	██████████████████ 36.5 s	62 of 63	about $0.08
OpenAdapt program, screen steps	███████████ 22.5 s	18 of 18	$0
OpenAdapt program, API step	██ 3.3 s	18 of 18	$0

The API step replaces the program's 41 screen steps with the one form submission the app itself sends, then reads the record back. In both program rows a possible-duplicate referral stopped without saving, as it should. Both program rows ran on 19 Oct on one copy of the app, interleaved, with the approval gate off; the benchmark harness admitted the API step. The agent row is the 14 to 15 Oct timing study, on its own copy of the app. SCOREBOARD.md has every run.

On a test copy of openIMIS, the open-source insurance and claims system that runs Nepal's national health insurance scheme, the compiled program entered 149 synthetic claims with no model calls: 148 read back correct and none were saved wrong (SCOREBOARD.md).

A screen can't show a save that went wrong behind it. On a test copy of OpenEMR, with faults injected in its database (a field blanked, a value changed, a duplicate chart, a chart removed, another chart deleted or edited), the agent claimed success on all 30 bad saves. Reading the record back caught 25 of the 30 and confirmed all 5 clean saves, with no false alarms. On MockMed, a test app that shows a save as done before its server answers, the agent also claimed all 30 bad saves, and the read-back caught all 30 (details below).

One harness, three modes

Mode	What happens	Model calls
Explore	Your agent, or a person, does the task once while OpenAdapt records every action. The save is read back before anything compiles.	Your agent's, on your own key
Run	The compiled program repeats the task with new inputs. It checks the record before each save and reads it back after.	None
Repair	When the app changes, the run stops before saving. An agent shows the fix, the program tries it up to the save without saving, and a person approves it.	Only when something changed, on your own key

Why the read-back matters

A success screen is not proof of a save. We injected 8 kinds of bad save into a test app (a save that never landed, landed partly, landed twice, went to the wrong record, or changed another record), 9 runs each, alongside clean runs. Each check below was asked whether those 72 bad saves had succeeded:

How the save was checked	Bad saves reported as saved
The app's "Saved" message on screen	54 of 72
A read of the record over the app's REST API	9 of 72 (all writes to a table it didn't read)
A read of every table in the database	0 of 72

EFFECT_E2E.md has the fault set, the method and the raw results.

An agent working alone can't see these faults either. Claude Sonnet 5.5, using computer use, did a task on a test app whose server broke the save in 6 ways, 5 times each. The agent reported all 30 bad saves as done. Given the agent's run, openadapt-flow verify-run read each record back: it flagged all 30 and confirmed all 10 clean saves, with no false alarms. The test app shows "saved" before the server answers, so the screen gave the agent nothing to notice. That gap is what the read-back closes. SCOREBOARD.md has the method, the raw runs and the cost.

Try it in one command

uvx --python 3.12 openadapt quickstart --break-it

No account or API key. uv fetches Python 3.12 if you need it; with Python 3.10 to 3.12 already installed, python -m pip install --upgrade openadapt then openadapt quickstart --break-it does the same.

What it shows

The command runs a synthetic clinic task twice on your computer. The first run saves an entry and confirms it through the app's read-only API. For the second run, the test server is set up to reject the save, so it never reaches the record. This is the output from openadapt 1.17.0, with the break-it lines in the current wording:

[1/5] Record the demonstration against a real persistence boundary
[2/5] Compile and propose a fixture-specific effect contract
[3/5] Certify against the clinical-write policy
[4/5] Admit and execute under the standard profile
      VERIFIED in 5.6s; 0 model calls; the system of record holds 1 record(s)
[5/5] Emit the local run receipt
...
--- break-it: the same certified bundle, with the save set to fail ---

  Injected fault:      'optimistic' -- the server rejects the write
  On screen:           every on-screen check passed, including the saved message
  The verifier found:  1/2 declared effect(s) REFUTED by an independent
                       read of the system of record, which holds 0 record(s)
  The engine did:      HALTED at the consequential step instead of claiming
                       success (transaction: RECONCILIATION_REQUIRED, billable: no)
  Result:              Check the record. A save may have gone through. A person checks the record before anything is retried.

The quickstart tutorial walks through every line.

Build with it

Compile your agent's run

Your agent (Claude or OpenAI computer use, or your own loop, on your own model key) does the task once while OpenAdapt records it. A person reviews and admits the draft program, or a registered agent admits it under a policy a person signed. After that, the program runs instead of the agent.

from openadapt_flow.jit import compile_agent

jit = compile_agent(my_agent, task=task, app=app, verifier=verifier,
                    policy="clinical-write", cache_dir="jit-cache", model_key=key)
jit.run(note="Follow up in two weeks")    # the agent does it; a DRAFT is saved
# a person reviews, runs the canary, and admits the draft
jit.run(note="Recheck knee in ten days")  # the program runs: 0 model calls, save read back

Compile a computer-use agent's run has the full example. The same path takes an agent's run through the web, Windows, macOS, Linux, RDP and Citrix backends.

Call programs from your agent

claude mcp add openadapt -- uvx --python 3.12 openadapt-agent serve

Each compiled workflow becomes one tool call. The agent decides what to enter; OpenAdapt makes the entry, checks it, and returns an outcome instead of the agent driving the screen. With no bundles, the server offers a synthetic sandbox workflow that returns every outcome. To serve your own workflows, run openadapt agent serve --mode production --bundles ./bundles; without --mode production the agent can list them but not run them. openadapt-agent emit-skill <bundle> writes an Agent Skill for one workflow, and openadapt flow verify-run judges a write made by any agent or script against the system of record. Connect an agent covers other clients.

Record a task by hand

python -m pip install --upgrade openadapt
openadapt flow record --url https://staging.your-app.example --out rec
openadapt flow compile rec --out bundle --name my-first-workflow
openadapt flow replay bundle --url https://staging.your-app.example --run-dir run

replay drives the app at --url and performs every recorded action, including saves you can't undo. It doesn't ask first and doesn't check the saved record. Point it at a test or staging copy, never at production data.

A program that writes to a real system also needs record checks, a deployment file, certification, qualification, and an admission; then openadapt flow run runs it. By default each save waits for a person to approve it in OpenAdapt Desktop or on a paired phone. Your first workflow walks through it. Several recordings of the same task can become one program with branches and loops (openadapt flow induce).

To record a desktop app, install the recorder and your operating system's adapter, for example python -m pip install "openadapt[capture,windows]", or use OpenAdapt Desktop.

Judge or train any agent

openadapt flow verify-run judges a write made by any agent or script against the system of record. openadapt-flow serve-reward turns the same check into a reward for training or evaluating computer-use agents (REWARD_WORKER.md).

An agent alone, and an OpenAdapt program

The same referral task on a local OpenEMR 8.0.0.3 test copy with synthetic patients, 62 timed runs each (14 and 15 October 2026), judged by the same database read-back:

	Claude computer-use agent	OpenAdapt program
Model calls	On every step of every run	0
Screenshots sent to a model	Throughout every run	0
Model cost per referral	About $0.08 at list price	$0
Path through the app	Chosen again each run	The same compiled steps
Median time	36.5 s	29.7 s
Referrals saved correctly	62 of 63	62 of 63

The program drove every page, as the agent does. The API step compiled from the recording's network traffic removes those page steps: 3.3 s per referral, in the table at the top. Every result is published. OpenAdapt isn't affiliated with the OpenEMR project.

Every backend

Record, compile and run take every surface; the evidence differs by surface (BACKEND_PARITY.md). Self-repair is qualified on browser runs (benchmark/auto_repair) and end to end on native macOS AX (#1715). Outside a browser it runs with repair.session (#1717), and native and pixel backends admit its new clicks by element identity (#1709). Teaching the fix for a halted run and rollback are tested on browser runs. Where the app exposes its structure, OpenAdapt uses it; on remote desktops it works from pixels.

Surface	What OpenAdapt reads	Validated on
Web apps	The screen, plus the page's DOM	OpenEMR, Odoo and openIMIS test copies; CI runs in-tree apps on every build
Windows apps	The screen, plus UI Automation	3 of 3 saves in a WinForms test app
macOS apps	The screen, plus Accessibility	3 of 3 saves in TextEdit
Linux apps (X11)	The screen, plus AT-SPI	Recording and an AT-SPI read in CI; backend actions are unit-tested
Remote desktop (RDP)	The screen	105 runs through the stock Remote Desktop client on a real AVC444 session: 55 saves confirmed in the database, 0 wrong-record saves, 40 false stops
Citrix	The screen	A Citrix stand-in app with no DOM: 3 healthy runs, 3 stops on drift

Each application on each surface needs its own qualification. Surfaces and Limits have the details.

How it works

flowchart LR
    S["Record, or let an agent do it once"] --> C["Compile"] --> Q["Certify and qualify"] --> A["Admit"] --> R["Run with checks"]
    R -.->|"stopped: approve a fix"| C

For each step the compiler uses the most reliable layer the app offers: page structure first (DOM, Windows UI Automation, macOS Accessibility, Linux AT-SPI), a direct API call where one is bound, and pixels (template, text and layout matching) only where nothing else exists, such as a remote desktop. Qualification runs the program against normal and fault cases before an admission lets that exact version run. At each save the program checks that the right record is on screen, makes the entry, and reads the record back.

How it compares

	Computer-use agent	Agent with a replay cache	Traditional RPA	OpenAdapt
How you define the task	A goal in plain language	One agent run	A script or flow built in a designer	A person's recording or an agent's run
What runs each time	A model chooses each step from the screen	The recorded steps; the agent takes over on a miss	The fixed script	A compiled program you can read
Model calls per run	At least one per step	None on a hit; the agent's calls on a miss	None	None with default settings
When the screen changes	The model tries to adapt	The agent improvises live	A selector stops matching, or matches the wrong control	The run stops before saving; a fix is proposed and a person approves it
Right record?	The model judges from the screen	Usually only that the screen looks as recorded	Whatever the developer scripted	Checked before every save
Did the save land?	The model judges from the screen	The screen	Whatever the developer scripted	Read back from the system of record
Strongest for	New or changing tasks	Repeated tasks where a wrong save is cheap	Stable processes with a team to maintain the scripts	The same entry many times, where a wrong save is costly

What it can do

• Learns from any successful run, a person's demonstration or a computer-use agent's verified run, and compiles it into a program that runs with no calls to a hosted AI model.
• Uses the most reliable layer the app allows: page structure (DOM, UI Automation, Accessibility), direct API calls where one is bound, and pixels only for remote desktops.
• Builds branching programs from several demonstrations, repeated halts and background runs, and routes each item to the right verified variant.
• Works across surfaces: web, Windows, macOS, Linux and RDP backends, chained in one workflow.
• Confirms the right record before every write and reads the saved record back from the system of record, never just the screen.
• Stops instead of guessing, never retries an uncertain save, and says what changed before a stop.
• Binds each approval to exactly what will be saved, with one-tap answers on a paired phone.
• Repairs itself when the app changes, probing a fix up to the save and asking a person to approve it (qualified on browser runs and end to end on native macOS).
• Works as a safety layer under any agent: MCP tools, a guarded save, Agent Skills and a CLI.
• Runs locally: no calls to a hosted AI model on healthy runs, patient data redacted, signed receipts.

CAPABILITIES.md has the evidence and code for each.

What your agent gets back

You see	transaction_outcome	An agent gets	What happens next
Done and checked	VERIFIED	done	Nothing. Only this result gets a shareable receipt.
Stopped before saving	HALTED_BEFORE_EFFECT	needs_review	Fix the cause, or show the program the fix, and run again.
Check the record	RECONCILIATION_REQUIRED	not_sure_if_saved	A save may have gone through. A person checks the record before anything is retried.
Finished, not checked	COMPLETED_UNVERIFIED	not_sure_if_saved	Fine for testing; not a production success.
Saved, waiting for its check	SAVED_AWAITING_CHECK	not_sure_if_saved	The save's check is a later report (a claim acknowledgment, an export). Don't rerun it. openadapt-flow deferred settle counts it as done only when the report confirms it.
Didn't run	REJECTED_POLICY, CANCELED, FAILED_PLATFORM	did_not_run	Fix the gate or the environment and run again.

The console's last line can read HALTED for both the second and third rows. transaction_outcome in report.json tells them apart, so base every rerun on it. Run outcomes lists every value.

When to use something else

• The app has an API for the task. Call the API.
• Every task is new. Use a computer-use agent. OpenAdapt pays off when the same entry repeats.
• There's no way to read the record back. The record check needs read access to the system of record: an API, the database, FHIR, or a separate signed-in session. Without one, a run can only end as finished but not checked.

Packages

pip install openadapt installs the openadapt command, the engine, and the MCP server. The product packages live in this repository:

Package (PyPI)	What it is	Source
openadapt	The openadapt command	openadapt/
openadapt-flow	The engine: record, compile, check, qualify, run, repair	packages/flow/
openadapt-agent	The MCP server and Agent Skills	packages/agent/
openadapt-types	Shared request, result, and receipt models	packages/types/
openadapt-capture	The desktop recorder: openadapt[capture]	packages/capture/
openadapt-privacy	Finds and redacts personal and health data: openadapt[privacy]	packages/privacy/
openadapt-desktop	OpenAdapt Desktop (download)	packages/desktop/
openadapt-tray	Menu-bar and system-tray controls and status for OpenAdapt Desktop	packages/tray/

Each package installs only what it imports. For example, pip install openadapt-privacy doesn't install the engine or the desktop app. CI checks each package's install size and dependency list (install weight).

Research on GUI agents lives in separate repositories: openadapt-evals (benchmarks) and openadapt-ml (model training). The packages map has the full list.

Security and procurement

Screens, typed values, and recordings stay on the computer or runner that runs OpenAdapt. The shareable receipt and the Cloud dashboard hold counts, digests, and closed values. A sanitized copy leaves only after a person approves it, and screenshots go to a model host only if you turn on model grounding. Security & procurement covers data handling, deployment boundaries, evidence, subprocessors, and release integrity. Plans are on the pricing page.

The block below is generated from the signed public lifecycle record.

The signed ledger has seven active target admissions. Their evidence class is remote-safe-synthetic, not MockMed production_acceptance. They remain valid until revoked; there's no expiry date. pip install openadapt is the admitted runtime (launcher 1.16.0, Flow 1.35.1). Desktop's admitted artifact is the 0.16.0 wheel. Native Mac DMGs stay ad-hoc signed 0.15.0; don't treat those as admitted. A Production run still needs a workflow admission for the exact compiled bundle, application, and environment. Check the live signed Production record.

Earlier versions

The pre-1.0 version (v0.46.0) is frozen under legacy/. docs/LEGACY_FREEZE.md explains how to install it and how to migrate.

Contributing

Launcher, engine, MCP server, and contract changes all belong in this repository; engine changes go in packages/flow/. Contribution guide · Issues · Discussions · Report a vulnerability · Code signing policy

Maintained by OpenAdaptAI under the MIT License.