Computer-use automation

Teach an agent a legacy bank screen once. Replay it forever.

An LLM learns a back-office UI that has no API. The result is a typed, versioned capability that replays deterministically, with no model in the loop.

Live demo: coming soon Source on GitHub Design write-up
No model in the replay Exactly-once money moves Human takeover, even headless
127.0.0.1:8000 · transfer-funds
Replay
Transfer confirmation page at the end of a recorded replay
The last page of a recorded replay of transfer-funds, kept as evidence in the repo.
The problem

Credit unions run consoles with no API.

Table layouts, iframes, one URL for every screen, slow pages, sessions that expire. An agent that re-reasons about the screen on every run is slow, costly and unpredictable, and it should not be moving money on a guess.

The idea

The model discovers. The artifact is the capability.

Deterministic replay executes it. Three phases, kept strictly apart.

1

Discover

Claude reads an accessibility snapshot of the page and completes a goal once. A person can take over the live browser, and what they do is recorded as steps.

2

Save

The run becomes a versioned, typed JSON artifact: ranked locators, typed inputs and outputs, a text success checkpoint, declared business outcomes and recoverable conditions.

3

Replay

No model. Each step is resolved by ranked locators and checked. The result is a success, a business outcome such as "member not found", or a failure that says what the page showed.

Goal→Claude drives the browser→Typed artifact→Deterministic replay→Success / business outcome / failure
Architecture

The real import graph, with boundaries that tests enforce.

One Python process. Replay never imports the model client or the discovery loop, and the router never imports a browser.

Click to enlarge 
flowchart TD
  cli["cli.py
entry points"]
  agent["agent/
discovery loop, perception"]
  replay["replay/
deterministic engine"]
  router["router/
catalog, proposer, validator"]
  artifact["artifact/
schema, recorder, store"]
  handoff["handoff/
session, gesture, remote"]
  guardrails["guardrails/
allowlist, policy, redact"]
  subgraph outside["External"]
    mock["mock_app/
target app"]
    anthropic(["anthropic SDK"])
    pw(["playwright"])
    ollama(["Ollama, local"])
  end
  cli --> agent
  cli --> replay
  cli --> router
  cli --> artifact
  agent --> artifact
  agent --> guardrails
  agent --> handoff
  replay --> artifact
  replay --> guardrails
  replay --> handoff
  router --> artifact
  artifact --> guardrails
  handoff --> guardrails
  agent --> anthropic
  agent --> pw
  replay --> pw
  handoff --> pw
  router -. "HTTP" .-> ollama
  agent -. "browser" .-> mock
  replay -. "browser" .-> mock
Solid arrows are imports.Dotted arrows are runtime calls over a socket.Nothing in artifact/ imports agent/.
What it handles

Built for the ways legacy systems actually fail.

Each one is declared data or a bounded rule, tested, and visible in the result.

Slow legacy apps

Replay waits for the page's state, not a page event, and checks a success phrase on the page rather than a URL.

Money that moves once

Every request carries a per-attempt run ID. After an ambiguous failure, replay asks the app whether the write registered and retries once only if it did not.

Expired sessions

Replay signs back in and still moves money exactly once, whether the expiry hits before, on, or after the confirm click.

Risky steps

A draft capability asks a person first. An approved one runs unattended and is flagged in the result.

Human in the loop

A person can take over the live session, including a headless one through a DevTools link, and their clicks become steps in the artifact.

Guardrails

One allowlist and one risk classifier for both phases, and PII redacted from logs and evidence.

Plain English, optional

A local Llama proposes a capability and its arguments. A deterministic validator decides, and anything doubtful becomes a discovery. A wrong value is refused, never guessed.

See it run

Recordings.

Discovery and human handoff need a model key and a debugging port, so they are shown as recordings rather than hosted.

Discovery runComing soon
Replay with new valuesComing soon
Session expires mid-transferComing soon
Human takeoverComing soon
Honest limits

What this is not.

  • The target is a mock. A credit-union console I built, not a real core banking system.
  • The hosted demo is replay only. Its plain-English tab needs a local Llama (Ollama), so it works when run on your own machine, not on the hosted copy.
  • Guardrails have edges. The allowlist is checked on navigation and action types, not on every link click. The risk classifier reads button names only.
  • Everything else is listed in sections 6 and 7 of the write-up.