Cua Computer-Use Toolkit
Background desktop computer-use for AGNT, powered by the Cua Driver (trycua/cua, verified live against 0.19.3 — "computer-use 2.0"). Agents drive real Windows apps WITHOUT touching your mouse or stealing focus: input routes through the accessibility layer while the driver paints its own session-coloured cursor overlay. cua-act runs a full observe→reason→act→MEASURE loop using whatever AI provider/model you have selected in chat. Deterministic verify_state postconditions replace screenshot guesswork, invoke_menu resolves native menus instead of clicking blind, and paste_text reaches editors that expose no ValuePattern.
Get Cua Computer-Use Toolkit
Cua Computer-Use Toolkit
A plugin is a bundle — tools, workflows, agents and widgets shipped together — that installs as one unit and extends what AGNT can do.
Runs on your own machine. Free, with unlimited local runs. Published by Nathan Wilbanks.
- Type
- Plugin
- Category
- Operations & Tools
- Version
- 0.5.0
- Price
- Free
- Publisher
- Nathan Wilbanks
About this plugin
# Cua Computer-Use Toolkit (AGNT plugin)
Background desktop computer-use for AGNT, powered by the [Cua Driver](https://github.com/trycua/cua) (`trycua/cua`, MIT). Verified **live against driver 0.19.3** — the "computer-use 2.0" line.
Agents drive real Windows applications **without touching your mouse and without stealing focus**. Input routes through the accessibility layer (UIA Invoke / ValuePattern) rather than through the OS input queue, so nothing warps your cursor, raises a window, or flashes the taskbar — while the driver paints its *own* session-coloured cursor overlay so you can still watch the work happen.
## Tools
| Tool | Category | What it does | |------|----------|--------------| | `cua-setup` | utility | Install / update / doctor / **health** / **permissions** / **config** / **tools** / serve / stop. `ensure` = one-call bootstrap. | | `cua-session` | utility | **New.** Declare a run identity: agent cursor overlay + capture policy. Escalate to desktop scope explicitly. | | `cua-windows` | utility | Open windows *and* (`include=apps`) installed/running applications with a ready-to-use `launchPath`. | | `cua-observe` | utility | `window` (elements + tokens + tree + screenshot) · **`verify`** (deterministic predicates) · **`zoom`** (native-res crop) · **`desktop`** (full display). | | `cua-input` | action | click / type / **paste_text** / press_key / hotkey / scroll / set_value / drag / **invoke_menu** / **clipboard** / **set_window_frame** / launch_app / bring_to_front / kill_app. | | `cua-act` | action | Autonomous observe→reason→act→**measure** loop with a real success gate. |
## What changed in v0.5.0
### 1. Sessions and the agent cursor Every `cua-act` run now declares a driver session. That buys a colour-coded cursor overlay animated to each target (your physical pointer never moves) and a capture policy: `auto` scope starts window-only and keeps desktop-wide capture **locked** until you escalate on purpose. Escalation is **permanent** for that session id — the driver's design, not ours, which is why it is `confirm`-gated. The session is ended in a `finally` block so an abort can't strand a cursor on screen.
### 2. The closed outcome contract — and the trap inside it 0.19 answers every action with `{ effect, route, delivery, escalation }`, and **a refusal still exits 0**. There are three envelopes in the wild, all measured here:
```jsonc { "effect": "unverifiable", "route": "accessibility", "delivery": { "mode": "background" } } { "status": "refused", "refusal": { "code": "stale_element_token", "message": "…" } } { "effect": "refused", "code": "ambiguous_window_target", "pid": 52672 } ```
> **`effect: "unverifiable"` is NOT a failure.** A measured click on the Calculator's `Seven` button returned exactly that — and `verify_state` then proved the display read `Display is 7`. The driver is declining to overclaim, not reporting a no-op. Treat it as failure and a working agent gives up; treat it as success and a broken one keeps going. The only correct answer is to **measure**.
The old version regex-matched raw text for `delivery_failed`, which cannot tell these three apart. `lib/driver.js` now normalises all of them in one place.
### 3. Verification replaces vibes The weakest link in any screenshot loop is the model grading its own homework. `cua-observe mode="verify"` evaluates real structured state and returns `satisfied` / `unsatisfied` / **`unknown`** — and **unknown never counts as success**.
Before you install
Windows 10/11 in an interactive logon session. The Cua Driver (trycua/cua, MIT) is installed on first use by cua-setup action=ensure.