Whether the vision models that drive a screen are now good enough to take over an RPA bot that already runs, and that a business already pays to keep running, is the question this month kept putting in front of us, and it has no clean general answer. Most of the writing on it is pitched either at selling the capability or at writing it off, and the material worth having sits further down, in the descriptions of how these agents fail. The 2026 benchmark papers and the vendor documentation are what this month's note works from, and they are more detailed on the failure modes than on the capability.
Our position, before any of the detail: for a bot that already drives a stable screen at close to zero maintenance, the answer is no, and we say so in assessments. For a system with nothing programmatic behind it, the answer is sometimes, at a cost per run you have to measure rather than assume, and only with verification the agent does not carry out on itself. The rest of this is why we land there.
How screen operation actually works
A screen-operating agent runs a loop: it captures the screen as an image, sends it with the task and the actions so far to a vision-capable model, and gets back one action such as click here or type this string. A harness executes that action through a virtual mouse and keyboard, captures the screen again, and repeats until the model reports completion or a step budget runs out. Microsoft describes computer use in Copilot Studio as Computer-Using Agents combining vision with reasoning, positioned for tasks where no API exists.
Browser use is a narrower version of the same loop, with the agent reading the DOM or accessibility tree as well as the pixels, which gives it structured element references and cheaper text extraction. Google has said the Gemini 2.5 Computer Use model is primarily optimised for web browsers and not for desktop operating-system-level control.
The structural difference from a recorded selector, and the one that drives everything below, is when the decision about which control to touch gets made. A UiPath selector or Power Automate desktop flow fixes it at build time as an attribute match or an XPath expression. A screen-operating agent makes it at run time from the rendered screen.
Where it behaves differently from a recorded selector
We are not making a general claim that screen operation outperforms a recorded selector. The differences below hold in specific circumstances. Because the control decision is made at run time from the rendered screen, a screen-operating agent can still complete a task after changes that break a selector: a moved button, a renamed label, a re-ordered column, an added field. Microsoft claims the same for computer use in Copilot Studio. That is a vendor statement about its own product, and no independent measurement of it has been published.
It can also operate screens that were never recorded during the original build: a consent dialogue box or a variant page for one customer type stops a scripted bot at the selector that fails to resolve, and an agent reading the rendered screen can act on it and carry on. The same run-time judgement produces both the adaptation and the wrong decisions, which is where the failure modes below come from.
What the benchmarks measure
The numbers in circulation come from a handful of harnesses, and they do not all measure the same thing, which is worth holding on to before any of them get quoted at you in a proposal. OSWorld runs agents against real desktop applications. Anthropic stated in September 2025 that Claude Sonnet 4.5 led it at 61.4%. We would not read that figure next to the 2026 results below, because it comes from a 2025 harness and model version and does not line up with them.
Scores drop sharply on longer tasks, which is what OSWorld 2.0 measures: 108 long-horizon workflows, each taking human users a median of about 1.6 hours and averaging 318 tool calls with Claude Opus 4.7 at maximum thinking, against about 30 in OSWorld 1.0. Under its primary binary-completion metric at 500 steps, Claude Opus 4.8 with maximum thinking and batched tool calls scored best at 20.6% completed with a 54.8% partial score, and GPT-5.5 plateaued near 13%.
The measurement methods are contested as well, which is the part we put the most weight on. A 2025 reassessment attributed over-optimistic web agent results to shortcomings in existing benchmarks and introduced Online-Mind2Web, 300 tasks across 136 websites. A 2026 WebVoyager evaluation paper reports that automatic evaluators disagree with human annotations 20% to 40% of the time, citing earlier work for that figure without measuring it itself. An independent open-source evaluation could not reproduce Browser Use's reported 89% WebVoyager result, and measured 77.3% self-reported performance against 60.2% LLM-verified success. We treat 60.2% as the number that means something, and it is the figure we plan against.
Failure modes and what they mean in practice
The failure modes are the part of this month's material we would put in front of anyone considering one of these agents for real work. The OSWorld 2.0 authors characterise current failures as losing track of constraints, missing information that arrives mid-task, guessing where a person would ask, and skipping verification, with the worst results where a task hinges on hidden state, and they attribute these failures to task management rather than to GUI control or coding ability. Reflex's benchmark shows the same problem at a smaller scale: asked to process pending reviews, its vision agent found one of four, accepted it and moved on without paginating, and could not finish the task until a 14-step walkthrough was supplied.
Variance is the second problem, and Reflex reported it as the most surprising part of its vision results: across three trials, wall-clock time spanned 749s to 1257s and input tokens spanned 407k to 751k. The open-operator-evals maintainers attribute high variance to non-deterministic models on a constantly changing web and recommend averaging over multiple runs, so one successful pilot run says little about the range in production.
Prompt injection is a failure mode with no equivalent in a selector bot, and Anthropic reported browser use without safety mitigations showing a 23.6% attack success rate when deliberately targeted, reduced to 11.2% in autonomous mode with mitigations, and a four-attack browser challenge set falling from 35.7% to 0%. Those figures come from Anthropic's own testing, so what they show is how far the mitigations moved the result. Against an adaptive Best-of-N attacker given 100 attempts per environment, Anthropic called a 1% attack success rate still meaningful risk and stated that no browser agent is immune. Microsoft warns that its access control stops the model acting on sites outside the allow list without stopping it opening them, and that sharing an agent with maker-provided credentials lets anyone act with the author's access.
Those properties set the minimum engineering we would put around a screen-operating process. Verification has to be external to the agent: code that reads the target system afterwards and confirms the intended records exist with the intended values. Runs need a step budget, a dedicated least-privilege account, network-level allow-listing independent of the model, and a stored trail of screenshots and actions.
Cost and latency
Cost is the question that decides most of these, and it is worth being precise about what is being compared. In a Reflex head-to-head on the same application with Claude Sonnet, a vision browser agent needed 53 steps plus or minus 13 and 550,976 tokens plus or minus 178,849 for an admin-panel task a structured API agent completed in 8 calls and 12k tokens, roughly a 45x difference. The API agent finished in 19.7s plus or minus 2.8s and the vision agent in 1003s plus or minus 254s. Microsoft Copilot Studio bills computer use per model step, at 5 Copilot Credits standard and 15 premium.
We do not quote a cost per run for a process we have not measured. The figure is set by the steps your screens require, the image tokens per screenshot, the context accumulated across the run, and the retries the reliability target demands, and a pilot measures those four across repeated runs.
How we decide per process
None of the published work settles a single one of your processes, so this is the sequence we work through instead. The first question we ask is whether the system exposes anything programmatic: a REST or SOAP endpoint, a database interface, a vendor SDK, a documented file import, a scheduled export. Where one exists it is the better foundation for a rebuild, giving typed fields, real error responses, testability without a screen and idempotency on retry. Screen operation is for where nothing programmatic exists.
Where there is no API, AI screen operation can take the place of the existing bot. Whether it should is a decision we make per process on five things: volume, since cost per run times monthly runs has to be set against the bot's current maintenance hours; the reliability requirement, since a lower success rate is survivable where the outcome can be verified in the target system and failures retried; cost per run from repeated pilots; whether part of the process has a programmatic path, since a hybrid where code handles the stable spine and the agent only the step with no API is easier to verify; and the consequence of a wrong action, since anything that pays, lodges, sends or deletes needs human confirmation.
In some cases the right decision is to leave the bot in place. Where a selector bot drives a screen unchanged in two years and carries close to zero maintenance hours, replacing it adds cost and a non-deterministic runtime without removing any maintenance burden. The processes we look at first are those with recurring selector failures, those that have written wrong values without stopping, and those whose system has an API the original build never used.
Where the market sits
The RPA vendors are moving this way: UiPath launched UiPath for Coding Agents on 12 May 2026, Automation Anywhere paired its Process Reasoning Engine with OpenAI reasoning models in a 20 January 2026 announcement, and SS&C Blue Prism ran a launch broadcast for WorkHQ on 27 April 2026, with general availability since March 2026.
Gartner predicted more than 40% of agentic AI projects will be cancelled by the end of 2027, and its analyst Anushree Verma described most current projects as hype-driven experiments. We read that as a comment on how the projects are set up rather than on what the models can do, and the setup is the part you control. Where a system has no API, AI screen operation can take the place of the existing bot. Whether it should is a per-process decision made on volume, reliability requirements and cost per run, and it belongs in an assessment of that process.
The measurement that settles it for a given process is a run under a step budget with external verification in the target system, repeated often enough to show the spread in cost and duration. Published harnesses do not stand in for that, and the distance between Browser Use's reported 89% and the 60.2% an independent evaluation measured shows how far those numbers travel. That is the test we would want run before anyone spends money replacing a bot that works, and we will keep reporting what we see as the figures move.