What it is
Computer use is an AI model operating a computer the way you do: by looking at the screen, moving a mouse pointer, clicking, and typing. Instead of connecting to a program through a special back door built for software, it uses the same front door built for people, the windows, buttons, menus and text boxes you see every day.
Most AI tools that "do things" work differently. When a chatbot checks the weather or adds an event to your calendar, it usually calls an API, which is a doorway one program offers to another so they can exchange data directly and precisely. That's fast and reliable, but it only works where someone has built the doorway. A huge amount of software never got one: old accounting packages, in-house tools a company built in 2009, government portals, the scheduling screen at a dental practice, the settings panel of a printer. Computer use lets an AI work there anyway, because every program with a screen has, in effect, a human-shaped interface.
It works as a loop, and the loop is surprisingly simple to describe:
- Look. The software running the AI takes a screenshot, a picture of what's on the screen right now, and hands it to the model along with the task, such as "copy the totals from this spreadsheet into the invoice form".
- Decide. The model studies the picture, works out where things are, and chooses one small action: click at this spot, type this text, press Enter, scroll down.
- Act. The software carries out that action on the real (or virtual) computer, exactly as if a person had moved the mouse.
- Repeat. It takes a new screenshot to see what changed, and the model decides the next step. This continues until the job is done, the model gets stuck, or it hits a limit someone set.
If that sounds like the general pattern of an agent, it is. Computer use is agent behaviour with an unusually powerful set of tools: "look at the screen", "click here", "type this". Under the hood, those are ordinary tool calls of the kind described in Tool Use and Function Calling. The cycle of reasoning, acting, and checking the result has its own element, the ReAct Loop.
The hard part is the "decide" step. The model is seeing a picture, not a list of buttons, so it has to recognise that a grey rectangle with "Save" on it is a button, and then say exactly where to click, in pixels, which are the tiny dots that make up the screen. Anthropic wrote in 2024 that training its model to count pixels accurately was critical, because without it the model struggled to give correct mouse commands.
A note on scope. Many of the best-known products in this area only control a web browser, not the whole computer. Those have their own element, Browser Agents. This page is about the broader idea: an AI that can use any program on a desktop, including the file manager, the terminal, and software that has nothing to do with the web.
Why it matters
It reaches software that nothing else can. Businesses run on programs that will never get a modern API, and people spend hours shuttling information between them by hand: reading a number in one window and typing it into another. Computer use is the first kind of AI that can do that work without anyone rebuilding the software. Microsoft pitched exactly this when it announced computer use for its Copilot Studio agent builder in April 2025: agents that can work with any system that has a graphical interface, even where no API connection is available.
It's a fresh take on an old idea. Companies have automated screen work for years with a technique called robotic process automation, or RPA, where a script records someone's clicks and replays them. RPA is brittle: move a button or rename a menu and the script breaks. An AI that actually reads the screen can, in principle, cope when the layout changes, because it's looking for "the Submit button", not "the pixel at 840, 512".
It suits one-off chores. Writing a proper integration makes sense for a job that runs ten thousand times. It makes no sense for renaming 200 files once, or pulling figures out of an old program before you switch it off. Computer use is closest to handing a task to a temp: describe it in plain English and let it get on with it.
It's slow. Every step means taking a screenshot, sending it to the model, waiting for a decision, and acting. A task you'd do in thirty seconds of confident clicking can take the AI several minutes. Screenshots are also expensive in AI terms: each one uses a noticeable chunk of the model's context window, the amount of material it can hold in mind at once, so long sessions get costly and the model can lose track of earlier steps.
It's error-prone, and errors compound. Anthropic described its first release as "at times cumbersome and error-prone". The model can misread the screen, click a pixel or two off target, miss a pop-up that appeared and vanished between screenshots, or fail at actions people take for granted, such as dragging, scrolling to the right spot, or zooming. In a long task, one early mistake can send every later step down the wrong path. The models have improved sharply since 2024 (see the examples below), but none is flawless.
It's risky in a new way. An AI that can use your computer can do anything you can do on it: send email, delete files, make purchases, change settings. And because it reads whatever is on screen, the screen itself becomes a way to give it instructions. A web page, a document, or even an email subject line can contain text written to trick it, a problem called prompt injection. That's why every responsible guide to computer use starts with safety:
- Run it in a sandbox. A sandbox is a separate, walled-off computer, usually a virtual machine, which is a pretend computer running inside a real one. If the AI does something wrong there, you throw the virtual machine away. Anthropic's documentation recommends a dedicated virtual machine or container with minimal privileges.
- Keep secrets out of reach. No saved passwords, no logged-in banking, no personal files the task doesn't need. What the AI can't reach, it can't leak or misuse.
- Limit where it can go. Restricting internet access to a short list of approved sites reduces the chance it stumbles onto a malicious page.
- Make it stop and ask. Anything with real-world consequences, such as sending a message, buying something, accepting terms, or deleting data, should need a person's yes. That's the idea behind Human in the Loop, and it's one of the most useful Guardrails you can set.
Real-world examples
Anthropic releases computer use, warts and all (22 October 2024)
On 22 October 2024, Anthropic released computer use as a public beta for developers, alongside an upgraded Claude 3.5 Sonnet model. It was the first time a leading AI company offered a general-purpose model that could look at a screen, move a cursor, click buttons and type text through a public API. Anthropic said companies including Asana, Canva, DoorDash, Replit and The Browser Company had begun exploring it.
What made the launch unusual was how frank Anthropic was about the limits. It called the feature experimental, and in a companion post on how it developed the capability, it described mistakes from its own testing. While the team was recording demonstrations, Claude accidentally clicked to stop a long-running screen recording, losing all the footage. In another demo, it "suddenly took a break from our coding demo and began to peruse photos of Yellowstone National Park". The same post explained that Claude sees the screen as a "flipbook" of separate screenshots rather than a video, so it can miss short-lived notifications, and that it couldn't yet attempt common actions such as dragging and zooming.
The launch-day demo video showed the feature at its best: asked to fill in a vendor request form, Claude looked around the computer for the missing company details in another program, then filled in and submitted the form, one screenshot and one click at a time. The honest reading of October 2024 is that both were true. The idea worked, and it wasn't yet reliable enough to leave alone.
OSWorld: measuring how well AI can use a computer (2024 to 2026)
How do you tell whether an AI is any good at this? In April 2024, a team led by Tianbao Xie at the University of Hong Kong and collaborators published OSWorld, a test of 369 real computer tasks on Ubuntu Linux, Windows and macOS, such as working with spreadsheets, editing images, managing files, and tasks that span several applications. Each task runs in a virtual machine and is checked automatically. It became the standard benchmark for computer use.
The starting point was sobering. People completed about 72 percent of the tasks. The best AI system in the original paper managed about 12 percent, and the authors found the models struggled most with locating things on screen and knowing how to operate the software.
Then the scores climbed quickly. Anthropic reported 14.9 percent for Claude 3.5 Sonnet in October 2024 using screenshots only, and 22.0 percent when allowed more steps. In January 2025, OpenAI reported 38.1 percent for its Computer-Using Agent, the model behind its Operator product. In September 2025, Anthropic reported 61.4 percent for Claude Sonnet 4.5, noting that Claude Sonnet 4 had led four months earlier with 42.2 percent. By then the test had been cleaned up as OSWorld-Verified, released in July 2025 with improved tasks and grading. In February 2026, Anthropic's chart for Claude Sonnet 4.6 showed 72.5 percent, roughly the human figure from the original paper.
Two cautions go with those numbers. First, each score comes from the company that built the model, measured its own way. Second, the benchmark is not the job. Anthropic itself wrote that OSWorld is one of the best measures available but not a complete picture of real-world computer use, and that Sonnet 4.6 "still lags behind the most skilled humans". A task in a tidy test machine is not the same as your cluttered desktop with seventeen notifications.
ZombAIs: a web page takes over the AI (24 October 2024)
Two days after the launch, security researcher Johann Rehberger published a demonstration he called "ZombAIs". He set up a web page containing a short message asking the computer to download a file called Support Tool and launch it. The file was actually malware that would connect the machine to a server under his control, a setup known as command and control.
He then asked Claude, running Anthropic's computer use demo in an isolated test environment, to visit the page. Claude opened the browser, clicked the download link, searched for the file when it couldn't find it at first, made it executable, and ran it. The test computer joined Rehberger's control server. Writing about it the next day, the programmer and blogger Simon Willison noted that Anthropic had warned about exactly this risk, and that it was still jarring to see how easily the attack worked.
Rehberger pointed out that Anthropic had documented this risk openly, and that the problem is not unique to one company: any AI that reads untrusted content and can take actions can be tricked this way. It's the clearest possible argument for the sandbox rule. On a disposable virtual machine, ZombAIs is a lesson. On your work laptop, logged into everything, it's a breach. AI companies have since trained models to resist such tricks and added automatic detection; Anthropic's documentation, at the time of writing, says it flags likely injections in screenshots. Resistance has improved, but no vendor claims it's solved.
Microsoft brings computer use to business automation (2024 to 2025)
Microsoft researchers published UFO, an agent that operated Windows applications by reading the screen, in February 2024, one of several early research systems in this area. On 15 April 2025, the company announced computer use for Copilot Studio, its tool for businesses to build their own agents, as an early-access research preview. The examples it gave were deliberately mundane: entering large volumes of data into central systems, collecting market information from websites, and copying invoice details into accounting software. Microsoft said it would run on Microsoft-hosted machines rather than the customer's own computers.
That positioning tells you where computer use first earns its keep: not as a flashy personal assistant, but as a more flexible replacement for the screen-clicking scripts that companies already use to move data between old systems. As with any product, the details, availability and pricing change often, so check Microsoft's current documentation if you're considering it.
An everyday scenario: the old invoicing program
This is an illustration, not a real case. Marcus runs a small plumbing firm whose invoices live in a desktop program bought fifteen years ago. He's moving to a modern online system, and the old one has no export button. He sets up a computer-use agent on a spare virtual machine with a copy of the old program and nothing else: no email, no saved logins. He asks it to open each customer record and copy name, address and balance into a spreadsheet. It's slow, about a minute per customer, and on the first run it skips every record whose name contains an apostrophe. Marcus spots the gap because he'd asked it to report the total count, which didn't match. He tells it about the apostrophes, re-runs the missing ones, checks twenty records by hand, and has in an afternoon what would have taken him three evenings.
Try it yourself
You don't need to hand an AI your computer to understand computer use. This exercise is risk-free and takes about 45 minutes. The optional last steps are for readers who already have access to a computer-use feature.
- Watch the loop happen. Open Anthropic's launch demo, "Claude | Computer use for automating operations", on YouTube (linked in the resources below). It's a few minutes long. Watch it once straight through.
- Watch it again and take notes. This time, pause after every action and write down three things: what the AI could see, what it decided, and what it did. You'll see the look, decide, act, repeat cycle from this page play out step by step. Count the steps. Notice how many are just checking where things are.
- Spot the risky moments. Mark every step where a mistake would matter in real life: submitting a form, typing into a field that might be the wrong one, opening a new program. These are the moments a person should approve.
- Write your own safety checklist. Imagine you'll let an AI do one real chore on a computer, such as copying data out of an old program. Write down, before you start: which machine it runs on, what it must not be able to reach, which websites it may visit, which actions need your approval, and how you'll check the result.
- Ask a chatbot to attack your checklist. Paste this into any chatbot:
I'm planning to let an AI agent operate a computer by taking screenshots,
clicking and typing. The task is: [describe the chore].
Here is my safety checklist: [paste your checklist].
Act as a careful security reviewer. List the ways this could still go wrong,
including prompt injection from text on the screen, mistakes that compound
over many steps, and anything the agent could reach that it doesn't need.
Suggest specific changes to my checklist. Keep it practical for a
non-technical person.
- Read the answer critically. Keep suggestions that make sense for your chore and ignore the ones that don't. The aim is a checklist you'd actually follow.
- Optional: try it in a safe place. At the time of writing, several AI apps offer some form of computer or browser control, usually on paid plans or as a preview, and the details change often, so check each provider's current terms and requirements yourself. If you have access to one, try it only on a harmless task, on a machine or browser profile with no saved passwords and nothing personal logged in, and watch it the whole time. A good first task is "Open the calculator and add up these ten numbers", which lets you see how it finds things on screen without any risk.
- Notice the speed and the stumbles. Time how long it takes compared with doing it yourself, and note every hesitation or mis-click. That gut sense of how reliable it really is will serve you better than any benchmark number.
Common mistakes and misunderstandings
Thinking it's the best way to automate everything. Computer use is the slowest, most expensive and least reliable way for an AI to work with software. If a program has an API or an existing integration, use that. Computer use is for when there's no other door, or for one-off jobs where building one isn't worth it.
Running it on your everyday computer. Your own laptop is logged into email, banking, social media and cloud storage. A confused or tricked agent could reach all of it. Use a separate virtual machine or a dedicated, stripped-down account, as the AI companies' own documentation advises.
Giving it passwords. Don't type passwords into the task, and don't let it use a browser with saved logins. Many products are designed to hand control back to you when a login or payment appears; let them, and do those steps yourself.
Forgetting that anything on screen is input. The model reads every word visible on the screen and can mistake text written by a stranger, on a web page, in a document, in an email, for instructions from you. Treat any page it visits as potentially hostile, and never let it act on what it reads without limits.
Walking away. Early on, watch it. The failure modes are often silly rather than sinister, such as clicking the wrong row, getting distracted, or looping on the same step, but they're hard to spot from a final summary. Checking a count or a sample at the end, as Marcus did, catches the quiet mistakes.
Reading benchmark scores as a promise. A model that completes around 70 percent of test tasks still fails around 30 percent, and your tasks aren't the test's tasks. The scores show how fast the technology is improving, not how well it will handle your software.
Letting it approve its own risky actions. "Are you sure?" dialogs exist for people. An agent will often just click Yes. Set the rule at your end: anything that sends, buys, deletes, or agrees to terms needs your approval, every time.
Confusing computer use with browser agents. They overlap, but a browser agent only works inside a web browser, while computer use can operate any program on the machine, including the file system and the terminal. That extra reach is extra power, and extra risk.
Expecting it to remember. Each session usually starts fresh. If it learned last week that your old program hides the export button in a sub-menu, it won't know that today unless the product has some form of memory or you tell it again. Keep a short note of such tips and paste it into the task.
Related elements
- Agent: the broader idea of an AI that takes steps toward a goal, of which computer use is one kind.
- Browser Agents: the web-only cousin, covering products that browse and fill in forms inside a browser.
- Tool Use: screenshots, clicks and keystrokes are simply tools the model is allowed to call.
- Function Calling: the mechanism that turns the model's decision into an actual click or keystroke.
- ReAct Loop: the think, act, observe cycle that every computer-use session runs.
- API: the faster, sturdier alternative whenever the software offers one.
- Workflow Automation: the business automation that computer use extends to software without integrations.
- Guardrails: the limits and checks that keep an agent with this much reach out of trouble.
- Human in the Loop: why sends, purchases and deletions should wait for a person's approval.
- Benchmarks: how tests like OSWorld measure progress, and why their scores need caution.
- Memory: how an agent could carry lessons from one session to the next.
- Autonomy Levels: how much independence to give an agent, which matters most when it can use a whole computer.