Anthropic releases a new Claude 3.5 Sonnet model that can interact with desktop apps by imitating mouse and keyboard input via a “computer use” API, now in beta
In a pitch to investors last spring, Anthropic said it intended to build AI to power virtual assistants that could perform research …
TechCrunchKyle Wiggers
Context & Ripple Effects
Anthropic had already positioned Claude as an interactive work surface through Artifacts that let users work with Claude’s output, alongside the June Claude 3.5 Sonnet launch. This release extends that direction from generating and displaying results to operating existing desktop software through an API.
The capability also foreshadows Anthropic’s later move to put computer use into its desktop products in research preview, including Claude Cowork and Claude Code on macOS. The arc is from model capability to a more persistent, application-level assistant.
First-order effects
Developers can test Claude 3.5 Sonnet as an interface operator, using mouse and keyboard-style actions to carry out tasks in desktop applications rather than relying only on purpose-built integrations.
Anthropic broadens Claude’s product surface from chat and generated artifacts to agentic workflows, while keeping the computer-use API in beta.
Second-order effects
Software vendors and enterprise teams gain a potential automation route for tools without dedicated APIs, but must evaluate reliability and controls when an AI acts through a graphical interface.
Competing AI providers are pushed to pair model performance with practical computer-control capabilities, shifting differentiation toward whether assistants can complete multi-step work in existing software.
Third-order effects
If computer-use interfaces mature, automation may increasingly be delivered at the work-surface layer—across existing applications—rather than requiring each vendor to build a separate AI integration.
The trade-off is likely to make operational safeguards a defining feature of agent platforms: broader access to desktop workflows raises the importance of permissioning, oversight, and recoverability.
The trend: This is part of the shift from conversational AI toward agentic work surfaces that can act across the tools people already use.
Introducing an upgraded Claude 3.5 Sonnet, and a new model, Claude 3.5 Haiku. We're also introducing a new capability in beta: computer use. Developers can now direct Claude to use computers the way people do—by looking at a screen, moving a cursor, clicking, and typing text. [im…
Beyond inspiring to see the different Anthropic teams come together for this launch — our upgraded Sonnet is even better at coding and other tasks, and we have a beta computer API for you to try as well.
this is extremely huge for people who need accessibility. this is in line with what I said on primeagen's stream yesterday: AI should enable a large swathe of people that were previously disconnected, due to language or disability
Computer use API We've built an API that allows Claude to perceive and interact with computer interfaces. You feed in a screenshot to Claude, and Claude returns the next action to take on the computer (e.g. move mouse, click, type text, etc). [image]
listen I think one reason lots of people are intrigued by prediction markets is there's a couple of scenarios where stuff like this means online polling is deeply screwed.
I'm excited to share what we've been working on lately at Anthropic. - Computer use API - New Claude 3.5 Sonnet - Claude 3.5 Haiku Let's walk through everything: [image]
This is a fundamentally new capability for AI models. Instead of developing bespoke tools for individual tasks, we're teaching Claude base computer skills—enabling it to naturally use the same everyday software and tools that people use.
Claude 3.5 Sonnet's current ability to use computers is imperfect. Some actions that people perform effortlessly—scrolling, dragging, zooming—currently present challenges. So we encourage exploration with low-risk tasks. We expect this to rapidly improve in the coming months.
We've built an API that allows Claude to perceive and interact with computer interfaces. This API enables Claude to translate prompts into computer commands. Developers can use it to automate repetitive tasks, conduct testing and QA, and perform open-ended research. [video]
We're trying something fundamentally new. Instead of making specific tools to help Claude complete individual tasks, we're teaching it general computer skills—allowing it to use a wide range of standard tools and software programs designed for people. [video]
The new Claude 3.5 Sonnet is the first frontier AI model to offer computer use in public beta. While groundbreaking, computer use is still experimental—at times error-prone. We're releasing it early for feedback from developers. [video]
Beyond computer use, the new Claude 3.5 Sonnet delivers significant gains in coding—an area where it already led the field. Sonnet scores higher on SWE-bench Verified than all available models—including reasoning models like OpenAI o1-preview and specialized agentic systems. [ima…
Claude 3.5 Haiku is the next generation of our fastest model. Haiku now outperforms many state-of-the-art models on coding tasks—including the original Claude 3.5 Sonnet and GPT-4o—at the same cost as before. The new Claude 3.5 Haiku will be released later this month. [image]