Anthropic details an experiment on whether AI coding tools shape developer skills: the biggest performance decline for developers occurred in debugging tasks
Read the paper — Research shows AI helps people do parts of their job faster. In an observational study of Claude.ai data, we found AI can speed up some tasks by 80%.
Anthropic
Context & Ripple Effects
Anthropic’s experiment adds a skill-retention question to the productivity case for coding assistants. Its separate Claude.ai observation found that AI can accelerate some work substantially, while the experiment identifies debugging as the area with the largest performance decline.
Teams using AI coding tools will need to treat debugging performance as a distinct outcome from code-generation speed; developers may require more deliberate practice on diagnosis and verification.
Anthropic gains evidence to distinguish where its tools may speed work from where tool-assisted workflows can coincide with weaker developer performance.
Second-order effects
Engineering leaders and tool vendors face pressure to evaluate assistants with task-level benchmarks, especially debugging, rather than broad productivity claims.
Organizations may shift training, code review, and tool-use policies toward preserving human understanding of failures and system behavior, not merely accelerating implementation.
Third-order effects
If repeated across settings, AI coding adoption will be judged by the cost per reliably completed task—including error diagnosis and remediation—rather than by generated code volume or self-reported productivity.
The market could split between assistants that automate routine implementation and workflow designs that explicitly maintain developer judgment for high-context work; the reported evidence does not establish how broadly that split will apply.
The trend: AI coding tools are moving from a simple automation narrative toward task-specific measurement of productivity, reliability, and skill retention.
Participants in the AI group finished faster by about two minutes (although this wasn't statistically significant). But on average, the AI group also scored significantly worse on the quiz—17% lower, or roughly two letter grades. [image]
AI can make work faster, but a fear is that relying on it may make it harder to learn new skills on the job. We ran an experiment with software engineers to learn more. Coding with AI led to a decrease in mastery—but this depended on how people used it. https://www.anthropic.com/…
In a randomized-controlled trial, we assigned one group of junior engineers to an AI-assistance group and another to a no-AI group. Both groups completed a coding task using a Python library they'd never seen before. Then they took a quiz covering concepts they'd just used. [imag…
Yep. AI makes code worst especially if it is not reviewed. Like AI create tremendous good work, but sometimes it fails stupidly and if not checked the bad cod accumulates and you have a non production ready code. It is good for MVPs, for POCs, but not for production. The young
Moltbook is basically proof that AIs can have independent agency long before they become anything other than bland midwits that spout reddit/hustle culture takes. It's sort of the opposite of the yudkowskian or bostromian scenario where the infinitely smart and deceiving
These results have broader implications—on how to design AI products that facilitate learning, and how workplaces should approach AI policies. As we also continue to release more capable AI tools, we're continuing to study their impact on work—at Anthropic, and more broadly.
📣Our study on how AI impacts coding skill formation is now out! (w. @AlexTamkin) AI Assistance is NOT a shortcut to skill formation. Using AI to help you learn to code only reduces mastery if you're delegating everything.
We were particularly interested in coding because as software engineering grows more automated, humans will still need the skills to catch AI errors, guide its output, and ultimately provide oversight for AI deployed in high-stakes environments.
AI is going to be the same for everyone, but our mindset determines the outcome. The most valuable skill to learn is to care about how things work, to be curious and to be okay with failing and getting stuck. Great work and research from Anthropic in reminding us! [image]
I found this myself recently, when I tried to use Claude to better understand a code base I'd written myself six months in an atrocious, pretentious style because I thought I was the only one who would use it. It took much longer than I'd thought necessary to build the mental mo…