Anthropic details an experiment on whether AI coding tools shape developer skills: the biggest performance decline for developers occurred in debugging tasks
Read the paper — Research shows AI helps people do parts of their job faster. In an observational study of Claude.ai data, we found AI can speed up some tasks by 80%.
Anthropic
Context & Ripple Effects
AI coding assistance has long been framed as a route to faster, more accessible software work, but measured outcomes have been uneven. A prior METR evaluation of experienced open-source developers found participants slower with AI tools even as they believed they were faster.
Anthropic's own workforce had reported heavy Claude use, particularly for debugging and code understanding, alongside a self-reported productivity gain. This experiment puts a sharper question behind that adoption: whether speed on individual tasks can coexist with weaker performance in a core engineering skill.
First-order effects
The findings make debugging a focal risk area for developers and teams adopting Claude.ai or comparable coding assistants: faster task completion does not necessarily translate into stronger debugging performance.
Anthropic gains evidence to distinguish high-speed AI-assisted tasks from work where developers may need more deliberate verification and practice.
Second-order effects
Engineering leaders may evaluate coding tools by task type rather than using a single productivity metric, adding review, testing, or unaided debugging exercises where performance appears to deteriorate.
If repeated across teams and tools, AI-assisted development could shift software engineering toward a split workflow: automation for routine production, with debugging competence treated as a capability that must be actively maintained.
The durable measure of AI coding value may become cost per verified, maintainable outcome rather than code volume or nominal task speed; the evidence here is directional, not a conclusion about all developer work.
The trend: AI coding is moving from broad claims of developer acceleration toward task-level measurement of where assistance improves output and where it may erode capability.
Participants in the AI group finished faster by about two minutes (although this wasn't statistically significant). But on average, the AI group also scored significantly worse on the quiz—17% lower, or roughly two letter grades. [image]
AI can make work faster, but a fear is that relying on it may make it harder to learn new skills on the job. We ran an experiment with software engineers to learn more. Coding with AI led to a decrease in mastery—but this depended on how people used it. https://www.anthropic.com/…
Yep. AI makes code worst especially if it is not reviewed. Like AI create tremendous good work, but sometimes it fails stupidly and if not checked the bad cod accumulates and you have a non production ready code. It is good for MVPs, for POCs, but not for production. The young
📣Our study on how AI impacts coding skill formation is now out! (w. @AlexTamkin) AI Assistance is NOT a shortcut to skill formation. Using AI to help you learn to code only reduces mastery if you're delegating everything.
Moltbook is basically proof that AIs can have independent agency long before they become anything other than bland midwits that spout reddit/hustle culture takes. It's sort of the opposite of the yudkowskian or bostromian scenario where the infinitely smart and deceiving
In a randomized-controlled trial, we assigned one group of junior engineers to an AI-assistance group and another to a no-AI group. Both groups completed a coding task using a Python library they'd never seen before. Then they took a quiz covering concepts they'd just used. [imag…
These results have broader implications—on how to design AI products that facilitate learning, and how workplaces should approach AI policies. As we also continue to release more capable AI tools, we're continuing to study their impact on work—at Anthropic, and more broadly.
We were particularly interested in coding because as software engineering grows more automated, humans will still need the skills to catch AI errors, guide its output, and ultimately provide oversight for AI deployed in high-stakes environments.
AI is going to be the same for everyone, but our mindset determines the outcome. The most valuable skill to learn is to care about how things work, to be curious and to be okay with failing and getting stuck. Great work and research from Anthropic in reminding us! [image]
“We found that using AI assistance led to a statistically significant decrease in mastery.” — Props to Anthropic for studying the effects of their creation and reporting results that are not probably what they wished for — www.anthropic.com/research/AI- ...
Very interesting research paper that shows that using AI with programming can significantly reduce mastery over topics. Perhaps unsurprising, but the lack of significant speed gains in this exercise are remarkable — www.anthropic.com/research/AI- ... [image]
I found this myself recently, when I tried to use Claude to better understand a code base I'd written myself six months in an atrocious, pretentious style because I thought I was the only one who would use it. It took much longer than I'd thought necessary to build the mental mo…
This is both important and unsurprising. If you use AI to do something unfamiliar, you learn less and are not more productive. The weakest area was in finding bugs, exactly what is needed most. — Their previous research showing large productivity gains was with people who wer…
Really important research out of Anthropic: In a RCT study, they found AI coding assistance resulted in a 𝟏𝟕% 𝐝𝐫𝐨𝐩 in mastery for users. — While tasks were slightly faster, offloading thinking to AI stunted skill growth. …
We found that using AI assistance led to a statistically significant decrease in mastery (...) Using AI sped up the task slightly, but this didn't reach the threshold of statistical significance.
You do have to give Anthropic credit here. It is rare for a lab to publish data questioning its own tools. — Meta constantly buries internal findings that challenge their business model. This kind of transparency is uncommon and should be encouraged instead of dog-piled. — …
“As companies transition to more AI code writing with human supervision, humans may not possess the necessary skills to validate and debug AI-written code if their skill formation was inhibited by using AI in the first place.” [embedded post]
This lines up with my experience: those who use language models as unstructured conceptual search engines and intellectual foils do well and learn. Those who use them to do the work without judging their output produce rubbish and atrophy their own skills. www.anthropic.com/rese…