METR study: experienced open-source developers using Cursor, Claude, and other AI tools were 19% slower to complete tasks, despite thinking they were 20% faster
Devs estimated time saved with AI before & after each task, while screen recording to measured real-world AI productivity gains. @shimminykricket : A randomized controlled study found that Gen AI actually slows down the development time of experiences software developers for exactly the reason you'd expect. — www.reuters.com/business/ai- ... @andrewfray.dev : Using decent methods (but only 16 devs), this paper showed devs initially estimated AI tools would be a 24% speed up, after estimated it was 20%, but actually it had slowed them down 19%. — Don't get too excited, it's just one setting and one study. But it's interesting! — metr.org/blog/2025-07... Mastodon: Martijn Faassen / @faassen@hachyderm.io : This is a good review of what looks like a decent study of the impact of AI on productivity. In this study, devs thought it sped them up, but in actuality they were slowed down. The article goes in depth into various possible explanations. … Hailey / @hailey@hails.org : “When developers are allowed to use AI tools, they take 19% longer to complete issues—a significant slowdown that goes against developer beliefs and expert forecasts. This gap between perception and reality is striking: developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%.” https://metr.org/... X: @metr_evals : We ran a randomized controlled trial to see how much AI coding tools speed up experienced open-source developers. The results surprised us: Developers thought they were 20% faster with AI tools, but they were actually 19% slower when they had access to AI than when they didn't. [image] Elizabeth Barnes / @bethmaybarnes : 2. It's not easy to get current models to quickly write [high-quality code in mature codebases that meets stringent requirements]. We haven't totally ruled this out, but developers don't benefit from AI [starting from Cursor's basic setup] even with 30+ hours of practice. Ruben Bloom / @ruben_bloom : I was one of the developers in the @METR_Evals study. Thoughts: 1. This is much less true of my participation in the study where I was more conceintious, but I feel like historically a lot of my AI speed-up gains were eaten by the fact that while a prompt was running, I'd look at something else (FB, X, etc) and continue to do so for much longer than it took the prompt to run... Joe Weisenthal / @thestalwart : Um.. Domenic Denicola / @domenic : I was happy to help participate in this study, with the jsdom project as the codebase. You can see the issues, PRs, and experience reports from my work at https://github.com/.... I found it a really interesting experience, and will probably do a bigger writeup later! Elizabeth Barnes / @bethmaybarnes : 1. One clear takeaway is that expert forecasts and even user self-reports are not reliable indicators about AI capabilities. Actually Measuring Things is very valuable! John S. Dvorak / @johnsdvorak : Strikingly similar to the study that showed that people perceive themselves as more productive multi-tasking than they actually are, and similarly, people on “smart drugs” also perceive themselves as more productive. In both cases, they engage in more effort but get less done. @sadsciencee : since i joined shopify Ive been trying out a more cursor based flow since its really encouraged here my “senior dev” skills have improved a lot. but ive been feeling like my “get shit done” skills have atrophied a bit i used to be so fucking fast with just jetbrain autocomplete Rune Kvist / @runekvist : It's always interesting when people post results that are in tension with their own core beliefs. METR folks think superhuman AI is near, but find that open-source developers are slower when they use AI coding tools Adam Rodman / @adamrodmanmd : For what it's worth, this is the same finding we saw in our RCTs for clinical decision support — (significantly) increased physician time, with no- to-minimal performance gains. Much more to come! Elizabeth Barnes / @bethmaybarnes : The lack of speedup is a bit less surprising to me considering high developer experience with the codebases (>5 years and >1500 PRs on average) and high quality bar - anecdotally I've heard AI tools don't usually speed up experts in their core areas/on very familiar codebases LinkedIn: Yonas Bekele : 🚨For anyone thinking AI tools are always a shortcut to faster coding... think again. 🧠💻 … Adrian Booth : I'm glad there's a study that backs up empirically what many of us have experienced anecdotally when using AI tools. — https://lnkd.in/... Chris Baker : I've feared for a long time that AI will slow down the most experienced, productive and talented people in the workforce. … Sara Kubik : The TLDR version: when vibe coding produces code that breaks, it takes time to fix it. — My experience with this is the same. … Gerben Wierda : Interesting observation by Gary Marcus about AI-assisted coding. It is a narrow case (senior engineers in a well-known complex environment) … Fedir Skitsko : 🤯 METR study: AI makes expert developers 19% SLOWER — They tested 16 veterans on their own codebases (1M+ lines, 5+ years experience). … Forums: Hacker News : AI coding tools can reduce productivity r/technology : AI coding tools make developers slower but they think they're faster, study finds. r/BetterOffline : AI coding may not be helping as much as you think r/programming : Not So Fast: AI Coding Tools Can Actually Reduce Productivity r/singularity : Randomized control trial of developers solving real-life problems finds that developers who use “AI” tools are 19% slower than those who don't. r/theprimeagen : AI makes you slower r/slatestarcodex : METR finds that experienced open-source developers work 19% slower when using Early-2025 AI Lobsters : Not So Fast: AI Coding Tools Can Actually Reduce Productivity BeauHD / Slashdot : AI Slows Down Some Experienced Software Developers, Study Finds