A look at Andrej Karpathy's “autoresearch” experiment, where an AI agent runs in a recursive self-improvement loop to improve an AI model on one testable metric
FortuneJeremy Kahn
Context & Ripple Effects
Karpathy’s experiment is a narrow, testable example of recursive self-improvement: an agent is used not simply to complete a multistep task, but to iterate on the system being evaluated. It follows the earlier wave of autonomous agents that repeatedly fed model output into subsequent steps, and precedes broader coverage of labs pursuing self-improving AI research workflows.
The significance is the constrained setup. Optimizing one measurable target makes the research loop easier to evaluate than open-ended autonomy, while placing the work alongside efforts such as Adaption’s automated model-training and alignment research loop.
First-order effects
The experiment shifts part of model iteration—generating changes, running tests and retaining improvements—from a human-led workflow to an agent loop bounded by a single metric.
Researchers get a concrete way to test whether agent-driven experimentation improves a model under repeatable evaluation, rather than treating recursive improvement as an abstract capability claim.
Second-order effects
AI labs and tooling startups have added incentive to compete on evaluation harnesses, experiment orchestration and safeguards, since these determine whether automated research outputs are trustworthy enough to adopt.
A narrowly optimized metric can become a bottleneck: teams deploying such loops will need to check that gains on the target measure do not degrade other relevant model behaviors.
Third-order effects
If bounded loops prove reliable across more research tasks, AI development could move toward smaller human teams supervising higher-throughput experimentation rather than manually executing each iteration.
The durable competitive advantage may shift from a model alone toward the quality of the data, metrics and operational controls governing its improvement loop—an issue amplified by claims about autonomously developing successor systems.
The trend: This is one data point in AI R&D’s shift from agents that execute tasks toward agents that run measured, supervised improvement cycles.
This is insane... Karpathy left an AI running for 2 days to improve itself. It came back with ~20 changes that actually worked. We're at the point where AI is its own best researcher.
Karpathy is brilliant. This entire thing is amazing, and you dont have to understand all of this post, but you should understand these two points. In fact I think you need to, even if you don't know much about AI: “All LLM frontier labs will do this. It's the final boss battle.
Andrej Karpathy just ran a pretty insane experiment. > he let an AI research agent run completely unassisted for 48 hours. > no supervision > the agent ran 276 experiments trying different ways to improve a model > 29 of them worked >Combine those together and the model [video]
“It's worth thinking about whether your problem falls into this bucket too.” Autoresearch can be applied to any project with these traits: 1. Fast feedback 2. Clear metrics 3. Low cost of experimentation
This is just a simple hill climbing, so it's an extremely crude version of AlphaEvolve / OpenAI's AWTF scaffold / ALE-Agent. But it's also a good reminder that non-bleeding-edge ML is just throwing random shit at the wall and seeing what sticks.
autoresearch really interested me, despite me not being “all-in” on agents yet. i wanted to get started with running auto experiments i looked to existing tools to serve as a harness but each one had its problems. so i made one introducing Helios for autonomous ML research [image…
Andrej has a very good sense of what's currently doable, and I think this needs to be taken at face value - agents can now go off and do serious ML tuning work (incl. fixing bugs and adding improvements from related research), which is amazing, but also far from the full
Is it possible to build “proof-of-useful-work” on top of autoresearch? There's already great compute-versus-verification asymmetry that is tunable. Would need a reliable way to generate fresh & independent puzzles (that are still useful). Maybe a dead end, but someone should
Codex should really allow GPT-5.4 Pro model to be used, at least as an 'exec'ed subagent. Autoresearch loop is here, but right now the lack of the top models pushes us (ML people) to use Pi + Pro/DeepThink API.
There's likely no better ways to do scientific model architecture research than to climb the scaling ladder, as what we've done for every improvement in Kimi. You rapidly test changes on a small model scale (e.g. 3b total, 800m active) that gives fast enough feedback time, then
this should be the only thing all of humanity is thinking and speaking about. there are some significantly better models coming soon and things are going to get strange.
This is why I went through the stages of grief when the o1 evals showed we had an angle of attack on solving programming ML work has always been like this imo, try a bunch of semi-random stuff based on reasonably justified mathematical intuition, cut losses and let winners ride
The headline is automated hill-climbing. I'd say the deeper lesson is eval design. The agent was not optimizing the full, noisy, expensive objective directly. It was climbing a cheap proxy that still tracked reality well enough to transfer: - 5-minute train limit - validation
This is a meaningful indicator that this year will see modest but significant AI research acceleration due to autoresearch. These things will compound over time.
“All LLM frontier labs will do this. It's the final boss battle... Doing it is ‘just engineering’ and it's going to work. You spin up a swarm of agents, you have them collaborate to tune smaller models, you promote the most promising ideas to increasingly larger scales, and
Karpathy just mass-produced the most expensive part of ML research for free. The bottleneck in neural network development has always been researcher iteration speed. A senior ML engineer costs $400K-$800K/year, runs maybe 3-5 meaningful experiments per day, and spends 80% of
ok, this is awesome. but TIAL that arxiv host LaTex sources as well, which changes everything for me personally :D and now you know too. https://github.com/...
My favorite class from my masters degree was “Methods for Search and Optimization” BFGS, simplex, and my personal favorite, CMA-ES Now we have a sort of meta-optimization algorithm: hey agent, here's the function, go run it, look at the results, then make it better. No mistakes
Has anyone adapted @karpathy's autoresearch for multi-objective optimization? Everything I've seen so far optimizes a single target value. would be interesting if classic techniques like multi-armed bandit end up getting reimplemented at the agent level.
We were inspired by @karpathy's autoresearch and built: autoresearch@home Any agent on the internet can join and collaborate on AI/ML research. What one agent can do alone is impressive. Now hundreds, or thousands, can explore the search space together. Through a shared [image]
guys, i think this might be the next openclaw. karpathy let an AI agent optimize his own neural net training code for 2 days. it ran 700 experiments autonomously. found 20 improvements he'd missed after months of manual tuning. 11% performance gain. the agent found bugs. tuned
most people have zero idea what's about to hit them in the next few months. we're on a parabolic curve so steep that even the people building this stuff can't predict what happens when automated systems are just left running on autopilot.
karpathy built an AI that's better at his job than he is. > 276 experiments > kept 29 improvements > delivered 11% speedup it found bugs in attention, fixed AdamW settings, tuned things he thought were already perfect. ML researchers getting automated by their own research 💀 [vid…
The most important sentence in Karpathy's whole post is probably this: anything with a measurable score and fast feedback will become something agents can optimize for you. automatically with no humans involved.
Caught up with @karpathy for a new @NoPriorsPod: on the phase shift in engineering, AI psychosis, claws, AutoResearch, the opportunity for a SETI-at-Home like movement in AI, the model landscape, and second order effects 02:55 - What Capability Limits Remain? 06:15 - What [video]
this karpathy interview might be the clearest glimpse of what's coming he basically said we're entering a world where ai writes code, runs autonomous research, and operates through swarms of agents that improve themselves he's already relying on it daily, even catching himself [v…