At the Man vs. Machine hackathon, co-hosted by AI nonprofit METR to test if AI helps people code faster and better, the top prize went to an “AI-supported” team
roughly 100 people were randomly assigned “human” or “AI-supported” projects, and the winner nabbed a $12,500 cash prize. www.wired.com/story/san-fr...
Context & Ripple Effects
METR’s event turns the coding-assistant debate into a randomized, task-based comparison: about 100 participants were split between unassisted and AI-supported projects, with the latter taking the prize. The result is a useful outcome signal, though a single winner does not by itself establish the size or consistency of any productivity gain.
It joins a growing use of competitive settings to test AI capability, from public efforts to probe generative-AI flaws and bias to the Laude Institute’s multi-round AI coding challenge, where low answer accuracy underscored the difference between contest performance and dependable output.
First-order effects
- The AI-supported winning team receives the $12,500 prize, while the human-only teams have a concrete comparison point from the same event.
- METR and the co-hosts gain an observed contest outcome for their question of whether AI assistance improves coding speed and quality; the reported result alone does not isolate which factor drove the win.
Second-order effects
- Teams evaluating coding copilots are likely to put more weight on task-level, controlled trials rather than demonstrations or model claims, especially where quality matters alongside speed.
- Hackathons and challenges can become a feedback channel for tool builders: a winning AI-supported workflow raises demand for evidence about prompting, review practices, and error rates—not merely access to a model.
Third-order effects
- Software-AI evaluation is shifting from abstract benchmark scores toward human-plus-AI performance in realistic workflows, with reproducible study design becoming more important as adoption decisions move beyond experimentation.
- If such trials repeatedly show gains across teams and tasks, competition may increasingly center on lowering the cost of reliable AI-assisted coding and on workflow integration; mixed results would instead reinforce the need for human oversight and narrower deployment.
The trend: AI coding tools are being judged increasingly by measured human-AI workflow outcomes rather than standalone model capability.