Missing data is hindering replication of AI studies as survey of 400 algorithms presented at major conferences finds just 6% had code and 30% included test data
Context & Ripple Effects
This 2018 survey put hard numbers on a problem AI researchers had mostly discussed anecdotally: across 400 algorithms presented at major conferences, only 6% shipped code and only 30% included test data, making most published claims unverifiable by anyone outside the original lab. It was an early quantification of what later coverage named the reproducibility crisis in AI research.
The arc since then has been persistence rather than resolution: by late 2020 scientists were still publicly criticizing the field's lack of transparency, citing replication failures and unequal access to code, proprietary data, and hardware (MIT Technology Review). Later audits extended the problem from algorithms to the datasets underneath them — an analysis of 1,800 AI datasets found roughly 70% lacked clear licenses or were mislabeled more permissively than intended.
First-order effects
- Researchers trying to build on the surveyed work cannot verify or extend it: with 94% of algorithms lacking code and 70% lacking test data, replication requires reconstructing methods from prose alone.
- Conference organizers and reviewers face direct pressure to treat code and test data as required artifacts rather than optional extras, since the survey names major conferences as the venue where unreplicable work is being accepted.
Second-order effects
- Labs and companies that keep code, data, and hardware closed gain a verification advantage over academic competitors — the unequal access dynamic that 2020 transparency criticism flagged becomes a competitive moat, not just an inconvenience.
- Dataset provenance becomes its own audit surface: once replication fails on missing test data, attention shifts downstream to whether training corpora are even legally and geographically representative, the territory later mapped by the 1,800-dataset license analysis.
Third-order effects
- If the pattern holds, AI publishing converges on enforced artifact evaluation — code, test data, and dataset licensing as gate conditions — turning reproducibility infrastructure into shared critical infrastructure rather than a per-lab courtesy.
- The alternative path is stratification: a tier of well-resourced industrial labs whose results are verifiable only internally, with the open research commons thinning out — the structural risk implied by the access complaints running through the 2020 coverage.
The trend: AI research is moving from paper-only publication toward enforced reproducibility standards, with the gap between what conferences accept and what outsiders can verify narrowing only under sustained audit pressure.