Missing data hindering replication of AI studies as survey of 400 algorithms presented at major conferences finds just 6% included code, 30% included test data
Last year, computer scientists at the University of Montreal (U of M) in Canada were eager to show off a new speech recognition algorithm … Tweets: @hanno , @jorendorff , @silverjacket , @meaningness , and @mattmight Tweets: @hanno : This detail of the recent article about AI+replication tells you one thing: Just trying to replicate isn't enough, replications need to be preregistered and published regardless of outcome. http://www.sciencemag.org/... http://twitter.com/... Every_jorendorff / @jorendorff : “Researchers say there are many reasons for the missing details: The code might be a work in progress, owned by a company, or held tightly by a researcher eager to stay ahead of the competition.” you're all fired http://twitter.com/... Matthew Hutson / @silverjacket : Artificial intelligence faces reproducibility crisis. My story in this week's @sciencemagazine: http://www.sciencemag.org/... David Chapman / @meaningness : AI doesn't replicate. Having worked in the field, I can usually see why a paper's result is nonsense, but the public can't, and many researchers can't. http://twitter.com/... Matt Might / @mattmight : This looks like a job for the CRAPL: http://matt.might.net/... Papers which don't release code and data shouldn't be published. No exceptions. End of story. This is a huge embarrassment for the field of computer science. http://twitter.com/...