Scientists criticize a lack of transparency in AI research, citing issues of replication in research and unequal access to code, proprietary data, and hardware
with dangerous consequences. Replication by different researchers would expose problems sooner, making AI stronger for everyone” #NLProc stop publishing what is not replicable https://www.technologyreview.com/ ... Delia Ferreira / @deliaferreira : Algorithmic transparency is essential. Lack of transparency in AI may have dangerous consequences👇 #Algocracy https://twitter.com/... @techreview : But according to its critics, the Google team provided so little information about its code and how it was tested that the study amounted to nothing more than a promotion of proprietary tech. @techreview : Google was describing successful trials of an AI that looked for signs of breast cancer in medical images. https://www.nature.com/... @techreview : “When we saw that paper from Google, we realized that it was yet another example of a very high-profile journal publishing a very exciting study that has nothing to do with science,” says @bhaibeka. “It's more an [ad] for cool technology. We can't really do anything with it.” @techreview : Replication also allows others to build on those results, helping to advance the field. Science that can't be replicated falls by the wayside. @techreview : Science is built on a bedrock of trust, which typically involves sharing enough details about how research is carried out to enable others to replicate it, verifying results for themselves. This is how science self-corrects and weeds out results that don't stand up. @techreview : Haibe-Kains and his colleagues are among a growing number of scientists pushing back against a perceived lack of transparency in AI research. @techreview : “We couldn't take it anymore,” says @bhaibeka, the lead author of the response, who studies computational genomics at the @UofT. “It's not about this study in particular—it's a trend we've been witnessing for multiple years now that has started to really bother us.” @techreview : But in fields like biology and physics—and computer science overall—researchers are typically expected to provide the information needed to rerun experiments, even if those reruns are rare. @techreview : In practice, few studies are fully replicated because most researchers are more interested in producing new results than reproducing old ones. @techreview : Last month @nature published a damning response written by 31 scientists to a study from Google Health that had appeared in the journal earlier this year. https://www.nature.com/... @techreview : Tech giants dominate AI research but the line between real breakthrough and product showcase can be fuzzy. Some scientists have had enough. https://www.technologyreview.com/ ... @facebookai : We're committed to open research. As Joëlle Pineau, Managing Director of Facebook AI Research highlights in this @techreview article, open and reproducible research benefits the entire community. https://twitter.com/... Mike Schroepfer / @schrep : Joëlle Pineau, Managing Director of Facebook AI Research: “For the last few years, she has been the driving force behind a change in how AI research is published” https://www.technologyreview.com/ ... Great article - Joelle is pushing the whole industry towards open and reproducible research @techreview : This is a thread about how artificial intelligence is wrestling with a replication crisis. https://www.technologyreview.com/ ...
This has also been an issue in other computing/engineering areas though recently there has been more emphasis on reproducibility: “What's stopping AI replication from happening as it should is a lack of access to three things: code, data, and hardware. ”. https://www.technologyre…
“machine-learning models that work well in the lab can fail in the wild—with dangerous consequences. Replication by different researchers would expose problems sooner, making AI stronger for everyone” #NLProc stop publishing what is not replicable https://www.technologyreview.com…
But according to its critics, the Google team provided so little information about its code and how it was tested that the study amounted to nothing more than a promotion of proprietary tech.
“When we saw that paper from Google, we realized that it was yet another example of a very high-profile journal publishing a very exciting study that has nothing to do with science,” says @bhaibeka. “It's more an [ad] for cool technology. We can't really do anything with it.”
Science is built on a bedrock of trust, which typically involves sharing enough details about how research is carried out to enable others to replicate it, verifying results for themselves. This is how science self-corrects and weeds out results that don't stand up.
“We couldn't take it anymore,” says @bhaibeka, the lead author of the response, who studies computational genomics at the @UofT. “It's not about this study in particular—it's a trend we've been witnessing for multiple years now that has started to really bother us.”
But in fields like biology and physics—and computer science overall—researchers are typically expected to provide the information needed to rerun experiments, even if those reruns are rare.
Last month @nature published a damning response written by 31 scientists to a study from Google Health that had appeared in the journal earlier this year. https://www.nature.com/...
Tech giants dominate AI research but the line between real breakthrough and product showcase can be fuzzy. Some scientists have had enough. https://www.technologyreview.com/ ...
We're committed to open research. As Joëlle Pineau, Managing Director of Facebook AI Research highlights in this @techreview article, open and reproducible research benefits the entire community. https://twitter.com/...
Joëlle Pineau, Managing Director of Facebook AI Research: “For the last few years, she has been the driving force behind a change in how AI research is published” https://www.technologyreview.com/ ... Great article - Joelle is pushing the whole industry towards open and reproduci…