Analysis: Google's C4 dataset, used to train LLMs like Meta's LLaMA, has troubling content from 4chan, Kiwi Farms, white supremacist site Stormfront, and more
AI chatbots have exploded in popularity over the past four months, stunning the public with their awesome abilities … Tweets: @goldman , @timmarchman , @daveleebbg , @nitashatiku , @aaronjschaffer , @justinhendrix , @maxkennerly , @justinhendrix , and @blackamazon Tweets: Jason Goldman / @goldman : Very cool article on where the training data comes from. Bot needs to read more wowhead tho https://twitter.com/... Tim Marchman / @timmarchman : The internet-reading robot your boss wants you to partner up with has apparently, with the history of human civilization at its disposal, been reading “5 Reasons ‘The Marvels’ Will Be The Best Movie Ever (And Five Reasons Why It Will Suck)” https://www.washingtonpost.com/ ... https://twitter.com/... Dave Lee / @daveleebbg : Fascinating look at the more-than-iffy datasets behind AI https://www.washingtonpost.com/ ... @nitashatiku : Here's our analysis of the 15 million websites in just one highly-filtered CommonCrawl web scrape-used to train models like Google's T5 & Facebook's LLaMA -copyright symbol appears >200M times -pirated sites, 1 for e-books -half the top 10 = news sites https://www.washingtonpost.com/ ... Aaron Schaffer / @aaronjschaffer : Google's C4 data set, which has been used to instruct high-profile AIs including Google's T5 and Facebook's LLaMA, includes Russian propaganda site RT, anti-immigration site VDARE, white supremacist site Stormfront, anti-trans site Kiwifarms and 4chan https://www.washingtonpost.com/ ... Justin Hendrix / @justinhendrix : Thinking about the old phrase “garbage in, garbage out” https://www.washingtonpost.com/ ... Max Kennerly / @maxkennerly : I don't recall giving Google, Facebook, or anybody else permission to scrape 180k tokens—roughly the length of Orwell's 1984—from my blog for commercial purposes. And putting my copyrighted work in a blender with bigots doesn't make it better. https://twitter.com/... https://twitter.com/... Justin Hendrix / @justinhendrix : Holy cow. “The Post's analysis suggests more legal challenges may be on the way: The copyright symbol — which denotes a work registered as intellectual property — appears more than 200 million times in the C4 data set.” https://twitter.com/... @blackamazon : BUT RACISM ISN'T BUILT IN THOUGH RIGHT We had all these people and “experts” who SWORE they were gonna “listen” and doing all this bs smart talk But sure just takes years to figure that out And the way they kept blocking critics of racism in tech wasn't a clue https://twitter.com/...