Internal docs: Facebook employees created a test account in 2019 and within days it was recommended extreme and conspiratorial content, including QAnon Groups

In the summer of 2019, a new Facebook user named Carol Smith signed up for the platform, describing herself as a politically conservative mother …

NBC News 2021-10-23 Brandy Zadrozny

Context & Ripple Effects

The test-account finding sits alongside other internal evidence of Facebook’s knowledge of extremist Groups, while a 2020 analysis found that QAnon could still be amplified by the platform’s recommendations despite Facebook’s announced crackdown. Facebook later tested prompts asking users about possible extremist exposure, indicating that the company was also exploring user-facing interventions.

First-order effects

The documents give Facebook a concrete internal example of its recommendations steering a newly created conservative-profile account toward QAnon and other extreme material within days.
Facebook’s earlier anti-QAnon enforcement stance is harder to separate from the behavior of its recommendation system, which related coverage found was still aiding QAnon’s reach.

Second-order effects

Facebook’s extremist-content prompts face a sharper product-design problem: warning users about exposure does not address recommendation paths that may create that exposure.
The account test adds weight to scrutiny of whether Facebook’s moderation and recommendation teams acted on internal evidence, a gap the related documents do not fully resolve.

Third-order effects

If internal tests repeatedly identify harmful recommendation pathways, platform-safety debates shift from removing individual Groups to governing the ranking systems that connect users to them.
The pattern points toward accountability standards centered on what platforms know from their own experiments and how quickly that knowledge changes distribution systems.

The trend: Platform safety is increasingly being judged by the behavior of recommendation systems, not only by the enforcement actions platforms publicly announce.

Discussion

@brandyzadrozny Brandy Zadrozny on x
New from me: New internal documents show Facebook has long known—via years of experimentation and research—that its algorithms and recommendation systems push users to extremes. To researchers they gaslit, “Facebook literally helped facilitate a cult.” https://www.nbcnews.com/...
@ttp_updates @ttp_updates on x
THREAD: For months, TTP monitored and recorded threats from militia and far-right actors online In January, we released what we gathered in that time: militia recruiting, threats to public officials, even calls for martyrdom—all posted openly on Facebook https://www.techtranspare…
@oneunderscore__ Ben Collins on x
New from the Facebook leaks, by @BrandyZadrozny: Facebook staffers created a Trump-loving, Fox News-watching fake identity. The account went wherever the algorithm told it to go. It went full-on QAnon in two days. Facebook told nobody about this. https://www.nbcnews.com/...
@jesselehrich Jesse Lehrich on x
full QAnon in 2 days. & Facebook scoffs at anyone who suggests their algos are problematic. https://twitter.com/...
@melissaryan Melissa Ryan on x
So many reporters and researchers have run similar experiments over the years. Always with the same results. Every time Facebook would say the experiment was flawed and that wasn't actually how the platform worked. Every. Single. Time. https://twitter.com/...
@robpegoraro Rob Pegoraro on x
“a barrage of extreme, conspiratorial, and graphic content”: how one of Facebook's own researchers described the path of a test user created in 2019 with tastes in politics, parenting, Christianity, Fox News and Trump to see how Facebook's recommendation algorithms would respond.…
@jeffhorwitz Jeff Horwitz on x
This story gets into important territory about the continuity between different conspiracy movements — and how one engenders the next. Also, the renaming of a tool called “Rabbithole” as “Drebbel” makes more sense now that I know it's the name of the inventor of the submarine. ht…
@karenkornbluh Karen Kornbluh on x
Shocking new details. Confirms that government has been derelict in not acting. There were always ways to regulate *the design* upstream from the content - it was never about the “truth police.” Now so much harm has been done https://twitter.com/...
@juliacarriew Julia Carrie Wong on x
I just want to scream https://twitter.com/...
@evilgalprods @evilgalprods on x
Every. Single. Word. https://twitter.com/... https://twitter.com/...
@wiczipedia Nina Jankowicz on x
Important reporting from @BrandyZadrozny. Critically, as she notes, many researchers were confirming and re-confirming the danger of Group recommendations for years before Facebook took action. So much wasted time. https://twitter.com/...
@eliotwb Eliot Brown on x
Facebook exec (and a Harvard pal of Zuckerberg) recommends curbs to virality to stave off bad effects Head of ads responds: “'If we remove a small percentage of reshares from people's inventory,' he wrote, ‘they decide to come back to Facebook less.’” https://www.wsj.com/...
@fchollet @fchollet on x
FB's algorithmic mind-steering has only one stable attractor: extreme radicalization. https://twitter.com/...
@peterwsinger Peter W. Singer on x
This may be one of the most damning of the Facebook leaks... https://twitter.com/...
@davidoatkins David Atkins on x
Facebook is poison. It has always been poison. And they knew it. https://twitter.com/...
@jimhansondc Jim Hanson on x
That's fun What happened when they made a woke identity? Wait They never did that? Hmmm https://twitter.com/...
@cydharrell Cyd Harrell on x
couple more things: 1. some of the resistance to user research is because we find inconvenient things that people would rather not know (@mulegirl talks about this a lot) 2. researchers do a service by documenting, but need to be realistic about our chances of changing anything h…
@davidoatkins David Atkins on x
All the big journo influencers on twitter are beating on each other over Popularism, when THIS is the answer. People, especially low-trust voters, get their news on Facebook. Facebook is driving them to right-wing conspiracy theories. Policy and activist slogans are irrelevant. h…
@edzitron Ed Dead Redemption on x
The years and years and years of the media taking Facebook at their word and giving them puff press was a bad idea https://twitter.com/...
@zeynep Zeynep Tufekci on x
“The body of research consistently found Facebook pushed some users into “rabbit holes,” increasingly narrow echo chambers where violent conspiracy theories thrived.” https://twitter.com/...
@anneapplebaum Anne Applebaum on x
“Facebook pushed some users into “rabbit holes,” increasingly narrow echo chambers where violent conspiracy theories thrived...at Facebook's scale, that can mean millions of individuals.' https://www.nbcnews.com/... via @nbcnews
@c_c_krebs Chris Krebs on x
Important reporting including insights from @noUpside. I'm convinced Covid-related drop in community-centric social activities (kids sports, offices, etc) lowered barriers to crazytown & accelerated radicalization along multiple conspiracy theories. https://twitter.com/...
@amir Amir Efrati on x
Looks like the piece we published may have ended the embargo? In case, first Facebook consortium member out of the gate is the great @BrandyZadrozny from NBC: https://twitter.com/...
@grimlyspoken Grimm on x
@oneunderscore__ @BrandyZadrozny What happened in Myanmar is a literal precursor to what is happening here. Facebook watched an ethnic group get slaughtered, did nothing, and now had facilitated and even pushed a homegrown domestic threat in the United States. This article is gre…
@clearing_fog ClearingTheFog on x
Please read this. So many of us have been shouting for years that Facebook is *knowingly* facilitating extremism, and yet they've continued to get a pass in our media, as if they were somehow “trying.” Their own internal documents repeatedly prove that they knew. https://twitter.…
@jaxalemany Jacqueline Alemany on x
Facebook has yet to fully comply with requests for data from the House select committee investigating the 1/6 events. Read @craigtimberg, @lizzadwoskin, @ReedAlbergotti on FB & Jan. 6 > https://www.washingtonpost.com/ ... https://twitter.com/...
@resistbot @resistbot on x
“That researcher said Smith's Facebook experience was ‘a barrage of extreme, conspiratorial, and graphic content.’” https://www.nbcnews.com/...
@jbadomics @jbadomics on x
“...this damage control goes to eleven.” https://twitter.com/...
@timmarchman Tim Marchman on x
Incredible. Facebook internally reproduced the outside reporting that has been done on it, showing that the way its products were built promoted QAnon and the anti-vax movement in ways that fed on one another, and did ... nothing. https://twitter.com/...
@jeffhorwitz Jeff Horwitz on x
This story with @ScheckWSJ shows how Facebook's platforms might be different - if it wanted them to be. We know the controversies over content moderation - per @masnick, it can't be done well at scale. (The “at scale” part might be optimistic, though). https://www.wsj.com/...
@blackamazon @blackamazon on x
Only white sources. No mention of the reports on racial overtones or targeting. Makes the entire story about how “white people are mislead” And not that Facebook let itself be hunting grounds for people of color for YEARS Gotta love the “learning” going on from white news https:/…
@cwarzel Charlie Warzel on x
this facebook internal experiment reminds me of this 2017 @broderick piece! https://www.buzzfeednews.com/ ... https://twitter.com/...

Chronicles