New Facebook test prompt asks users if someone they know might be “becoming an extremist”; another notifies users who may have been exposed to extremist content

CNN 2021-07-03 Oliver Darcy

Context & Ripple Effects

Facebook had already used in-product prompts for harm response, from suicide-prevention links to trained experts to Safety Checks activated around confirmed crises. The extremism test extends that intervention model from immediate personal safety and events to users’ content exposure and social networks.

Its earlier trust surveys also show Facebook using user feedback to shape information-quality decisions. Here, the platform is testing whether users can help identify warning signs around extremist material and behavior.

First-order effects

Users selected for the test may receive an exposure alert or be asked to flag concern about someone they know, creating new reporting and referral signals for Facebook.
Facebook gains a way to test intervention prompts alongside its existing content-distribution and moderation systems, rather than relying solely on removal or group-level enforcement.

Second-order effects

Moderation teams must determine how to handle user-generated concern signals, including separating credible reports from mistaken or abusive referrals.
The test puts greater weight on Facebook’s ability to identify exposure to extremist content, making recommendation and distribution pathways part of the safety response rather than only the reporting flow.

Third-order effects

If Facebook retains such prompts, platform safety work shifts further toward preventative, behavior-oriented interventions that act before content is formally removed.
The approach would make social platforms more responsible for detecting and responding to perceived pathways into harmful communities, not merely policing individual posts.

The trend: Social platforms are expanding safety systems from reactive content enforcement toward prompts and signals meant to intervene earlier in users’ exposure and social connections.

Discussion

@sanjanah Sanjana Hattotuwa on x
Will this work in countries where extremists are elected to govern & normalise violence through political office? (Hint: It won't.) | #Facebook tests prompts that ask users if they're worried a friend is ‘becoming an extremist’ https://cnn.com/...
@scalzi John Scalzi on x
I've seen friends on Facebook talking about (and mostly mocking) this, in reference to some of their other FB friends, but the friends who have seen it have all been friends who I know are conservative/libertarian, which is an interesting correlation https://www.theverge.com/...
@ross Ross Mayfield on x
Anyone who lived in autocracy like the USSR or studied it recognizes this for what it is immediately. Will divide friends, neighbors, even families. And deploying it at global scale to avoid accountability for harming democracy by regulators would do even greater dystopian harm. …
@sub8u Subrahmanyam Kvj on x
The @newyorker has a nice cartoon for this from some time ago 😂 https://twitter.com/... https://twitter.com/...
@ross Ross Mayfield on x
Relationship Status: so complicated I'm turning them in. NextDoor must copy this Soviet feature to realize the monetization gains of properly channeled outrage. https://twitter.com/...
@disclosetv @disclosetv on x
NEW - Looks like Facebook's AI is now starting to “think” about your exposure to “harmful extremist content” and your “extremist” friends. https://twitter.com/...
@strassenkaempfe Tuk on x
it's like when pornhub asks you if you're over 18 lmaooooo https://twitter.com/...
@alexberenson Alex Berenson on x
Yeah, I'm becoming an extremist. An anti-@Facebook extremist. “Confidential help is available?” Who do they think they are? Either they're a publisher and a political platform legally liable for every bit of content they host, or they need to STAY OUT OF THE WAY. Zuck's choice. h…
@rachblevins Rachel Blevins on x
Why in the cornbread hell is Facebook asking me if I'm concerned about someone I know becoming an extremist? I'm much more concerned about the world's largest social media site censoring journalists and removing alternative media that challenge the US Gov't narrative... https://t…
@chadloder Chad Loder on x
This is the weakest possible response to what has become Facebook's most profitable demographic. Facebook is YEARS behind. https://twitter.com/...
@alt_uscis ALT-immigration on x
Facebook tests prompts that ask users if they're worried a friend is ‘becoming an extremist’ Facebook will have to hire every out of work adult to deal with this @brooklynmarie https://www.cnn.com/...
@asranomani Asra Q. Nomani on x
Facebook is following the playbook of Mao's Cultural Revolution. From K-12 to social media, we now have snitch squads being created, turning people against each other and creating the platforms to turn each other in as “extremists.” https://twitter.com/...
@sharylattkisson @sharylattkisson on x
(Facebook knows, because Facebook is the one pushing the propaganda.) (Language warning) https://twitter.com/...
@t_s_p_o_o_k_y Tony Shaffer on x
When did @Facebook become an extension of the @FBI? How is this legal? https://twitter.com/...
@realerincruz Cruz For on x
Dear Facebook, I am concerned communists/socialists are out of control & are daily infringing on our lives as a free people. Their extremist behavior is costing millions of lives via abortion funding, neglect of regular care for cancer & other patients during COVID panD. @RepMTG …
@chrisblec Chris Blec on x
Today, @Facebook told thousands of people that they may be extremists. These people had no idea they could be extremists before this. Now, they're thinking “Hmm. Maybe I *am* an extremist! Maybe I should start *acting* like one!” Is there not one sane person left at Facebook? htt…
@oliverdarcy Oliver Darcy on x
Facebook spox confirms to me that this is a small test it is running: “This test is part of our larger work to assess ways to provide resources and support to people on Facebook who may have engaged with or were exposed to extremist content, or may know someone who is at risk...”…
@tprstly Theo Priestley on x
It's becoming easier to shop your friends and neighbours to the thought police. https://twitter.com/...
@patrickmoorhead @patrickmoorhead on x
Sounds like what happened in East Germany in the 70s with the Stasi. Sleep well everybody. https://twitter.com/...
@johnwilson John Wilson on x
So creepy. Is the point to snitch on your friends? What happens next? Law enforcement becomes involved, too? https://twitter.com/...
@kingfishfan @kingfishfan on x
Every time I open the Facebook app now I get that notification that I “may have been exposed to extremist content” when the funny part is I never even post anything on FB and only ever interact to like ppl's baby and animal pics and sometimes recipes. So that's not creepy at all
@cnnbusiness @cnnbusiness on x
Some US Facebook users are being served a prompt that asks if they are worried someone they know might be becoming an extremist. It's part of a test that stems from a company initiative aiming to combat violent extremism, a Facebook spokesperson says. https://www.cnn.com/...
@oliverdarcy Oliver Darcy on x
Facebook is directing users to a variety of resources, including Life After Hate, an advocacy group that helps people leave violent far-right movements. Here's our full story: https://www.cnn.com/...
@alexberenson Alex Berenson on x
Sorry. “Others in your situation have received confidential support.” Which is somehow even more infuriating, with its patronizing tone.
@fboversight @fboversight on x
“Are you feeling like an insurrectionist? Tell Facebook today!” https://twitter.com/...
@drscottjensen Scott Jensen on x
Have gotten numerous reports about this. When everyone is an “extremist”, no one is. It's a huge reason our campaign isn't taking a DIME from big tech. @Facebook https://twitter.com/...
@minnie_mag Mini on x
Build a system that constantly manipulates with peoples emotions to drive engagement Encourage inflaming outrage bait because it makes money Radicalize boomers so they'll click on your ads ???? Oops we should probably do something about grandpa now that we turned him into a nazi …
@hamburger Ellis Hamburger on x
do you know someone that is addicted to our products? we have videos about that, but we won't stop hosting, recommending, and monetizing the extremist groups 😁 https://twitter.com/...
@sirhottest @sirhottest on x
Facebook thinks they're yo momma now https://twitter.com/...
@primalpoly Geoffrey Miller on x
Facebook: 'You're angry and disappointed' People: ‘No, not really’ Facebook: ‘Yes, you really are. We are here to protect you from your anger and disappointment’ People: 'No thanks. I'm cool. ' Facebook: ‘WE WILL PROTECT YOU LIKE IT OR NOT’ https://twitter.com/...
@mattbraynard Matt Braynard on x
If you *didn't* get this message, you're probably doing something wrong. WORK HARDER! https://twitter.com/...
@rightwingcope Right Wing Cope on x
BREAKING: Facebook is implementing a new system to combat the spread of extremist content on their app https://twitter.com/...
@riptari @riptari on x
Facebook is warning Facebook users about Facebook? https://twitter.com/...

Chronicles