Internal docs: a Facebook study about US users' vaccine hesitancy finds “substantial harm” from content that doesn't violate the company's rules

Internal study finds a QAnon connection and that content that doesn't break the rules may be causing ‘substantial’ harm

Washington Post 2021-03-15 Elizabeth Dwoskin

Context & Ripple Effects

Facebook’s internal research had already identified a recommendation problem: roughly 70% of its most active civic Groups were deemed too toxic to recommend. The vaccine-hesitancy study extends that concern beyond overt rule-breaking, tying potentially harmful exposure to content Facebook had already judged unsuitable for recommendation.

The finding became part of a wider accountability trail. A later DC attorney general subpoena for Facebook’s COVID-misinformation records specifically sought material on vaccine hesitancy, while Facebook’s eventually released transparency report acknowledged the reach of vaccine-doubt content.

First-order effects

Facebook’s existing rules leave a documented category of vaccine-related content that may harm US users’ attitudes without triggering enforcement, sharpening the gap between policy compliance and health impact.
The study’s QAnon connection gives Facebook a concrete internal signal that vaccine hesitancy was linked to a broader content ecosystem rather than isolated posts.

Second-order effects

Facebook faces pressure to adjust distribution and recommendation decisions, not just removals, because its own earlier research had found highly active civic Groups too toxic to recommend.
Regulatory scrutiny gains a more specific evidentiary target: Facebook’s internal assessment of how permitted content affected vaccine hesitancy, later reflected in the DC AG’s document request.

Third-order effects

If platforms treat measurable harm from permitted content as a governance problem, content moderation shifts from a binary rules-enforcement model toward accountability for ranking, recommendations, and comment-driven distribution.
The pattern points toward transparency obligations centered on what platforms know internally about reach and harm, not solely on the volume of content they remove.

The trend: Platform governance is expanding from policing prohibited misinformation to examining how recommendation systems amplify harmful but policy-compliant content.

Discussion

Vox Rani Molla on x
Who isn't getting vaccinated, and why — A new report highlights why different groups …
@glichfield Gideon Lichfield on x
With all the pushback from Facebook against @_KarenHao 's recent story, and as one of the editors on the piece, I thought it worth making some observations on the PR strategy Facebook has adopted in response. Other journalists may find this useful.
@ashleevance Ashlee Vance on x
My inclination has been to think that the extent of the Facebook brainwashing effect is over-hyped. Now, however, my 87-year-old dad and 73-year-old mum are refusing the vaccine 100 percent based on stuff they saw on Facebook, and, um, I have some concerns
@lizzadwoskin Elizabeth Dwoskin on x
SCOOP: Facebook is going beyond vaccine misinfo to conduct a major DL study of vaccine doubters, new docs show. Finds “substantial harm” from stuff that doesn't break the rules. Just 111 people are behind the worst comments. https://www.washingtonpost.com/ ...
@capuletpoehner Capulet Poehner on x
@lizzadwoskin @Shayan86 These past years I've watched in disgust as former friends and colleagues became radical anti-vaxxers on Facebook and even more on Instagram. Those platforms are optimized to spread mass hysteria at scale
@qorigins @qorigins on x
Oh, word? Facebook CAN look into the wildfire spread of lies on its platform if it wants to, but just hasn't wanted to until now? Huh. Didn't see that coming. 🙄 https://twitter.com/...
@noupside Renee DiResta on x
Internal FB research discussed in this article: “Its early findings suggest that a large amount of content that does not break the rules may be causing harm in certain communities, where it has an echo chamber effect.” Yes. https://www.washingtonpost.com/ ...
@alkapdc Alex Kaplan on x
It is not a coincidence that one of the most influential anti-vaxxers on Facebook (before he was banned) became a QAnon supporter. There has been an increasing overlap between anti-vaxxers and QAnon supporters. https://twitter.com/...
@jameshohmann James Hohmann on x
Massive Facebook study on users' doubt in vaccines finds a small group appears to play a big role in pushing the skepticism. Internal study finds a QAnon connection and that content that doesn't break the rules may be causing “substantial” harm. https://www.washingtonpost.com/ ..…
@mattdoyle76 Matthew Doyle on x
Never really thought something could be worse for society than 24/7 cable news but Facebook's lapped them twice in the past five years. https://twitter.com/...
@bricey16 Bryan Rice on x
Please we need to throw Facebook into the Pacific Ocean https://twitter.com/...
@aslavitt Andy Slavitt on x
If you have questions about taking a vaccine, look elsewhere. https://www.washingtonpost.com/ ...
@mollyjongfast @mollyjongfast on x
Oh you don't say...... https://twitter.com/...
@jason_kint Jason Kint on x
Unreal. This doesn't happen because of “free expression.” It's a result of algorithms micro-targeting harmful content (and ads) to vulnerable users - providing concentrated amplification without counter speech - tuned for maximum Facebook profits. /1 https://www.washingtonpost.co…
@dr_scottk Scott Krugman on x
Seems like @Facebook could do something with the 111 accounts responsible for the majority of anti-vaccine disinformation.... and many of us probably know who they are already.... at least name them and label them, right? #VaccinesWork https://twitter.com/...
@davidaxelrod David Axelrod on x
“The research effort also discovered early evidence of significant overlap between communities that are skeptical of vaccines and those affiliated with QAnon, a sprawling set of baseless claims that has radicalized its followers...” https://www.washingtonpost.com/ ...
@markruffalo Mark Ruffalo on x
It's real. @Facebook is pushing vaccine misinformation for money. They are willful leading people to death and illness and hampering recovery efforts. https://twitter.com/...
@alkapdc Alex Kaplan on x
“The research effort also discovered early evidence of significant overlap between communities that are skeptical of vaccines and those affiliated with QAnon.” https://twitter.com/...
@colgo Paul Colgan on x
Good data point showing vaccine hesitancy is pushed by a tiny group https://twitter.com/...
@jason_kint Jason Kint on x
Facebook leadership always prioritizes policy and communications to protect its profit model and advertising business instead of protecting their user base. ICYMI, this is important context around last week's blockbuster report on related matters. /4 https://twitter.com/...
@tlowdon Don Wolt on x
I remember when “pushing skepticism” was generally considered to be a good thing, instead of a Scarlett Letter crime. https://twitter.com/...
@russellsieg Russell on x
Yet again, Facebook is studying super important stuff all by themselves (Vaccine Hesitancy) and it's pretty much a black box of unknown implications to our Democracy We'll find out way too late that Facebook's massive reach is a danger to society. https://www.washingtonpost.com/ …
@jason_kint Jason Kint on x
These 2 paragraphs are critical context: * The world depends on Facebook * but Facebook cannot be trusted As regulators and lawmakers have yet to deal with the core. Including US FTC and SEC's pressure/decision to settle for $5+B rather than expose Facebook leadership. /2 https:/…
@katestarbird Kate Starbird on x
Repeatedly, and across contexts, researchers are finding that a small number of dedicated disinformers are doing a significant amount of damage to the health of discourse, democracy, and society at large. https://twitter.com/...
@sam64754068 Sam on x
@davidaxelrod They are killing people. One of my patients was diagnosed with COVID, called it a “hoax”, left against medical advice, spread it all over the community for 3 days, then died in our ER....I have never been so angry in my professional life
@jason_kint Jason Kint on x
To properly deal with these issues requires new leadership and new biz model: unwinding Facebook's profit model (ubiquitous data collection across our lives in order to maximize profits from an experience feeding users a balance of micro-targeted ads and user content). /3
@grandmagrit Grandma Grit on x
A study done by Facebook traced most of the disinformation about the Covid vaccine back to just 111 people, who had links to Qanon. It shows how influential our voices can be, for good and ill. I think Trump is patient zero for Covid denial. #COVIDIOT https://www.washingtonpost.c…
@qorigins @qorigins on x
This article is a good reminder that most QAnon conversation really IS driven by a small number of influencers. We don't just focus on them for convenience. On Q-focused subreddits, for instance, 80% of the content came from 5% of users (see p. 10 below). https://arxiv.org/... ht…
@donie Donie O'Sullivan on x
“Covid vaccine victims” is the 10th result when I typed “vaccine” into Instagram right now.
@caseynewton Casey Newton on x
“Harm from non-violating content may be substantial,” Facebook researchers find with posts expressing vaccine hesitancy. In how many other areas of Facebook is this true? https://www.washingtonpost.com/ ...
@frankluntz Frank Luntz on x
“Facebook's data scientists divided their U.S. users, groups, and pages into 638 different population segments Some of the early findings are notable: Just 𝟭𝟬 out of the 638 population segments contained 𝟱𝟬% of all vaccine hesitancy content on the platform.” https://twitter.com/.…
@rjcc Richard Lawler on x
If only everyone had told them https://twitter.com/...
@jkr_on_the_web @jkr_on_the_web on x
@MollyJongFast “Herd Immunity” https://twitter.com/...
@shannonpareil Shannon Bond on x
“in the population segment with the most vaccine hesitancy, just 111 users contributed half of all vaccine hesitant content.” This seems to happen frequently on Facebook: posts from a small # of people have disproportionate effects https://www.washingtonpost.com/ ...
@donie Donie O'Sullivan on x
A Facebook study finds the obvious. https://www.washingtonpost.com/ ...
@hypervisible @hypervisible on x
Facebook conducting more internal research they can ignore. https://www.washingtonpost.com/ ...
@timnitgebru Timnit Gebru on x
Very informative thread. Like I said before the manner in which the execs and PR ppl responded also undermines the work that many are trying to do inside the company. That response alarmed me more than the actual article itself. https://twitter.com/...
@clarajeffery Clara Jeffery on x
This piece by (former MoJo fellow) @_KarenHao is outstanding as is this thread by @glichfield. Both add to our understanding of how deeply malignant and duplicitous Facebook is. https://twitter.com/...
@aaschapiro Avi Asher-Schapiro on x
Facebook gave @_KarenHao access to their “responsible AI” team hoping, it seems, for a puff piece. When she wrote a critical story, FB PR went into high gear-this thread, from her editor @glichfield, explains how FB tried to undermine the reporting. https://twitter.com/...
@elonmusk Elon Musk on x
@ashleevance For sure wise for elderly or immunocompromised to take the vaccine. Some debate about the second jab though. Quite a few negative reactions to that.
@alexvoica Alexandru Voica on x
Today, we're expanding our COVID-19 Information Center to Instagram, adding labels on posts about COVID-19 vaccines to show additional information from @WHO, and making real-time trends in COVID-19 vaccinations available to public officials: https://about.fb.com/... https://twitt…
@fbnewsroom @fbnewsroom on x
We're all eager to put COVID-19 behind us, and vaccines are our best bet to get there. So today Mark shared our plan to help bring 50 million people one step closer to getting vaccinated. https://about.fb.com/...

Chronicles