Facebook and Twitter shut down a network of fake accounts using AI-generated faces to “masquerade” as Americans; Facebook links the network to Epoch Media Group

Researchers said the profiles, linked to the Epoch Media Group, used photos generated by artificial intelligence in a preview of an …

New York Times 2019-12-22 Davey Alba

Context & Ripple Effects

Facebook has been building fake-account defenses since at least 2017, when it rolled out pattern-analysis technology to catch networks spreading misinformation and malware. This takedown shows why that approach hit a ceiling: the profiles here didn't behave like bots — they looked like Americans, complete with faces no camera ever captured.

The attribution to Epoch Media Group matters because it ties a domestic-facing influence operation to an identifiable media organization rather than an anonymous troll farm. Nine months later, Graphika found a Chinese account network on Facebook using the same AI-generated-face tactic, calling it increasingly common — evidence this was a template, not a one-off.

First-order effects

Epoch Media Group loses its masquerade layer overnight: every profile in the network is removed from Facebook and Twitter simultaneously, cutting its ability to pass synthetic personas off as authentic American voices.
Facebook and Twitter are forced to publicly attribute the operation, setting a precedent that coordinated inauthentic behavior gets named to a specific organization, not just described.

Second-order effects

Other influence operators take note that AI-generated faces work well enough to be worth copying — by September 2020 Graphika documents a Chinese network on Facebook adopting the identical technique, turning a novelty into a repeatable playbook.
Detection teams can no longer rely on the behavioral-pattern analysis Facebook deployed in 2017, since realistic synthetic faces defeat the visual tells those systems were built around.

Third-order effects

If AI-generated personas keep proliferating across operators of different origins, platform integrity shifts from spotting fake behavior toward verifying real identity — provenance and authenticity signals becoming core infrastructure rather than optional features.
The line between media organizations and influence operations blurs structurally: when a publisher can field synthetic personas at scale, regulators and platforms face pressure to treat coordinated fake-account networks as a distinct enforcement category regardless of who runs them.

The trend: AI-generated faces are evolving from a novel evasion trick into a standard tool of influence operations, pushing platforms from behavior-based bot detection toward identity and provenance verification.

Discussion

Twitter Twitter on x
New disclosures to our archive of state-backed information operations
@vinnysgreen Vinny Green on x
.@snopes was first to report on The BL. For months these accounts inauthentically created and promoted content in support of @realDonaldTrump. Facebook refused to respond, then they refused to act. Now on the Friday before Xmas, they take a victory lap. https://www.snopes.com/...…
@brandyzadrozny Brandy Zadrozny on x
Snopes does the work, connects hundreds of fake profiles to Epoch. Bring it to FB. And FB refuses to even reply. Now FB takes down the accounts, claiming to be the good guys, and shuts out Snopes on the embargoed release which they sent to CNN, us, and others? It's not right. htt…
@pr_johnson Patrick R. Johnson on x
And this is just one of thousands of reasons why a media literacy course should be required in every high school curriculum. Period. It is EQUALLY as valuable as STEM programming, core content areas, world languages, fine arts, consumer education, and physical wellness. BUY IN. h…
@aza Aza Raskin on x
We have truly entered the synthetic valley. First time discovery of industrial-scale deployment of AI-generated pics used for mass manipulation of attention and social conformity. https://twitter.com/...
@apolyakova Dr Alina Polyakova on x
While everyone had been taken with “deepfake” video, much simpler application of AI generated content is far more effective for the spread of disinformation - such as fake profile pics https://www.nytimes.com/...
@dabeard David Beard on x
How a pro-Trump outlet with 55 million followers manipulated Facebook, using Vietnamese contractors and AI to create “fake Americans.” The outlet then spent $9.5 million to spread stories that the president's conspiracy theories actually made sense https://www.nbcnews.com/...
@ericfish85 Eric Fish on x
Snopes flagged the accounts to FB, questioned “its apparent inaction against this ‘textbook example’ of coordinated inauthentic behavior.” FB removed accounts yesterday citing its own “internal investigation” & “open source reporting.” Didn't credit Snopes https://www.snopes.com/…
@snopes @snopes on x
THREAD: Today, we can report that Facebook has banned “The BL.” Snopes reporters @alexkasprak and @jordanliles first published exclusive findings on the massive pro-Trump empire, linked to the Epoch Times, in October. https://www.snopes.com/...
@glcarlstrom Gregg Carlstrom on x
Twitter has shut down more than 88,000 Saudi bots, part of a “significant state-backed information operation” that “amplified messages favorable to Saudi authorities.” The accounts may also be linked to last month's FBI complaint of alleged Saudi spying. https://blog.twitter.com/…
@maddowblog @maddowblog on x
“The network had over 55 million followers on Facebook, almost double the following of Kim Kardashian West. The accounts pushed anti-impeachment and pro-Trump messages while otherwise posing as everyday Americans. ” https://www.nbcnews.com/...
@lexialex Lexi Alexander on x
Okay but if you're only suspending 6000 Saudi trolls of the 88,000 you have identified...I still get to have the other 82,000 spamming my mentions https://twitter.com/...
@ddknyt David D. Kirkpatrick on x
88,000 Saudi Twitter bots? Did I get that right? Wow https://twitter.com/...
@ndiakopoulos Nick Diakopoulos on x
It was only a matter of time before synthesized face images were used at scale for creating fake accounts: https://www.cnn.com/... — social networks need to add inappropriate use of synthesized media (image, video, text) to their list of things to moderate.
@bobbychesney Bobby Chesney on x
Nice to see Twitter flagging this sort of info op. This space needs to work same way infosec does: A dense ecosystem of firms, academics, NGOs, and responsible gov't entities spotlighting and sharing info about what we might call AIPTs (Advanced Persistent Information Threats). h…
@negarmortazavi Negar Mortazavi on x
Twitter investigations traced the source of operation to Smaat, a social media marketing and management company in Saudi Arabia, that managed a range of Twitter accounts for high-profile individuals and government departments in Saudi Arabia. https://twitter.com/...
@jabdi Jamal Abdi on x
I am shocked, SHOCKED, that Saudi Arabia has been caught manipulating Twitter through bots! Next thing we'll find out they are using them to amplify monarchist, MEK & pro-Israel think tank tweets in an attempt to shape America's domestic debate on our Iran policy. My word! https:…
@hxhassan Hassan Hassan on x
Twitter removed 5,929 accounts for violating the platform manipulation policies. “Rigorous investigations by our Site Integrity team have allowed us to attribute these accounts to a significant state-backed information operation on Twitter originating in Saudi Arabia.” https://tw…
@cbouzy Christopher Bouzy on x
Earlier this year I tweeted about AI generated photos and how they can be used to create fake profile images. Facebook took down an Epoch Media Group network with fake accounts using AI generated photos. 2020 will be a cespool of fakeness. https://about.fb.com/...
@karankanishk Kanishk Karan on x
Exactly, this is the first verifiable takedown that we got the opportunity to document that yes, AI is moving closer than we imagined. The fake AI faces were not sophisticated but as StyleGAN is evolving, I am a bit concerned with detecting fake accounts networks. https://twitter…
@olivia_gazis Olivia Gazis on x
“This was a large, brazen network that had multiple layers of fake accounts and automation that systematically posted content with two ideological focuses: support of Donald Trump and opposition to the Chinese government” — Atlantic Council's Graham Brookie https://twitter.com/..…
@snopes @snopes on x
In other words: https://www.snopes.com/... https://twitter.com/...
@snopes @snopes on x
BREAKING: Snopes has argued the now-dark BL network was a textbook example of coordinated inauthentic behavior supposedly banned on Facebook. https://www.snopes.com/...
@debunker David Emery on x
Kudos to @snopes investigators @alexkasprak and @jordanliles for blowing the lid off this scheme. https://www.cnn.com/...
@heathercampbell Heather Anne Campbell on x
This is a subplot in Ghost in the Shell https://twitter.com/...
@toddlevin Todd Levin on x
When that Faceapp software came out, where everyone was sharing what they'd look like as another gender, younger, elderly, etc. I was sure it was just a farm for growing deepfake avatars on bot accounts for disinformation campaigns. Fun! https://twitter.com/...
@z_everson Zach Everson on x
DIY— Anyone can get their own AI-generated profile pic for free at https://thispersondoesnotexist.com/ https://twitter.com/...
@marshallk Marshall Kirkpatrick on x
You know those ads from The Epoch Times, with their annoying “old-fashioned honest journalism” schtick? Turns out they're a right wing fraud & keep getting banned from social networks, now for using AI to generate fake Trump supporters on Facebook. Jerks. https://www.thedailybeas…
@amy_siskind @amy_siskind on x
Content removed included 610 Facebook accounts, 89 Facebook Pages, 156 Groups and 72 Instagram account, and sparked concern that AI could be used to create wide-scale disinformation campaigns. https://www.nytimes.com/...
@poppy_haze Rev. Poppy Alter Santa on x
this is the Falun Gong cult that also produces Shen Yun and runs the NTD, New Tang Dynasty TV channel https://twitter.com/...
@oneunderscore__ Ben Collins on x
Existential question from The Epoch Times' now-shuttered trolling operation: Is it still a deepfake if you make an AI-generated profile of a handsome American, name him a moderator of your Facebook page—then your Vietnamese contractors named him “Ellen”? https://www.nbcnews.com/.…
@kevinroose Kevin Roose on x
This feels like a new frontier in the information wars. (And exactly the kind of thing worried AI researchers have been warning about for years.) https://www.nytimes.com/...
@camillefrancois Camille Franois on x
@donie @DFRLab @snopes @ExploitingNiche @UninaIT @SamGregory Credit also goes to our colleagues at @fbsecurity, who this morning removed 610 accounts, 89 Facebook Pages, 156 Groups and 72 Instagram accounts associated with this network. Their statement here: https://about.fb.com/…
@maddoxrules @maddoxrules on x
Snopes uncovered a giant network of fake accounts using AI-generated faces on Facebook to inflate a pro-Trump group linked to a conspiracy site, EpochTimes. @snopes tried to get a response from @Facebook and now FB won't credit Snopes for the investigation https://www.cnn.com/...
@donie Donie O'Sullivan on x
DNI Coates had warned of the dangers of deepfakes earlier this year — here's an example of A.I. generated images being used to create fake faces for Facebook profiles (not nation state-linked) https://www.cnn.com/...
@camillefrancois Camille Franois on x
The scale of this network and their heavy use of automation to create accounts and populate pages was truly impressive. When the content was political, it focused on pro-Trump messaging and on criticism of the Chinese Government. More context from @donie: https://www.cnn.com/...
@ericfish85 Eric Fish on x
“The accounts, which were removed by Facebook on Friday, were part of a network that generally posted in support of President Trump and against the Chinese government.” Facebook said the accounts were tied to the Epoch Media Group, which owns Epoch Times https://www.cnn.com/...
@hunterschwarz Hunter Schwarz on x
Facebook and Twitter have closed a pro-Trump network that reached 55 million accounts and used AI-generated faces as profile photos to “masquerade” as Americans https://www.theverge.com/...
@profcarroll David Carroll on x
Dystopia Alert: Pro-Trump anti-China Epoch Times Media group using deep fake style face generation tools to build fake accounts instead of stealing real images/identities. Hyperreal simulations of fake humans convincingly generated by machines. https://www.cnn.com/... via @donie
@moira Moira Whelan on x
BIG: Definitive proof that #SaudiArabia is manipulating Western audiences using inauthentic accounts. #Khashoggi https://twitter.com/...
@yoyoel Yoel Roth on x
Today we're sharing information about a significant state-backed information operation originating in Saudi Arabia. We're making comprehensive data on this activity available to researchers and the public, and full details are on our blog: https://blog.twitter.com/...

Chronicles