Facebook and Twitter shut down a network of fake accounts using AI-generated faces to “masquerade” as Americans; Facebook links the network to Epoch Media Group

Researchers said the profiles, linked to the Epoch Media Group, used photos generated by artificial intelligence in a preview of an …

New York Times 2019-12-21 Davey Alba

Context & Ripple Effects

Two years after Facebook began rolling out pattern-analysis tech to catch fake accounts, the platform and Twitter hit a network whose profiles didn't behave like bots at all — they wore AI-generated human faces and posed as Americans. Facebook's attribution of the network to Epoch Media Group makes this the first high-profile case where a US-based media organization is publicly tied to synthetic-identity personas rather than a foreign operation.

First-order effects

Epoch Media Group loses its account presence on both platforms overnight, with Facebook's public attribution turning a routine enforcement action into a reputational charge against a named publisher.
Researchers gain an early, documented specimen of GAN-generated profile photos used for persona creation — a preview of what detection teams will face at scale.

Second-order effects

Platform detection has to move beyond the behavioral-pattern analysis Facebook deployed in 2017 toward image-level forensics, since these personas passed as ordinary Americans until researchers looked at the faces themselves.
The tactic spreads: less than a year later, Graphika documented a Chinese state-linked network on Facebook using the same AI-generated-face method, calling it an increasingly common evasion technique.

Third-order effects

If synthetic faces become cheap commodity tooling for any influence operation — domestic media groups and state networks alike — account authenticity can no longer rest on 'looks like a real person', pushing platforms toward provenance and verification layers across the whole content stack.

The trend: AI-generated identities are shifting from research curiosity to standard-issue infrastructure for coordinated inauthentic behavior, forcing platform moderation to evolve from behavior analysis to synthetic-media detection.

Discussion

Twitter Twitter on x
New disclosures to our archive of state-backed information operations
@snopes @snopes on x
THREAD: Today, we can report that Facebook has banned “The BL.” Snopes reporters @alexkasprak and @jordanliles first published exclusive findings on the massive pro-Trump empire, linked to the Epoch Times, in October. https://www.snopes.com/...
@jabdi Jamal Abdi on x
I am shocked, SHOCKED, that Saudi Arabia has been caught manipulating Twitter through bots! Next thing we'll find out they are using them to amplify monarchist, MEK & pro-Israel think tank tweets in an attempt to shape America's domestic debate on our Iran policy. My word! https:…
@ndiakopoulos Nick Diakopoulos on x
It was only a matter of time before synthesized face images were used at scale for creating fake accounts: https://www.cnn.com/... — social networks need to add inappropriate use of synthesized media (image, video, text) to their list of things to moderate.
@lexialex Lexi Alexander on x
Okay but if you're only suspending 6000 Saudi trolls of the 88,000 you have identified...I still get to have the other 82,000 spamming my mentions https://twitter.com/...
@bobbychesney Bobby Chesney on x
Nice to see Twitter flagging this sort of info op. This space needs to work same way infosec does: A dense ecosystem of firms, academics, NGOs, and responsible gov't entities spotlighting and sharing info about what we might call AIPTs (Advanced Persistent Information Threats). h…
@negarmortazavi Negar Mortazavi on x
Twitter investigations traced the source of operation to Smaat, a social media marketing and management company in Saudi Arabia, that managed a range of Twitter accounts for high-profile individuals and government departments in Saudi Arabia. https://twitter.com/...
@hxhassan Hassan Hassan on x
Twitter removed 5,929 accounts for violating the platform manipulation policies. “Rigorous investigations by our Site Integrity team have allowed us to attribute these accounts to a significant state-backed information operation on Twitter originating in Saudi Arabia.” https://tw…
@glcarlstrom Gregg Carlstrom on x
Twitter has shut down more than 88,000 Saudi bots, part of a “significant state-backed information operation” that “amplified messages favorable to Saudi authorities.” The accounts may also be linked to last month's FBI complaint of alleged Saudi spying. https://blog.twitter.com/…
@cbouzy Christopher Bouzy on x
Earlier this year I tweeted about AI generated photos and how they can be used to create fake profile images. Facebook took down an Epoch Media Group network with fake accounts using AI generated photos. 2020 will be a cespool of fakeness. https://about.fb.com/...
@ddknyt David D. Kirkpatrick on x
88,000 Saudi Twitter bots? Did I get that right? Wow https://twitter.com/...
@maddowblog @maddowblog on x
“The network had over 55 million followers on Facebook, almost double the following of Kim Kardashian West. The accounts pushed anti-impeachment and pro-Trump messages while otherwise posing as everyday Americans. ” https://www.nbcnews.com/...
@olivia_gazis Olivia Gazis on x
“This was a large, brazen network that had multiple layers of fake accounts and automation that systematically posted content with two ideological focuses: support of Donald Trump and opposition to the Chinese government” — Atlantic Council's Graham Brookie https://twitter.com/..…
@karankanishk Kanishk Karan on x
Exactly, this is the first verifiable takedown that we got the opportunity to document that yes, AI is moving closer than we imagined. The fake AI faces were not sophisticated but as StyleGAN is evolving, I am a bit concerned with detecting fake accounts networks. https://twitter…
@snopes @snopes on x
In other words: https://www.snopes.com/... https://twitter.com/...
@snopes @snopes on x
BREAKING: Snopes has argued the now-dark BL network was a textbook example of coordinated inauthentic behavior supposedly banned on Facebook. https://www.snopes.com/...
@debunker David Emery on x
Kudos to @snopes investigators @alexkasprak and @jordanliles for blowing the lid off this scheme. https://www.cnn.com/...
@toddlevin Todd Levin on x
When that Faceapp software came out, where everyone was sharing what they'd look like as another gender, younger, elderly, etc. I was sure it was just a farm for growing deepfake avatars on bot accounts for disinformation campaigns. Fun! https://twitter.com/...
@heathercampbell Heather Anne Campbell on x
This is a subplot in Ghost in the Shell https://twitter.com/...
@z_everson Zach Everson on x
DIY— Anyone can get their own AI-generated profile pic for free at https://thispersondoesnotexist.com/ https://twitter.com/...
@marshallk Marshall Kirkpatrick on x
You know those ads from The Epoch Times, with their annoying “old-fashioned honest journalism” schtick? Turns out they're a right wing fraud & keep getting banned from social networks, now for using AI to generate fake Trump supporters on Facebook. Jerks. https://www.thedailybeas…
@mczub Czub on x
these are the same people that run shen yun https://twitter.com/...
@amy_siskind @amy_siskind on x
Content removed included 610 Facebook accounts, 89 Facebook Pages, 156 Groups and 72 Instagram account, and sparked concern that AI could be used to create wide-scale disinformation campaigns. https://www.nytimes.com/...
@halsparks Hal Sparks on x
🎼Cheeeck Ouuut Thiiis 🐴💩 https://twitter.com/...
@poppy_haze Rev. Poppy Alter Santa on x
this is the Falun Gong cult that also produces Shen Yun and runs the NTD, New Tang Dynasty TV channel https://twitter.com/...
@oneunderscore__ Ben Collins on x
Existential question from The Epoch Times' now-shuttered trolling operation: Is it still a deepfake if you make an AI-generated profile of a handsome American, name him a moderator of your Facebook page—then your Vietnamese contractors named him “Ellen”? https://www.nbcnews.com/.…
@kevinroose Kevin Roose on x
This feels like a new frontier in the information wars. (And exactly the kind of thing worried AI researchers have been warning about for years.) https://www.nytimes.com/...
@camillefrancois Camille Franois on x
@donie @DFRLab @snopes @ExploitingNiche @UninaIT @SamGregory Credit also goes to our colleagues at @fbsecurity, who this morning removed 610 accounts, 89 Facebook Pages, 156 Groups and 72 Instagram accounts associated with this network. Their statement here: https://about.fb.com/…
@maddoxrules @maddoxrules on x
Snopes uncovered a giant network of fake accounts using AI-generated faces on Facebook to inflate a pro-Trump group linked to a conspiracy site, EpochTimes. @snopes tried to get a response from @Facebook and now FB won't credit Snopes for the investigation https://www.cnn.com/...
@camillefrancois Camille Franois on x
The scale of this network and their heavy use of automation to create accounts and populate pages was truly impressive. When the content was political, it focused on pro-Trump messaging and on criticism of the Chinese Government. More context from @donie: https://www.cnn.com/...
@ericfish85 Eric Fish on x
“The accounts, which were removed by Facebook on Friday, were part of a network that generally posted in support of President Trump and against the Chinese government.” Facebook said the accounts were tied to the Epoch Media Group, which owns Epoch Times https://www.cnn.com/...
@donie Donie O'Sullivan on x
DNI Coates had warned of the dangers of deepfakes earlier this year — here's an example of A.I. generated images being used to create fake faces for Facebook profiles (not nation state-linked) https://www.cnn.com/...
@hunterschwarz Hunter Schwarz on x
Facebook and Twitter have closed a pro-Trump network that reached 55 million accounts and used AI-generated faces as profile photos to “masquerade” as Americans https://www.theverge.com/...
@profcarroll David Carroll on x
Dystopia Alert: Pro-Trump anti-China Epoch Times Media group using deep fake style face generation tools to build fake accounts instead of stealing real images/identities. Hyperreal simulations of fake humans convincingly generated by machines. https://www.cnn.com/... via @donie
@moira Moira Whelan on x
BIG: Definitive proof that #SaudiArabia is manipulating Western audiences using inauthentic accounts. #Khashoggi https://twitter.com/...
@yoyoel Yoel Roth on x
Today we're sharing information about a significant state-backed information operation originating in Saudi Arabia. We're making comprehensive data on this activity available to researchers and the public, and full details are on our blog: https://blog.twitter.com/...

Chronicles