Twitter says it mistakenly suspended 12 accounts after “coordinated and malicious reports” targeted researchers and journalists, and will review its new policy

Far-right activists had urged their followers to use a new company rule to target anti-extremism researchers and journalists

Washington Post 2021-12-05 Drew Harwell

Context & Ripple Effects

Twitter’s new private-information rule was quickly turned into a reporting weapon against anti-extremism researchers and journalists: one researcher’s suspension followed a report on a 2019 tweet under the private-media policy. The company now acknowledges that a coordinated reporting campaign produced 12 mistaken suspensions and says it will review the rule.

The episode reverses Twitter’s usual posture toward coordinated activity. It had previously removed accounts for coordinated manipulation, but here coordinated reporting appears to have shaped enforcement against the targets rather than the organizers.

First-order effects

The 12 suspended researchers and journalists face immediate account-access and distribution disruption while Twitter reviews the policy and its enforcement decisions.
Twitter must reassess how reports alleging private-information violations are evaluated, because coordinated complainants successfully triggered mistaken suspensions.

Second-order effects

Anti-extremism researchers and journalists have a clearer incentive to treat platform-reporting campaigns as an operational risk, particularly when older posts can be surfaced for review.
Twitter’s trust-and-safety operation must distinguish genuine policy complaints from brigaded reports, or adversarial groups can use its own reporting process to suppress critics.

Third-order effects

Content-policy enforcement is becoming an attack surface: rules intended to limit harmful publication can be repurposed through coordinated reporting to affect who remains visible on a platform.
If platforms do not build stronger safeguards around report-driven enforcement, moderation disputes will increasingly center on the integrity of enforcement inputs, not only the text of the rules.

The trend: Platform moderation is shifting from a focus on prohibited content alone toward defending enforcement systems from coordinated manipulation.

Discussion

@ryanjreilly Ryan J. Reilly on x
“Twitter spokesman Trenton Kennedy said that the company had been overwhelmed with a ‘significant amount’ of malicious reports and that its ‘enforcement teams made several errors’ in the aftermath.” https://www.washingtonpost.com/ ...
@krisgoldsmith85 Kristofer Goldsmith on x
It looks like @Twitter's official statement is “oops.” https://www.washingtonpost.com/ ... https://twitter.com/...
@chadloder Chad Loder on x
URGENT: As we feared, @TwitterSafety is already locking and suspending the accounts of extremism researchers under its new “Private Media” policy. The video is from September (predating the policy) and shows two right-wing extremists IN PUBLIC, planning violent assaults. https://…
@krisgoldsmith85 Kristofer Goldsmith on x
NEW: Last night nazis of Charlottesville-riot-infamy instructed thousands of followers to mass-report accounts of 48 journalists, anti-racist activists, + anti-fascist researchers for retroactive “violations” of @TwitterSafety's new “Private Information Policy.” Some down already…
@ryanjreilly Ryan J. Reilly on x
“Neo-Nazis and far-right activists are coaching followers on how to use a new Twitter rule to persuade the social media platform to remove photos of them posted by anti-extremism researchers and journalists...” https://www.washingtonpost.com/ ...
@evan_greer Evan Greer on x
No one wants to talk about it, but stuff like this is the inevitable result of constant calls for platforms to remove more content faster without any concern about the harm done by over moderation and suppression of legitimate content https://twitter.com/...
@antihateca @antihateca on x
This sort of research is absolutely critical right now, and we appreciate @JaneQCitizen for putting this out there. @TwitterSafety has been forcing researchers to remove their work exposing the extreme right. This needs to stop. /3 https://www.washingtonpost.com/ ...
@humanrights1st @humanrights1st on x
@Twitter Innovation Lab researcher @KrisGoldsmith85 told the @washingtonpost that the rule could undermine Twitter's front-line role in distributing critical information about online and real-world hate campaigns. https://www.washingtonpost.com/ ...
@ross_stalker @ross_stalker on x
In case anyone was still working under the misconception that these so-called free speech warriors actually care about freedom of speech... https://twitter.com/...
@brianfishman Brian Fishman on x
This is an interesting case study of well-intentioned policy updates being abused. It also illustrates that coverage is driven by platforms rather than harms. Hence stories abt nazis et al sneakily abusing Twitter policy rather than the same folks coordinating openly on Telegram.…
@qualiaredux Qualia Redux on x
A side effect of policing is that people who do the work of warning the public of violent behavior in order to prevent it are silenced because they're outside the system. So all we're left with is a fascist police force that only responds after the violence is done—maybe. https:/…
@krisgoldsmith85 Kristofer Goldsmith on x
Check @chadloder's thread here for a list of accounts impacted by the nazi-led harassment campaign. It's pretty telling that even tweets in that very thread have been taking down for violating Twitter's new rules: https://twitter.com/...
@drewharwell Drew Harwell on x
Activists and researchers incl. @chadloder have been cataloging some of the account suspensions so far: https://twitter.com/...
@birgittaj Birgitta on x
Hey @TwitterSafety - this new rule is counter productive unless you actually invest more clarity and manpower to it. Perhaps just stop implementing until you have figured out what you are trying to do. https://twitter.com/...
@jeffjarvis Jeff Jarvis on x
Here we go, just as feared: https://twitter.com/...
@melissalyttle Melissa Lyttle on x
Well that went south quickly, like we all knew it would. Good job @Twitter. https://twitter.com/...
@frenchrainez @frenchrainez on x
What? Oh no! In less than 3 days?!😱 https://twitter.com/... https://twitter.com/...
@blackamazon @blackamazon on x
We tried to get them and multiple journalistic and tech foundations to handle this loophole when it was Black women and even in GG Everyone ignored and is gonna pretend this is the start of it Y'all have fun https://twitter.com/...
@emilygorcenski Extremily Gorcenski on x
Twitter is not even compliant with its own policies. Twitter's terms of service state that it must be a first party claim, meaning only the person in the photo can file the claim. This is clearly not the case. https://twitter.com/...
@carnage4life Dare Obasanjo on x
This is the most predictable outcome of Twitter's policy that anyone can issue a takedown request for non-consensual photos of them in public. This is just a massive unforced error. If it's due to “right to be forgotten” they should have kept it EU only. https://www.washingtonpos…
@mmasnick Mike Masnick on x
So @TwitterSafety's new media rules are basically being weaponized. Even if Twitter ignores bogus reports and brigades (and early evidence suggests they're not), this whole setup is just going to lead to constant abuse and extra work. https://twitter.com/...
@willoremus Will Oremus on x
GOP talking point: Twitter's new CEO is biased against white people Reality: First major policy approved by Twitter's new CEO ends up benefitting white supremacists https://twitter.com/...
@verambergen Vera Bergengruen on x
“Researchers fear the new ban on posts sharing people's private information will be ‘emboldening to the fascists’ eager to keep their identities concealed. ‘Things now unexpectedly work more in our favor,’ one Nazi sympathizer wrote.” https://www.washingtonpost.com/ ...
@jeffjarvis Jeff Jarvis on x
As I said, as predicted: Far right is using Twitter's new rule against anti-extremism researchers https://www.washingtonpost.com/ ...
@ericajoy Erica Joy on x
really? retroactively applying policy to old tweets and then locking and suspending accounts based on the retroactive application of policy? that's what we're doing? do better, @twitter. https://twitter.com/...
@__chicagojoe_ @__chicagojoe_ on x
“They won't let us doxx them anymore! This is so unfair!” https://twitter.com/...
@drewharwell Drew Harwell on x
New: Far-right activists and neo-Nazis are weaponizing Twitter's new rule against anti-extremism researchers. Fears of a chilling effect for those documenting real-world hate: “Emboldening to the fascists” https://www.washingtonpost.com/ ...
@evacide Eva on x
Well-meaning policies can have serious blowback. I'm particularly concerned about Twitter's new privacy policy because they appear to be enforcing it retroactively against tweets that predate the policy. https://twitter.com/...
@mathewi Mathew Ingram on x
I wrote about how Twitter's new privacy policy on posting images is likely to clash with journalism. How will the company decide who is a “public figure,” or what is “in the public interest?” The answers to those and other questions remain unknown https://www.cjr.org/...
@digiphile Alex Howard on x
“a well-intentioned idea that sounds like an absolute horrorshow to actually implement,” in part because social media itself has made it “incredibly thorny” to determine who is a public figure."-@Normative https://www.cjr.org/...
@eclecticbrotha Ragnarok Lobster on x
Well, it was exactly what we expected. On the other hand, I never knew twitter had actual humans working as enforcement teams. https://twitter.com/...
@dsilverman Dwight Silverman on x
Could anyone have guessed this would have happened? Anyone? Bueller? Twitter says it suspended accounts in error following flood of ‘coordinated and malicious’ reports https://www.washingtonpost.com/ ...
@jeffjarvis Jeff Jarvis on x
The Twitter saga is a good example of the problems of enforcement according to statues vs. judgment according to principles. The latter is hard & expensive (that is, doesn't scale — which is why tech companies resist it) but necessary. Principles are adaptable, statutes not. http…
@carnage4life Dare Obasanjo on x
How does a policy as obviously bad as allowing people to file takedown notices of pictures of themselves in public make it out of a brainstorming session let alone become policy at a multibillion dollar social media company with hundreds of millions of users? https://twitter.com/…
@nycantifa @nycantifa on x
It's almost like they knew what to expect... https://www.vice.com/...
@drewharwell Drew Harwell on x
@chadloder ... New: Twitter says “a significant amount of coordinated and malicious reports” have flooded in re: new rule and its “enforcement teams made several errors.” Says it's fixed errors and launched an internal review to ensure rule is “used as intended” https://www.washi…
@gwensnyderphl @gwensnyderphl on x
This is a big victory for all the folks who spent the past few days calling attention to this dystopic Nazi hug of a policy, but @TwitterSafety needs to 1) restore maliciously reported accounts/tweets 2) wipe strikes from our accounts 3) actually talk to impacted folks https://tw…
@madbastard_v2 @madbastard_v2 on x
Gosh, it's almost like this was completely predictable, and could have been avoided if anyone in Twitter's upper echelon had thought about it for more than half a minute AND not completely automated the goddamn process. https://twitter.com/...
@leahmcelrath @leahmcelrath on x
I'm glad to hear this admission from @Twitter @TwitterSafety, but I'll reserve judgment until we see actual changes in implementation. https://twitter.com/...
@emilygorcenski Extremily Gorcenski on x
In Twitter's defense, there are hardly any prior examples of coordinated malicious right-wing campaigns on this site for them to have learned from in the past few years. How could they have seen this coming. There's been only like three or four thousand. https://www.cnn.com/...
@digiphile Alex Howard on x
“We've corrected those errors and are undergoing an internal review to make certain that this @policy is used as intended — to curb the misuse of media to harass or intimidate private individuals,” @Twitter told @b_fung https://www.cnn.com/... If it's being used to harm, revert. …
@b_fung Brian Fung on x
Twitter is acknowledging a wave of “coordinated” and “malicious” activity on its platform as right-wing activists say they've been “weaponizing” the site's new private media policy: https://www.cnn.com/...
@ericajoy Erica Joy on x
next @twitter will “pause” the program as they “re-evaluate” to “learn more.” come on y'all. why is this script so predictable? https://twitter.com/...

Chronicles