Source: Twitter broke on Monday after the only site reliability engineer assigned to the project for migrating to a paid API made a “bad configuration change”

The high cost of cutting expenses — Twitter's website is breaking in novel new ways — and while the company managed …

Platformer 2023-03-07

Context & Ripple Effects

Twitter had already experienced a global failure affecting timelines days before the paid-API migration fault, followed by disruptions to links, images, and TweetDeck. The reported configuration error ties those visible reliability problems to an internal migration being handled by a single assigned site-reliability engineer.

First-order effects

Twitter users and services dependent on its website were immediately exposed to an outage caused by a configuration change in the paid-API migration.
The paid-API migration became an operational single point of failure for Twitter because its assigned reliability coverage was concentrated in one engineer.

Second-order effects

Developers building on Twitter's paid API face higher integration risk when platform changes can also disrupt core services; later Enterprise API customer complaints about outages and support reinforce that exposure.
Twitter's API monetization effort must absorb the operational cost of safer change management, rather than treating the migration solely as a pricing or access-policy change.

Third-order effects

If reliability staffing remains concentrated while access is moved behind paid API tiers, Twitter risks turning a developer platform into a less dependable paid dependency, weakening the value proposition for customers.
The episode points to a broader trade-off in cost-cutting platforms: reduced operational redundancy can transmit internal changes directly into user-facing and customer-facing failures.

The trend: Platform cost cutting is increasingly exposing how reliability capacity and API monetization are inseparable parts of the same product decision.

Discussion

@twittersupport @twittersupport on x
Some parts of Twitter may not be working as expected right now. We made an internal change that had some unintended consequences. We're working on this now and will share an update when it's fixed.
@elonmusk Elon Musk on x
@alx This platform is so brittle (sigh). Will be fixed shortly.
@elonmusk Elon Musk on x
@pmarca A small API change had massive ramifications. The code stack is extremely brittle for no good reason. Will ultimately need a complete rewrite.
@twittersupport @twittersupport on x
Things should now be working as normal. Thanks for sticking with us! https://twitter.com/...
@sarthakgh Sar Haribhakti on x
Today in I am never deleting this app https://twitter.com/...
@caseynewton Casey Newton on x
NEW: Twitter went down today because its migration to a paid API has a single site reliability engineer attached to it. Here's @ZoeSchiffer and me on the high price of cutting costs: https://www.platformer.news/ ... https://twitter.com/...
@zoeschiffer @zoeschiffer on x
Scoop: Twitter's latest outage is the culmination of Elon Musk's attempts to squeeze money out of the platform and his rampant firing of employees. Here's the inside scoop on what went down and why links and images were broken this morning: https://open.substack.com/...
@parismarx Paris Marx on x
Not only does Elon Musk not have a clue what he's talking about, but this is *exactly* what he tried to do at PayPal, claiming it needed a full “V2” rewrite of the code when it didn't. Instead, it shifted resources away from key areas and eventually got him ousted as CEO. https:/…
@popbase @popbase on x
Twitter issues recently: January 23 — error messages when tweeting February 8 — down for users globally February 15 — glitching for users globally February 18 — issues sending gifs February 28 — groupchat glitch with kicking March 1 — timeline not loading March 6 — content not...…
@pt Parker on x
The vibe I get from these guys is like a baby that hasn't mastered object permanence yet. Like, I'm here, so all the libs must be here too, right? https://twitter.com/... https://twitter.com/...
@alx @alx on x
Isn't it funny that the users who complained that they would leave Twitter when @ElonMusk took over are the first ones to notice when a feature is down on the platform?
@rhysmorgan Rhys Morgan on x
Of course. Nothing to do with the fact that you've sacked anyone half-competent, pulled the plug on a bunch of servers, and haven't paid your AWS bills, right? https://twitter.com/...
@hotpriestt @hotpriestt on x
the people working on it https://twitter.com/... https://twitter.com/...
@polotek Marco Rogers on x
A lot of people have bought into this rhetoric that Twitter had “too many” engineers. Now people are finding out why we have teams and redundancies. It's so no one person has to try to pretend to be infallible when making critical changes to systems. https://twitter.com/...
@jfreewright Jeffrey Wright on x
Twitter hard. Epidemiology easy. https://twitter.com/...
@martinsfp @martinsfp on x
“'There's so much tech debt from Twitter 1.0 that if you make a change right now, everything breaks,' one current employee says.” How a single engineer brought down Twitter on Monday https://www.platformer.news/ ...
@yevp @yevp on x
@Techmeme *former site reliability engineer?
@jyarow Jay Yarow on x
A lot of people thought they could improve Twitter if they were running it. He's been eyeing Twitter for about a year. Since he took over there hasn't been any material improvements, nor a plan or vision to make improvements. At best you can say it's the same. But it's worse.
@zachtratar Zach Tratar on x
Most code out there is brittle or weak in numerous ways for no good reason. The reason is: tech debt management is hard! Over a decade a codebase undergoes many waves of evolution. The rafters accrue dust. Experts leave the company and cracks form. https://twitter.com/...
@zenofdesign Damion Schubert on x
hey gang is it well advised to do a complete rewrite of a seventeen year old legacy codebase with millions of DAU and no capacity for unscheduled downtime? https://twitter.com/...
@mekkaokerekebye @mekkaokerekebye on x
I know that the one, single, engineer knows this already, but... No, a single engineer didn't bring down Twitter. Mismanagement and bad strategy brought down Twitter. Everyone else in the industry knows that it's not your fault*. Chin up. * 🤔Well, maybe not the Musk stans... http…
@minakimes Mina Kimes on x
Was gonna use the hot dog suit guy meme but I can't because the image won't work. Checkmate, Elon. https://twitter.com/...
@philliedrone @philliedrone on x
git commit -m “hope this doesn't break anything!” https://twitter.com/...
@reddit @reddit on x
Come on in, the links are fine.
@fake_api @fake_api on x
Everyone please stop talking about this. It was just a minor outage of the entire platform for over an hour caused by our need to relentlessly monetize every aspect of this website without any knowledgeable staff or adequate planning. This is not a big deal! https://twitter.com/.…
@callmeshitto Jake on x
hell yeah that's business baby https://twitter.com/...
@karaswisher Kara Swisher on x
Hey @Benioff, what up!?! — l mean, he can land a rocket on a surfboard!!! 🚀 🏄 https://twitter.com/...
@dz David Zhou on x
I understand the purpose of the headline, but this is better framed as “how inept leadership and laying off 90% of engineering brought down Twitter” whatever the individual engineer did is almost irrelevant [Link to The Verge's writeup: How a single engineer brought down Twitter]

Chronicles