/
Navigation
Chronicles
Browse all articles
Explore
Semantic exploration
Research
Entity momentum
Nexus
Correlations & relationships
Story Arc
Topic evolution
Drift Map
Semantic trajectory animation
Posts
Analysis & commentary
Pulse API
Tech news intelligence API
Browse
Entities
Companies, people, products, technologies
Domains
Browse by publication source
Handles
Browse by social media handle
Detection
Concept Search
Semantic similarity search
High Impact Stories
Top coverage by position
Sentiment Analysis
Positive/negative coverage
Anomaly Detection
Unusual coverage patterns
Analysis
Rivalry Report
Compare two entities head-to-head
Semantic Pivots
Narrative discontinuities
Crisis Response
Event recovery patterns
Connected
Search: /
Command: ⌘K
Embeddings: large
TEXXR

Chronicles

The story behind the story

days · browse · Enter similar · o open

GitHub says its 7+ hour August 17 outage was caused by a capacity failure when peak traffic overwhelmed an infrastructure component in a Central US data center

An update on the August 17 outage and the steps we're taking to improve reliability.  —  On August 17, GitHub experienced an outage that lasted 7 hours and 47 minutes.

The GitHub Blog Vlad Fedorov

Context & Ripple Effects

GitHub had already framed reliability as a priority amid AI-driven growth, putting availability ahead of capacity and new features in its earlier reliability response. The August incident turns that priority into a concrete capacity-planning test: a Central US infrastructure component was overwhelmed at peak traffic.

Initial coverage established that the disruption reached GitHub’s website, API, Actions and pull-request services before mitigation was reported. The later root-cause update matters because it identifies capacity, rather than an external attack, as the operational issue to correct.

First-order effects

  • GitHub must focus its reliability work on the Central US component and the capacity controls around peak traffic, after an outage lasting 7 hours and 47 minutes.
  • Developers and organizations dependent on GitHub’s API, Actions and pull-request workflows were directly exposed to a single infrastructure-capacity failure.

Second-order effects

  • Teams that build deployment and collaboration processes around GitHub services have a stronger incentive to design fallbacks for interruptions across code hosting, APIs and automation rather than treating those functions as independent.
  • GitHub’s stated ordering of availability, capacity and features gains operational weight: capacity work competes more directly with feature delivery for engineering attention.

Third-order effects

  • If traffic growth continues to expose concentrated components, developer platforms will increasingly treat capacity engineering as a product constraint rather than a back-end scaling task.
  • The episode fits a broader shift in which AI-linked demand growth makes reliability depend on how quickly infrastructure capacity and fault isolation keep pace with usage.

The trend: AI-era demand is pushing software platforms to make capacity resilience a first-order product and operations priority.

Discussion

  • @abyrd_9 Andrew Byrd on x
    If I read this sentence to a medieval peasant he would report me to the church and I would be burned at the stake as a wizard or sumn
  • @acolombiadev Andrea on x
    I work at GitHub. yesterday was rough and i'm not pretending otherwise. full root cause report is up if you want the timeline and numbers, and what we are doing to prevent this from happening again. https://www.githubstatus.com/ ...
  • @thehungrybird_ Aryan on x
    @Vlad_GitHub After seeing this graph, it feels like we are not appreciating folks at GitHub for what they deserve.
  • @sethrubenstein Seth Rubenstein on x
    It's too late. Yes the 17th was all day; but this has been a problem for a while. I pay for a teams account. My devs couldn't work. You bifurcate enterprise from the main product, if free repos are part of our core offering then you need to isolate paid teams accounts. You can
  • @dodyg @dodyg on x
    This is the reason why Codeberg banned AI projects. Many people were very critical of that decision. If they didn't band AI projects, they will face the same issue as GitHub. > Since April, monthly commits have grown from 1.4 billion to 2.9 billion. https://github.blog/...
  • @nickgeracehacks Nick Gerace on x
    Ouch! That definitely did not help. > Copilot Token Service traffic increased from a normal 7-9K RPS to 70-100K RPS
  • @jackellis Jack Ellis on x
    “We have since added more than 3 million CPU cores, 120 petabytes of high-speed storage, and significant network capacity” 😳
  • @kdaigle Kyle Daigle on x
    Our CTO shared a blog on GitHub's reliability, what we've already done, and what more we need to do to. Monthly commits and merged PRs alone have nearly doubled since April. As the home for devs, availability is critical. I'm sorry for the pain and thankful for the support. 🙏
  • @daniellockyer Daniel Lockyer on x
    Some crazy graphs of PR/commit growth on GitHub since 2023
  • @rhyssullivan Rhys on x
    I really don't think any of us have internalized just how much software development has changed in these few years, truly insane numbers
  • @perrymetzger Perry E. Metzger on x
    So, this is the wrong set of repairs. I've seen clients of mine make the same mistake time and again. You see a particular failure and you decide “I'll audit that particular configuration mistake better in the future!” The correct response is to make sure that you're doing
  • @hd_nvim @hd_nvim on x
    Seeing this graph reminds me that storing code for free is a privilege, especially in the AI era. Putting a rate limit/storage limit on free users makes sense for platform usability.
  • @edandersen Ed Andersen on x
    Had 10 years to port the Ruby codebase to C#
  • @grantjordan Grant Jordan on x
    @acolombiadev “I'm not pretending otherwise”, said Claude
  • @ikhwanuddin @ikhwanuddin on x
    Do people still use VS Code with Github Copilot? I find it bloated, which is why I switched to @zeddotdev and @pidotdev
  • @whatranjusaid Ranju on x
    GitHub's yesterday's outage RCA report Misconfigured policy
  • @0xblacklight @0xblacklight on x
    fascinating read btw
  • @dbreunig Drew Breunig on x
    This is insane demand during a time when infra is very expensive. Would-be competitors better have a very good plan and very, very deep pockets.
  • @1rjfinnegan RJ Finnegan on x
    @acolombiadev Most ppl complaining are on a free account with half baked projects that weren't going anywhere - me included - but appreciate the transparency
  • @thedavidtai @thedavidtai on x
    So it was self amplification as services responded to saturation induced downtime by retrying calls with aggressively until the retries did a running ddos against the platform.
  • @vlad_github Vlad F on x
    On August 17, GitHub experienced a significant outage that disrupted developers and organizations around the world. If you were trying to ship software that day, we let you down. I posted in March and April about the steps we're taking to make GitHub more reliable. The work is
  • @yordisprieto Yordis Prieto on x
    For everyone criticizing GitHub, go ahead, show me the scaling situation you actually deal with. THIS is exactly why, although, yes, please be professional and do your best, it is NOT acceptable to break production. Fine. Nevertheless, these are not trivial problems to face,
  • @kevvoh_ @kevvoh_ on x
    A misconfigured policy is what took GitHub down. Engineering is funny sometimes.
  • @ai_for_success AshutoshShrivastava on x
    if you want to understand how much AI has accelerated software development, look at what's happening on GitHub. Monthly commits have jumped from 1.4 billion to 2.9 billion since April 2026. That's more than 2× growth in just a few months. The amount of code being written is
  • @ahmedehab_01 @ahmedehab_01 on x
    What Andrea here didn't mention is why a VS Code retry bug was introduced so easily in their system. They really should try to look into their engineering practices and relax the feature pushing to improve GitHub's health.
  • @kevincodex Kevin on x
    this graph says it all if agentic coding is driving this much growth among developers today, imagine the demand for agentic infrastructure when it reaches non-technical users that is the massive market we're building for at @gitlawb, and this is only the beginning
  • @thepghcid @thepghcid on x
    @acolombiadev The first sentence, and in guessing most of this, was written by some variant of Claude.
  • @mehulmpt Mehul Mohan on x
    Microsoft owns GitHub for almost 8 years now but only 12% workload was on Azure even in May this year I wonder what took so long (and how it got bumped almost to 60% in 3 months)
  • @ethanclinick Ethan Clinick on x
    Imagine showing this to a senior dev 10 years ago. What a time to be alive. Thank you @Microsoft and @github for eating these costs.
  • @brsanthu Santhosh on x
    @Vlad_GitHub These numbers are so astronomical that I believe Github cannot support current free offerings as is (I'm benefiting from the free offerings). I think we should expect users to pay beyond some limit (like 10) private repos. Also time to think about pricing by repo act…
  • @acolombiadev Andrea on x
    our CTO @Vlad_GitHub wrote up the August 17 outage, good to read in full. tldr: outage was not due to code or config changes. both were capacity failures, we didn't scale ahead of demand. that's on us. we'll earn trust back through the platform actually holding up, not through
  • @v_jug Vjekoslav Jug on x
    @acolombiadev “VS Code bug amplified GH traffic by roughly 10 times”. Wow. Does people still use VS Code that much? It became pretty bloated with not so great AI features (I mean they are not so bad if you haven't seen other tools).
  • @vlad_github Vlad F on x
    @thehungrybird_ The team is working very hard to make sure we are there for you. Scaling at this rate and rebuilding at the same time for new agentic load is not simple
  • @ibuildthecloud Darren Shepherd on x
    It's always, “these five random things failed at once in ways we never thought of. So to fix it we are going to ensure these five random things wont ever fail in this exactly way again.”
  • @stanfordnlp @stanfordnlp on x
    Sounds like a pretty good advertisement for @BuoyantIO's linkerd!
  • @pamelafox Pamela Fox on x
    Blog post with more details on the GitHub outage: 1) the load increase the servers are seeing 2) specific fixes made to prevent retry storm and cascading load 3) general scaling plans
  • @acolombiadev Andrea on x
    To prevent recurrence, our follow-up actions include: - Correcting autoscaling policies to account for service-mesh sidecar concurrency and capacity. - Auditing Istio request, concurrency, and scaling limits across affected services. - Reviewing retry limits and backoff
  • @matteocollina Matteo Collina on x
    Autoscaling is incredibly hard.
  • @acolombiadev Andrea on x
    our CTO @Vlad_GitHub wrote up the August 17 outage, good to read in full. tldr: outage was not due to code or config changes. both were capacity failures, we didn't scale ahead of demand. that's on us. we'll earn trust back through the platform actually holding up, not through
  • @kylehugh.es Kyle Hughes on bluesky
    I know it sounds crazy, and I've been saying it for a while, but I don't see how consumer technology will ever regain a foothold in manufacturing preference.  Jevon's paradox seems to mean infinite commercial demand for compute.  Every graph is straight up, including the marginal…
  • @phillipcarter.dev Phillip Carter on bluesky
    Write-up on the giga-outage and more from GitHub.  Every chart I have seen internally too is literally doing a hockeystick on growth.  IMHO this is also a canary in the coalmine for many other systems and products who have an API github.blog/news-insight...
  • r/programming r on reddit
    The August 17 outage, and the work ahead