Discussion

• @_swanand Swanand on x

  Never build your business on GCP. This is an incredible risk surface area.

• @railway @railway on x

  Google Cloud has blocked our account, making some Railway services unavailable. We have escalated this directly with Google. The Railway Platform team has since confirmed access to Google Cloud and is working on restoring access to all workloads. We have access to some of our

• @mikejulian Mike Julian on x

  As I've said before, the biggest risk to many companies is Google's lack of respect for their customers

• @nick_white Nick White on x

  Now folks, this is why redundancy matters. Google suspended Railway's GCP account. Control plane + workloads nuked. Single-cloud PaaS kill switch.

• @gergelyorosz Gergely Orosz on x

  This wild. Two years ago, Google deleted a large customer's account: Unisuper, an Australian retirement savings fund. It seems they now blocked an even larger customer, cloud provider Railway. This kind of story you never hear with AWS, Azure, or even Oracle. GCP 👎

• @pragmatic_eng @pragmatic_eng on x

  Google Cloud blocked cloud infra provider Railway's account, taking it offline. It's not the first time GCP did something unacceptable: in 2024, they deleted the infra of a $124B Australian fund. Not fully trusting GCP saved the fund's data. Full: https://blog.pragmaticengineer.c…

• @quinnypig Corey Quinn on x

  Google will fix this as soon as @justjake escalates to the proper support channel, which for Google remains “the front page of Hacker News.”

• @oscargws Oscar Watson-Smith on x

  Some thoughts on the discourse around @Railway team's outage. I was a mid-level SE at @datadoghq ANZ in 2023 when the company had a major, long lasting outage (actually started while I was mid demo). I was in a sales role, and what was most interesting to me was the response from

• @championswimmer Arnav Gupta on x

  This is just unbelievable! Railway even has their own datacenters, but the APIs and edge network is on Google Cloud so everyone using Railway is down because Google just nuked their account 🤯

• @jazzychad Chad Etzel on x

  all of my websites are down while Railway fights with Google Cloud. the future is neat.

• @dexhorthy Dex on x

  railway and every railway service going down because Google Coud blocked them was NOT on my bingo card I have heard horror stories of people getting blocked by google cloud and response/support being slow but this is wild, esp with the amount of spend they must be doing

• @gergelyorosz Gergely Orosz on x

  When you are on GCP, you NEED to have a backup on another cloud, because Google can delete everything. Unisuper did this, and avoided full data loss only thanks to this. I assume Railway will no longer rely on only GCP. They would be foolish to do so. https://newsletter.pragmatic…

• @neetcode1 NeetCode on x

  I wonder how much damage this does to Google Cloud, let alone Railway. Has to be at least $1B for google long term. As someone who worked in gcp, I would describe the culture as the opposite of customer obsession. I have a few horror stories I'll keep to myself, but let's just

• @railway @railway on x

  We have recovered our compute on Google Cloud, but services are unable to start because of ongoing networking issues on Google Cloud's side. We are engaged with Google Cloud support to resolve this and will post the next update as soon as we have more to share.

• @justjake Jake on x

  It appears Google Cloud has blocked our account, and so some services are unavailable We've escalated this to Google and will keep people posted. Deepest apologies.

• @richardartoul Richard Artoul on x

  Does stuff like this ever happen with AWS? I only ever hear about it with GCP. It feels like they took the same technology they use to ban people's personal Gmail accounts and just pointed it directly at enterprise cloud customers running real businesses

• @simistern Simon on x

  I run a ton of stuff on railway and I'm honestly probably done. Really nice UX but you can't allow this situation if you do infra. You just can't.

• @railway @railway on x

  Railway services have fully recovered. Some workloads may still need a redeploy, we're automatically redeploying any we detect as unhealthy. If your service isn't responding correctly, please trigger a redeploy from the dashboard or CLI. We're sorry for the disruption. A