A bug in Ethereum's Nethermind client took down 8% of Ethereum's validators on January 21, raising concerns about Geth, which powers ~85% of the validators

The lion's share of Ethereum's validators rely on the same piece of software to power their operations. According to some experts, this could be a big risk.

CoinDesk 2024-01-23 Sam Kessler

Context & Ripple Effects

Ethereum has previously seen client software failures reach the protocol layer: a Go Ethereum flaw split part of the ecosystem onto a separate chain before it was fixed. Earlier coverage also found that many nodes delayed applying security patches, making operational discipline a recurring network risk.

The Nethermind outage shifts attention from an isolated client incident to the distribution of validator software. With Geth serving roughly 85% of validators, its reliability has become a network-wide dependency rather than merely a choice for individual operators.

First-order effects

About 8% of Ethereum validators running Nethermind went offline, disrupting those operators’ participation until they could address the client issue.
The incident puts immediate scrutiny on Geth’s dominant validator share: a comparable Geth failure would affect a far larger portion of Ethereum’s validating base.

Second-order effects

Validator operators and staking providers have a stronger incentive to assess client exposure, update procedures and the trade-off between Geth’s operational familiarity and alternative-client risk.
Client teams face greater pressure to harden releases and incident response, particularly given the earlier pattern of slow security-patch adoption among Ethereum nodes.

Third-order effects

If validator-client concentration persists, Ethereum’s resilience will increasingly depend on the failure modes and release practices of a single dominant implementation.
The episode reinforces client diversity as a form of infrastructure risk management, though shifting operators across clients can itself introduce new operational and software risks.

The trend: Ethereum’s staking era is making validator-client diversity an increasingly important measure of decentralization and operational resilience.

Discussion

@coinbasecloud @coinbasecloud on x
1/ Execution client diversity on Ethereum is a critical concern for all of us at Coinbase. Here's what we're doing about it. ↓
@0xcygaar @0xcygaar on x
Ethereum has terrible client diversity - here's quick overview of the current landscape: - Geth currently makes up a super majority of all nodes - Coinbase, Binance, Kraken are 100% using geth - Lido's stake is 76% in geth - RocketPool and StakeWise (two of the more diverse... [i…
@daddysether Liam on x
@EthereanMaximus I think the nuance people are missing here is that having a supermajority EL client doesn't just affect Geth users in the event of a nasty bug - it affects literally “everyone” who uses Ethereum (especially in the event of a chain split). Yes, geth is a great cli…
@yorickdowne Thorsten Behrens on x
This issue with @NethermindEth was a consensus bug. If this happens on Geth while it has a supermajority we are major screwed. Finding a Geth bug would be satisfying for those who need or want Ethereum to fail. Please come off Geth. Fixing a minority client >> losing staked ETH
@daddysether Liam on x
Today's beacon chain hiccup has once again highlighted the importance of EL client diversity. This could have been a very different outcome if the bug was caused by Geth, which is currently a supermajority. Switch to a minority client, and help keep Ethereum secure! [image]
@0xwittgenstein Ludwig Wittgenstein on x
This theoretical geth bug is nice because you know they'll roll the chain back, and if you are a reasonable person who doesn't waste their time staking you'll get another coin out of it too
@0xpatches @0xpatches on x
To make matters worse, if we run the supermajority client, and it splits due to a consensus bug, users on the minority clients who subsequently panic and connect to the Rescue Node would not be able to participate in the chain during quadratic leak, making recovery even slower.
@backthebunny @backthebunny on x
I recommend everyone take their correct geth/client diversity understanding and apply it to all of DeFi now. It's dangerous and fragile to have Ethereum be the mother layer with no other competitive L1s. I root for Solana, Fantom, AVAX, and the like. DeFi is a network, the...
@ether_fi @ether_fi on x
One of our core principles is “We will do the right thing for the Ethereum community, always” We think the geth team is incredible and owe them a huge debt of gratitude. Yet, we realize that more people using minority clients benefits the Ethereum network as a whole. As a...
@evan_van_ness Evan Van Ness on x
Geth is an incredible piece of software, powering countless alt-L1s and L2s But if your staking pool is still running Geth, they're negligently putting YOUR MONEY and Ethereum at risk https://twitter.com/...
@cryptoquoc @cryptoquoc on x
4/5 In principle, I would like to see more execution client diversity as well. Forcing/or choosing validator services solely for the purpose of client diversity may actually hurt your staking rewards due to downtime, and bug like we experienced with Nethermind.
@cryptoquoc @cryptoquoc on x
🧵1/5 On the topic of Ethereum client diversity in light of the Nethermind bug going around. The solution is not move away from geth (84%). Remember, it's not geth, but Nethermind that has the bug. Imagine if that was reversed. The real reason that most validators still rely on...
@icebergy_ @icebergy_ on x
I guess we can argue about geth and client diversity for a few weeks
@monetsupply @monetsupply on x
with current >80% dominance, if geth accepted an invalid block it could be finalized and prevent geth stakers from participating in the valid chain this would quadratic trigger inactivity leak where at least 50% and up to 100% of their stake would be destroyed
@0xpatches @0xpatches on x
Before we knew with certainty that Nethermind was on the wrong chain and Geth was on the right chain, people were passing around the @Rescue_Node link and encouraging Node Operators to use it to “get back online”. I don't think they were aware of the risks, which is our fault.
@0xnirlin @0xnirlin on x
Why Geth rhymes with Death? ☠️
@0xcygaar @0xcygaar on x
A simple primer on Ethereum client diversity - what it means, why it matters, the current state, and how Ethereum can be made more resilient 🧵: [image]
@ethereumthaila1 @ethereumthaila1 on x
Yep! Now we can mitigate the risk of Geth Dependency. Coinbase and all other Ethereum Network is shifting away from Geth starting now
@chainyoda @chainyoda on x
Ethereum does not have any decent execution clients other than Geth because execution clients don't have their own token
@orageux101 @orageux101 on x
Reasonable opinion: Geth has done a massive amount for Ethereum and is still massively important. All users moving to young and fragile clients is even worse for Ethereum. The transition from Geth to other clients will be super easy for those that want to when they mature.
@zeneca @zeneca on x
This is very important and more people should be aware of it, so we should all make as much noise as we can to try and get the big dogs to switch from geth to another client @coinbase @binance @krakenfx who's gonna be first to do something for the good of Ethereum and switch?
@llamaonthebrink @llamaonthebrink on x
If geth reliance falls below 66%, the last (systemically risky) centralized component of Ethereum will have decentralized Shout out to everyone who's been preaching about client diversity ✊
@iamdcinvestor @iamdcinvestor on x
with every new upgrade to Ethereum, the probability of there being a new, undiscovered bug in EVERY client increases would be ideal to get the geth supermajority situation resolved before Dencun is deployed i would almost say it's reckless to upgrade while it persists
@ethereanmaximus Etherean Maximus on x
@0xCygaar Agree with your overall point but there is good reason Geth is a super majority. You're asking people to switch from a battle tested client that's had little to no problems to clients that aren't as battle tested and have had problems. You can understand why people are …
@chainyoda @chainyoda on x
tl;dr: Execution clients other than geth are not good enough yet. 100% GETH dominance would also solve the problem kind of
@iamdcinvestor @iamdcinvestor on x
the ecosystem crucially needs to address the root cause of why everyone chooses geth over everything else and make other choices as attractive otherwise we will keep doing this and the only way to shift it will be to continually withhold upgrades unless healthy equilibria exist
@cryptoquoc @cryptoquoc on x
3/5 Before we jump on the move away from geth bandwagon. Why is it 84%? In our @stakefish experiences, we tested across all the clients, and still have configuration to run any of them. We chose geth as primary, and nethermind as backup. Geth has a long history before Eth2 and...…
@ryanberckmans @ryanberckmans on x
The geth supermajority risk highlights the danger of investing too much into orgs with significant vertical integration across Ethereum product categories. For example, Starknet and Arbitrum are amazing L2s that don't also run threatening LSTs and stablecoins. In a chain split...
@coinbasecloud @coinbasecloud on x
3/ Although we've evaluated execution clients since 2020, none have met Coinbase Cloud's requirements to date. Many other operators on the network have reached the same conclusion, which is part of the reason why 84% of Ethereum validators run Geth. However, the tide is turning.
@0xpatches @0xpatches on x
@Rescue_Node was able to survive the Besu and Nethermind issues because it runs Geth. We'll be moving off Geth in the coming days, but I wanted to quickly explain why we used it to begin with, and why we are moving away hastily.
r/CryptoCurrency r on reddit
Bug That Took Down 8% of Ethereum's Validators Sparks Worries About Even Bigger Outage

Chronicles