How Facebook keeps 100 petabytes of Hadoop data online
It's no secret that Facebook stores a lot of data — 100 petabytes, in fact — in Hadoop, but how it keeps that data available whenever it needs it isn't necessarily common knowledge. Today at the Hadoop Summit, however …
Context & Ripple Effects
Facebook has been drip-feeding its infrastructure story for years: the 30,000-server count in October 2009 established the sheer size of the estate, the 2010 Royal Pingdom teardown mapped the software stack, and the April 2011 decision to open-source its servers and data center designs turned that transparency into a public project. This week's Hadoop Summit talk extends the same playbook from hardware to the data layer, following the Scuba in-memory analytics deep dive Facebook published in April 2012.
The timing matters commercially: on the same day, Facebook confirmed Facebook Exchange, a cookie-based real-time bidding ad system, which only works if behavioral data stays queryable at scale around the clock. The 100-petabyte availability claim is therefore not an infrastructure boast in isolation — it underwrites the ad product line the company is pitching to advertisers.
First-order effects
- Facebook's disclosed practices give every large Hadoop operator — and the vendor ecosystem selling to them — a reference architecture from the single biggest known deployment, effectively setting expectations for what 'webscale' Hadoop means.
- The always-on guarantee directly supports the freshly confirmed Facebook Exchange real-time bidding system, whose targeted ads depend on continuous access to user behavioral data.
Second-order effects
- Rival webscale properties and enterprise Hadoop distributors now face pressure to publish comparable scale-and-availability numbers, since Facebook has made operational maturity a marketing artifact rather than a trade secret.
- Hardware and storage suppliers gain a demand signal: keeping petabyte-scale clusters continuously available pushes buyers toward the commodity-server designs Facebook already publishes through its open hardware effort.
Third-order effects
- If the pattern holds, infrastructure disclosure functions as a recruiting and credibility weapon — engineers join to work on problems they've read about publicly, reinforcing the gap between hyperscalers and everyone else.
- Hadoop's position as the default substrate for petabyte-scale analytics gets cemented not by benchmarks but by the existence of a 100-petabyte production proof point, pulling enterprise adoption further along.
The trend: Hyperscale operators are converting internal infrastructure know-how into public disclosure — part recruiting moat, part ecosystem leadership — while Hadoop hardens into the standard platform for always-on, petabyte-scale analytics.