Facebook open sources Corona — a better way to do webscale Hadoop
a better way to do webscale Hadoop — Facebook is at it again, building more software to make Hadoop a better way to do big data at web scale. Its latest creation, which the company has also open sourced, is called Corona and aims to make Hadoop more efficient …
Context & Ripple Effects
Facebook's Hadoop footprint was already the reference deployment by mid-2012 — GigaOM detailed in June how the company keeps roughly 100 petabytes of data online on its cluster — so any change to how that machinery schedules work lands as a template for everyone else running Hadoop at scale.
Corona, announced and open sourced on November 8, 2012, attacks Hadoop's best-known architectural bottleneck: the single JobTracker that both manages resources and schedules every job. The pickup was unusually broad for an infrastructure release — The Register, TechCrunch, VentureBeat, ReadWrite, ITworld and The Next Web all carried it the same day — reflecting how closely the Hadoop ecosystem watched whatever Facebook shipped.
First-order effects
- Operators of large Hadoop clusters gain an open-source alternative to the monolithic JobTracker, with Facebook's own production clusters serving as the proof point that the split-cluster-manager design holds up at petabyte scale.
Second-order effects
- Hadoop distributors and the Apache community face pressure to absorb Corona's resource-management ideas into their own roadmaps, since customers at web scale will now ask why their scheduler still centralizes what Facebook has decoupled.
Third-order effects
- If webscale operators keep upstreaming internal infrastructure — Corona being one instance of a pattern — the center of gravity for big-data innovation shifts from commercial Hadoop vendors to the hyperscale users who run it hardest, with open source as the distribution mechanism.
The trend: Big-data infrastructure is increasingly shaped by hyperscale operators like Facebook releasing their internal fixes back into the Hadoop ecosystem rather than by vendors alone.