Rainbird: The Way Twitter Counts Tweets In Realtime (Which Will Be Open Sourced)
Yesterday, Twitter analytics lead, Kevin Weil, gave a talk at O'Reilly's Strata 2011, a conference dedicated to big data. The main topic of the talk was Rainbird, Twitter's realtime counting system that's built on top of Cassandra.
Context & Ripple Effects
The talk closes the loop on a promise Twitter made in September 2010, when it said it would ship a real-time analytics solution in Q4 — Rainbird, Kevin Weil's Strata presentation explains, is the counting engine underneath that effort, running on Cassandra rather than a homegrown store. The reveal lands mid-push by CEO Dick Costolo to prove the service can make money, which makes an internal metrics system suddenly a public artifact.
It also continues a thread visible since the 2009 investor-focused data analysis of the firehose: Twitter's value has always been framed in counts — tweets, clicks, topics — and now the company is showing the machinery that produces them at scale.
First-order effects
- Developers building on Cassandra gain a battle-tested reference architecture from one of its highest-volume users, with Twitter committing to release Rainbird's code rather than keep it proprietary.
- Weil's team gets external scrutiny and contributions for free once the source is out, shifting maintenance of the counting layer from a purely internal cost to a shared one.
Second-order effects
- Other NoSQL stores competing for realtime workloads face a higher bar: when a system like Twitter's validates Cassandra publicly, infrastructure choices at startups tend to follow the proven deployment.
- Analytics vendors and agencies reselling tweet-level metrics now have a published model for how the platform itself tallies activity, tightening what 'realtime numbers' means across the measurement market.
Third-order effects
- If the open-sourcing holds, Twitter joins the pattern of web-scale operators releasing internal infrastructure as de facto standards — trading secrecy for ecosystem gravity and engineering credibility around a specific database.
- Counting infrastructure drifting into open source foreshadows a split where the differentiator is no longer owning the counter but owning the data feeding it, which is where the monetization question Costolo is pressed on actually lives.
The trend: Web-scale companies are turning internal big-data infrastructure into open-source projects to set standards and attract talent, with Twitter's Cassandra-based stack as a leading example.