Big data's challenge: gain new insight without making the same old statistical mistakes
Big data: are we making a big mistake? — Big data is a vague term for a massive phenomenon that has rapidly become an obsession with entrepreneurs, scientists, governments and the media
Context & Ripple Effects
Coverage from 2011 and 2012 tracked the rise of data sorting as a startup opportunity and the broader emergence of big data as a defining technology theme. By March 2014, the focus had shifted from collecting and processing large datasets to whether analysis can produce defensible insight.
The earlier case for combining big data with intuition framed data as an input rather than a substitute for judgment. This article extends that concern to statistical practice: scale does not itself protect entrepreneurs, scientists, or governments from flawed inference.
First-order effects
- Organizations deploying big-data analysis must put more weight on statistical validation, rather than treating a large dataset as proof that an observed pattern is meaningful.
Second-order effects
- Startups built around sorting data in the cloud face pressure to differentiate on the reliability of their analysis, not merely their ability to store or process more information.
Third-order effects
- If large-scale analysis repeatedly produces weak or misleading conclusions, verification becomes a constraint on adoption: advantage shifts toward organizations that can turn data volume into trustworthy inference.
The trend: Big data is moving from an infrastructure race over collecting and sorting information toward a competition over the quality of the conclusions drawn from it.