Researchers show how computer vision systems are susceptible to attacks by making subtle changes to images that trick them into seeing things that aren't there
TECH COMPANIES ARE rushing to infuse everything with artificial intelligence, driven by big leaps in the power of machine learning software. Tweets: @nxthompson Tweets: Nicholas Thompson / @nxthompson : The early Internet creators didn't think enough about bad actors and manipulation. Let's hope that the people building the most advanced AI systems do. @tsimonite http://www.wired.com/...
Context & Ripple Effects
This 2018 Wired piece lands at an inflection point in its own coverage arc: only weeks earlier, outlets were reporting that AI had made automated analysis of live surveillance video possible, meaning vision models were about to be trusted with real-world judgments. The demonstration that near-invisible pixel changes can make those models see things that aren't there turns that deployment enthusiasm into a security question.
Nicholas Thompson's framing — that the early Internet's builders underweighted bad actors and manipulation — sets up the throughline the corpus later confirms: by 2020 Facebook was running an AI red team that hacks its own systems to find these blind spots first, and by 2025 the cat-and-mouse between attackers and defenders using AI had become the defining dynamic.
First-order effects
- Anyone deploying computer vision on consequential decisions — surveillance operators among them, given the parallel coverage of live-video analysis — inherits a new attack surface where manipulated inputs produce confident wrong outputs.
- Security researchers gain a reproducible attack class to study, shifting adversarial machine learning from theory to an active defensive discipline.
Second-order effects
- Companies shipping vision systems are pushed toward offensive self-testing rather than passive patching — the path Facebook formalized with its internal AI red team two years later.
- Robustness against input manipulation becomes a hidden cost line in AI deployment, favoring vendors who can demonstrate resilience over those racing features alone.
Third-order effects
- If the pattern holds, every capable AI system gets a dual-use shadow — the same logic later surfaced when Microsoft researchers showed models could design toxins that evade DNA-order screening — making red-teaming and governance a structural requirement of the industry rather than an optional practice.
- The internet-era lesson Thompson invokes hardens into policy expectation: regulators and buyers increasingly treat untested AI like unpatched software, demanding evidence of adversarial testing before deployment.
The trend: AI security is moving from an academic curiosity about fooled classifiers to an institutionalized red-team-and-governance layer that every serious deployer is expected to build.