Apple says NeuralHash flaw found by researchers in CSAM detection system was in a generic version of the software and not in the final version it plans to use

Apple said the version of NeuralHash analyzed by researchers is not the final version that will be used for iCloud Photos CSAM detection.

VICE 2021-08-19 Joseph Cox

Context & Ripple Effects

Apple had already outlined an iCloud Photos system built around on-device matching and a 30-match threshold before manual account review. The researchers' analysis therefore became a test of whether the technical materials available for scrutiny represented the system Apple intended to deploy.

The dispute also sits within a rollout criticized as being developed without sufficient outside input; Apple ultimately did not launch the iCloud Photos detection tool and redirected its anti-CSAM work toward Communication Safety features.

First-order effects

Apple's response limits the researchers' finding to a generic NeuralHash implementation, leaving the final implementation's resistance to the reported flaw unverified in the public record.
Researchers and critics evaluating Apple's proposed system must distinguish the analyzed software from the unreleased version Apple says it plans to use.

Second-order effects

The gap between a public technical artifact and the asserted final system raises the value of independent access to the deployed implementation, rather than assurances about changes not yet available for review.
Apple's promised match threshold does not resolve concerns about the hashing component itself, keeping scrutiny focused on how the full detection pipeline is validated.

Third-order effects

The later cancellation shows how technical assurance disputes can become product-governance failures when a sensitive enforcement feature lacks a review path trusted by outside researchers.
Client-side detection proposals are likely to face durable pressure for verifiable implementation details, not just descriptions of thresholds and safeguards.

The trend: Safety systems that inspect user content are becoming an AI-enforcement governance issue in which independent auditability determines whether technical safeguards are credible.

Discussion

@atomicthumbs Site Specific Carnivorous Occurrence on x
someone found Apple's Neurohash CSAM hash system already embedded in iOS 14.3 and later, and managed to export the MobileNetV3 model and rebuild it in Python how many months until there's a GAN that creates innocuous images that're detected as CSAM? https://www.reddit.com/...
@sarahjamielewis Sarah Jamie Lewis on x
Both these images have NeuralHash: 1e986d5d29ed011a579bfdea Just a reminder that visually similar images are not necessarily semantically similar images. https://twitter.com/...
@josephfcox Joseph Cox on x
“the only thing you can do [...] is annoy Apple's response team with some garbage images until they implement a filter to eliminate those garbage false positives in their analysis pipeline” https://www.vice.com/... https://twitter.com/...
@ghidraninja Stacksmashing on x
Surprise: Someone already built a first working collision/preimage-attack for Apple's NeuralHash model for CSAM detection. https://github.com/... https://twitter.com/...
@matthew_d_green Matthew Green on x
To be clear, the threat model in Apple's system is “accidental false positives in our hash function are very unlikely” and whether that's true or not, it says nothing about deliberately-constructed false positives.
@matthew_d_green Matthew Green on x
It was always fairly obvious that in a perceptual hash function like Apple's, there were going to be “collisions” — very different images that produced the same hash. This raised the possibility of “poisoned” images that looked harmless, but triggered as child sexual abuse media.
@matthew_d_green Matthew Green on x
What's the German word for “don't build security systems that rely on obscurity but can't keep important details confidential for more than two weeks.”
@matthew_d_green Matthew Green on x
But there's a difference between saying “yeah that's almost certain to happen, in theory” and seeing it happen in real life. Apple went out of their way to keep the hash function secret — because they knew the risks.
@marcan42 @marcan42 on x
Also, apparently Apple's neural network, by virtue of having 200+ (!) layers and due to floating point rounding issues, actually produces wildly different hashes on different hardware (9 bits difference between iPad and M1 Mac!). That's... garbage. That's 9 bits of match noise.
@theregister @theregister on x
Apple didn't engage with the infosec world on its CSAM scanning - so get used to a slow drip feed of revelations https://www.theregister.com/ ...
@sarahjamielewis Sarah Jamie Lewis on x
As I said, finding specific (funny) collisions is trivial for perceptual hashes. To be fair to Apple NeuralHash does seem at least somewhat resistant to -random- collisions (over the small set of tens of thousands of images I've thrown at it today...). https://twitter.com/...
@marcan42 @marcan42 on x
It's not hard at all to attack such a hash to make it produce false positives. Say I am law enforcement and I want access to your photos. I send you >30 messages with non-CSAM but colliding images. Your phone now thinks you have CSAM and grants Apple access to your data.
@matthew_d_green Matthew Green on x
They say attacks only get better. But this is really quite a lot better, in less than a day. https://twitter.com/...
@balajis Balaji Srinivasan on x
Social media gave us volunteer thought police. And artificial intelligence now gives them the tools to frame anyone by planting false data on their phone. This could be the new SWATing. https://twitter.com/...
@sarahjamielewis Sarah Jamie Lewis on x
The Apple system dedupes photos, but burst shots are semantically *different* photos with the same subject - and an unlucky match on a burst shot could lead to multiple match events on the back end if the system isn't implemented to defend against that.
@balajis Balaji Srinivasan on x
Paradoxically, this whole debacle may make it harder to do what Apple nominally set out to do. Under many reasonable models of the world, many of the positives of this billion-person dragnet will be false positives - or planted positives. https://twitter.com/...
@sarahjamielewis Sarah Jamie Lewis on x
Hashes generated using the instructions / script found here: https://github.com/...
@meanhash @meanhash on x
Stop using your iPhone for pictures. Unless you want a real person to see all of your private pictures. https://www.theverge.com/...
@vickerysec Chris Vickery on x
And guess what has now been proven— “Apple's child-abuse scanner has serious flaw, researchers say” “Researchers have produced a collision in Apple's NeuralHash system” https://www.theverge.com/... https://twitter.com/...
@hoeglaw Richard Hoeg on x
Hash collisions with a tech company reporting you to law enforcement for some of the most heinous of crimes at stake. What could go wrong? https://twitter.com/...
@kennwhite Kenn White on x
Apple responds to reports of trivial NeuralHash collisions. TL;DR: A second, blackbox server-side image scanning algo (possibly PhotoDNA) would also have to be triggered as well before flagging for human review. https://www.theverge.com/...
@vteagueaus Vanessa Teague on x
This could allow an attacker to generate false matches to the CSAM database, triggering apparent matches from innocent-looking photos. The PSI protocol doesn't protect against this. (n.b. I haven't checked this collision or the claim it matches NeuralHash.) @bipr @rossjanderson h…
@secparam Ian Miers on x
Apple's hash function has been found. The thing to look for isn't collisions, though those are useful, it's if you can use it to search for say “all pictures of a gun” or all pictures with this text in it. Instead of just a specific picture. https://twitter.com/...
@dinodaizovi Dino A. Dai Zovi on x
The next question is whether the latter image could be applied as a perturbation to an existing image? Or to answer the question of whether NerualHash collisions could be created to make an arbitrary benign image appear to be a known bad image. https://twitter.com/...
@freedomofpress @freedomofpress on x
Independent security researchers appear to have generated images that could fool Apple's controversial new CSAM detection/surveillance tech into flagging false matches https://twitter.com/...
@vickerysec Chris Vickery on x
What this means is, according to Apple's “never wrong” photo detection algorithm, these two pictures are identical and there is no difference whatsoever in their appearance. Apple wants prosecutions based on the software that thinks these two pictures are identical: https://twitt…
@runasand Runa Sandvik on x
Apple said security researchers could verify its claims about the child safety feature. So they did. These two images generate the same hash. https://twitter.com/...
@ericrichards22 Eric Richards on x
“There's no way we're going to ever have hash collisions” https://twitter.com/...
@matthew_d_green Matthew Green on x
Apple should commit to publishing its algorithms so that researchers can try to develop “adversarial” images that trigger the matching function, and see how resilient the tech is. (I will be pleasantly but highly surprised if Apple does this.)
@ukzak Zak Doffman on x
Apple: Our production version won't make mistakes like this. It's secret and magic. Also Apple: We will have furtive human reviewers in the loop, so you'll never know how often we screw up and our people have to check out your photos. That's how “magic” works in the real world. h…
@r0wdy_ @r0wdy_ on x
Instead of just respecting privacy now Apple is going to need a build a team of people to sit around all day investigating CSAM hash collisions https://twitter.com/...
@josephfcox Joseph Cox on x
New: researchers are abuzz with claims that NeuralHash, tech underpinning Apple's CSAM, may mistake two clearly different photos as being the same. But Apple says this is a generic version of the tech, not final, the overall system is built to expect this https://www.vice.com/...
@jason_koebler Jason Koebler on x
It's been thing after thing since Apple has announced this https://www.vice.com/...
@motherboard @motherboard on x
Apple told Motherboard that the version of NeuralHash that users on GitHub tested is a generic version, and not the one final version that will be used for iCloud Photos CSAM detection. https://www.vice.com/...
@matthew_d_green Matthew Green on x
That was surprisingly fast. https://twitter.com/...
@jacknicas Jack Nicas on x
The tool would be the first technology built into a phone's operating system that can look at a person's private data and report it to authorities. https://www.nytimes.com/...

Chronicles