Apple is reportedly considering using on-device hashing algorithms to match photos with known child abuse material, a potentially problematic practice
Apple is reportedly set to announce new photo identification features that will use hashing algorithms to match the content of photos … Source: @matthew_d_green .
9to5MacBenjamin Mayo
Context & Ripple Effects
Apple had already used hashes of known child-abuse images in email enforcement, as a search warrant described, making the reported photo-matching plan an expansion of an existing detection approach rather than a wholly new safety capability.
Related coverage shows the report was quickly followed by Apple's stated plan to scan photos on iOS 15 and macOS Monterey, while security experts warned that device-scanning scope could broaden under government pressure. The issue is therefore where the matching occurs and what governance limits constrain it.
First-order effects
Apple’s proposed photo-identification feature would place matching for known child-abuse material on users’ iOS and macOS devices, extending the company’s use of hashes beyond the email workflow described in prior coverage.
Apple must defend the privacy and abuse-resistance of the system as security experts challenge the implications of scanning users’ devices.
Second-order effects
The accompanying on-device Messages warnings for sexually explicit photos broadens Apple’s child-safety push from identifying known illicit material to moderating family communications, increasing scrutiny of device-level safety features.
Governments and child-safety stakeholders gain a concrete enforcement mechanism to evaluate, while Apple’s privacy commitments face pressure to define whether the matching database and use cases can expand.
Third-order effects
If device-side matching becomes an accepted safety-control model, public-safety AI governance will increasingly hinge on who controls reference databases, review thresholds, and the process for adding new detection targets.
The dispute sets a durable precedent: consumer-device privacy will be judged not only by encryption and cloud access, but by the policy constraints on computation performed locally.
The trend: Child-safety enforcement is moving into consumer devices, making governance of on-device detection systems a central privacy boundary.
I've had independent confirmation from multiple people that Apple is releasing a client-side tool for CSAM scanning tomorrow. This is a really bad idea.
A small update from last night. I described Apple's matching procedure as a perceptual hash function. Actually it's a “neural matching function”. I don't know if that means it will also find *new* content on your device or just known content. https://twitter.com/...
So I wrote this previous thread in a hurry and didn't take time to spell out what it means, and what the background is. So let me try again. https://twitter.com/...
This means that, depending on how they work, it might be possible for someone to make problematic images that “match” entirely harmless images. Like political images shared by persecuted groups. These harmless images would be reported to the provider.
A number of people pointed out that these scanning technologies are effectively (somewhat limited) mass surveillance tools. Not dissimilar to the tools that repressive regimes have deployed — just turned to different purposes.
The ability to add scanning systems like this to E2E messaging systems has been a major “ask” by law enforcement the world over. Here's an open letter signed by former AG William Barr and other western governments. https://www.justice.gov/...
Regardless of what Apple's long term plans are, they've sent a very clear signal. In their (very influential) opinion, it is safe to build systems that scan users' phones for prohibited content. That's the message they're sending to governments, competing services, China, you.
Will they deploy this for iPhones in China? What if the govt asks (or passes a law) to know who has politically sensitive photos on their phone? https://twitter.com/...
If you have auto-save of received images on Whatsapp etc. enabled (the default setting), this means anyone can send you illegal content, and the police may pick you up shortly after https://twitter.com/...
These are bad things. I don't particularly want to be on the side of child porn and I'm not a terrorist. But the problem is that encryption is a powerful tool that provides privacy, and you can't really have strong privacy while also surveilling every image anyone sends.
Hashes aren't perfect by any means, but AFAIK law enforcement can't charge someone on the basis of a matched hash - they have to find CSAM on a suspect's device. The hash match provides probable cause. https://twitter.com/...
Initially I understand this will be used to perform client side scanning for cloud-stored photos. Eventually it could be a key ingredient in adding surveillance to encrypted messaging systems.
Initially Apple is not going to deploy this system on your encrypted images. They're going to use it on your phone's photo library, and only if you have iCloud Backup turned on. So in that sense, “phew”: it will only scan data that Apple's servers already have. No problem right?
Porn is always the lead excuse to deploy privacy-crippling technologies. CSAM today, classified material next, and eventually, whatever any power deems “objectionable” tomorrow. This playbook is so tired. Not fooled, @Apple https://twitter.com/...
Once again, this ultimately comes back to a simple question: Do our devices work for us, or for the manufacturers? Do we have a right to expect that they're designed to work in our best interests, not someone else's? https://twitter.com/...
This is bad. For all the reasons in the thread, AND This tool won't be contained to child sexual abuse material. Tools developed to fight it are often inappropriately redeployed to try to detect “terrorist content”- regardless of false positives. https://mnemonic.org/... (1/2) ht…
There are a lot of problems with this idea. Yes — client side scanning (and encrypted images on server) is better than plaintext images and server-side scanning. But to some extent, the functionality is the same. And subject to abuse...
The theory is that you will trust Apple to only include really bad images. Say, images curated by the National Center for Missing and Exploited Children (NCMEC). You'd better trust them, because trust is all you have.
(There are some other fancier approaches that split the database so your phone doesn't even see it — the databases are actually trade secrets in some cases. This doesn't change the functionality but makes the system even harder to check up on. Apple may use something like this.)
The way this will work is that your phone will download a database of “fingerprints” for all of the bad images (child porn, terrorist recruitment videos etc.) It will check each image on your phone for matches. The fingerprints are ‘imprecise’ so they can catch close matches.
Dumbed-down explanation: Apple's iPhones will soon start secretly calling the police if they find photos on your phone that match fingerprints of photos depicting child abuse and any content eventually deemed objectionable. This can/will be generalized to secure messaging https:/…
For the past decade, providers like Apple, WhatsApp/Facebook, Snapchat, and others have been adding end-to-end encryption to their text messaging and video services. This has been a huge boon for privacy. But governments have been opposed to it.
But ask yourself: why would Apple spend so much time and effort designing a system that is *specifically* designed to scan images that exist (in plaintext) only on your phone — if they didn't eventually plan to use it for data that you don't share in plaintext with Apple?