Apple says in the next couple of months it will begin scanning photos on iOS 15 and macOS Monterey devices in the US for known child abuse images

The feature lands later this year, but already faces resistance from security and privacy experts — Later this year, Apple will roll …

TechCrunch 2021-08-06 Zack Whittaker

Context & Ripple Effects

Apple’s move follows reports that it was considering on-device hashing to match known abuse material, shifting a contemplated technical approach into a stated U.S. rollout plan. The related coverage also shows Apple extending child-safety interventions beyond photo matching through on-device Messages warnings for explicit images.

Security and privacy experts have already challenged the device-scanning plan, while Apple’s internal response frames the feature as necessary to protect children. That conflict makes the implementation a test case for how far governments may press to expand the scope of safety scanning.

First-order effects

U.S. users of iOS 15 and macOS Monterey face a new photo-matching safeguard when Apple begins the rollout, making device-level abuse-image detection part of Apple’s platform policy.
Apple must defend the feature’s privacy boundaries immediately as security and privacy experts contest the plan and Apple publicly answers those concerns.

Second-order effects

Apple’s parallel Messages protections make child-safety detection a broader family of on-device controls, rather than an isolated photo-library feature.
Expert warnings that governments may seek broader scanning authority put pressure on Apple to make the feature’s limits legible and credible to users and policymakers.

Third-order effects

If device-level safety scanning becomes an accepted platform control, the central governance question shifts from whether platforms can scan to who defines the material and circumstances that qualify for scanning.
The dispute points toward public-safety AI governance in which technical safeguards and limits on government-driven scope are evaluated together, not separately.

The trend: Consumer platforms are moving safety enforcement onto devices, making the boundaries and governance of local scanning a core privacy issue.

Discussion

@snowden Edward Snowden on x
🚨🚨 Apple says to “protect children,” they're updating every iPhone to continuously compare your photos and cloud storage against a secret blacklist. If it finds a hit, they call the cops. iOS will also tell your parents if you view a nude in iMessage. https://www.eff.org/...
@evacide Eva on x
Louder, for the people in the back: it's impossible to build a client-side scanning system that can only be used for sexually explicit images sent or received by children. https://www.eff.org/...
@benedictevans Benedict Evans on x
There is something quite theological in Apple's idea that doing something in your devices is private and doing exactly the same thing in the cloud is not.
@a_greenberg Andy Greenberg on x
Seems like Apple's idea of doing iCloud abuse detection with this partially-on-device check only makes sense in two scenarios: 1) Apple is going to expand it to non-iCloud data stored on your devices or 2) Apple is going to finally E2E encrypt iCloud? https://www.wired.com/...
@nytimestech @nytimestech on x
Apple unveils changes to iPhone designed to notify parents of child sexual abuse. But some privacy watchdogs are concerned about the implications. https://www.nytimes.com/...
@caseynewton Casey Newton on x
I tried to sort through my feelings about Apple's child safety announcements today. The risk of the slippery slope is real — but there are also real harms taking place on iCloud today, and it's good that Apple has chosen to pay attention to them https://www.platformer.news/ ... h…
@vladsavov Vlad Savov on x
Wonder what 2019's Apple would think of today's Apple. https://twitter.com/...
@bzamayo Benjamin Mayo on x
If the new Apple photo scanning stuff was looking at any and all photos stored on your device, I think there'd be somewhat of an ethical debate to be had. But it's just applied to iCloud Photos, which makes it no different than any other major web service or social network.
@jakecoco Critical Jake Theory on x
Couldn't someone just hack this and then digitally plant images on the phones of dissenters? Quite powerful...Seems like an awful idea to me https://twitter.com/...
@samfbiddle Sam Biddle on x
re: today's Apple news I'm trying to think of another product that will, essentially, automatically report you to the police if it determines you've broken the law *based entirely on private conduct*—Can't come up with other examples yet.
@caparsons Christopher Parsons on x
Earlier today Apple introduced a whole new set of features to monitor what individuals do on their devices. Those features, currently, are intended to address child exploitation online. You can find the details at: https://www.apple.com/...
@carnage4life Dare Obasanjo on x
Being the only major cloud storage service that doesn't scan photos for CSAM was untenable but it's strange how Apple uses client side code as a privacy get out of jail card. The privacy issue is scanning & reporting a user's photos not where code runs. https://techcrunch.com/...
@migueldeicaza @migueldeicaza on x
One interesting outcome of this move by Apple is that it addresses one major concern from policy makers. Not doing anything could lead to end to end crypto being weakened or prohibited across the board by US/EU. This takes away a major source of pressure on tech vendors.
@sweatieangle @sweatieangle on x
Yeah can't stress enough how much I'm really not happy with a robot assessing my nudes. I'm going to be mad af either way, too, because if it shows them to the cops that's a huge breach of privacy, and if it doesn't show them to the cops it's telling me I look old, rude. https://…
@reneritchie Rene Ritchie on x
What are your thoughts on Apple's new Child Safety protections? It checks hashes of images going to iCloud Photo Library, on-device, against known Child Sexual Abuse Material, and if a high enough number of matches are found, prompts Apple to initiate action #poll
@senblumenthal Richard Blumenthal on x
Thank you @tim_cook for taking action to help catch sexual predators & prevent the horrific abuse of children—now it's time for others in Big Tech to step up & for Congress to pass the EARN IT Act.
@malwarejake Jake Williams on x
Anyone thinking this won't be abused for other purposes clearly has never worked in government (and honestly isn't even paying attention). https://twitter.com/...
@k8em0 Katie Moussouris on x
Apple: Stink different https://twitter.com/...
@samifathi_ Sami Fathi on x
Apple does normal journalism practice?! How incredibly scandalous!!! How dare they! https://twitter.com/...
@unapologeticyt @unapologeticyt on x
This is the opening salvo in the assault to end what shred of autonomy and privacy from corporations and the state that we have left https://twitter.com/...
@freedomofpress @freedomofpress on x
Apple's newly announced “protections” are a terrible step backwards for privacy on its devices, and create a backdoor ripe for governmental abuse https://www.eff.org/...
@caseynewton Casey Newton on x
Also worth saying: child safety advocates have been pushing Apple to do this for a very long time. I talked with Thorn's @juliecordua, who told me she was heartened by today's moves https://www.platformer.news/ ... https://twitter.com/...
@eff @eff on x
Apple's filtering of iMessage and iCloud is not a slippery slope to backdoors that suppress speech and make our communications less secure. We're already there: this is a fully-built system just waiting for external pressure to make the slightest change. https://eff.org/...
@andyyen Andy Yen on x
“Apple will enable all the suspect photos to be decrypted and, if apparently illegal, passed on to the relevant authorities.” —> Imagine what this could do in the hands of an authoritarian govt? Don't have to imagine too hard either given Apple's cozy relationship with China. htt…
@petersterne Peter Sterne🌹 on x
@Whey_standard @SwiftOnSecurity Apple thinks photo scanning is non-negotiable — that for legal and PR reasons, you can't be a major consumer tech company and not scan users' photos — so the only way to encrypt photos on-device was to develop & implement client-side scanning. I ho…
@mistresssnowphd @mistresssnowphd on x
am I reading this wrong or are they planning on comparing all photos to a massive database of child sexual exploitation materials a database in which I'd imagine CSEM survivors would prefer not to be included but I doubt they were asked for consent anyway, so who knows https://tw…
@laurengoode Lauren Goode on x
Apple has “driven a wedge between privacy and cryptography experts who see its work as an innovative new solution and those who see it as a dangerous capitulation to government surveillance.” https://www.wired.com/...
@kaepora Nadim Kobeissi on x
Apple managed an embargo across every tech journalist. They also invited a handful of cryptographers to review their design, but not one that also engaged in public education. The lengths they've put towards controlling the narrative on this rollout is worth noting. https://twitt…
@senblumenthal Richard Blumenthal on x
After a long & forceful push from me & @LindseyGrahamSC, Apple's plans to combat child sexual exploitation are a welcome, innovative, & bold step. This shows that we can both protect children & our fundamental privacy rights. https://www.reuters.com/...
@malloryknodel Mallory on x
It gets worse. The official news from Apple today goes beyond images. It breaks the e2ee of iMessage. https://cdt.org/...
@snazzyq Quinn Nelson on x
Look, I can understand disagreement towards this action but one thing needs to be made clear: Apple isn't just scanning your phone for images of your tiny pp. They're only looking for cryptographic hashes of child porn that's already known or in circulation provided to them.
@gojomo Gordon Mohr on x
personally, I think this sort of “if you have nothing to hide, you have nothing to fear” attitude should disqualify someone from being a “privacy chief” (from: https://www.nytimes.com/...) https://twitter.com/...
@swiftonsecurity SoS on x
Companies doing CSAM (used to be called 'CP') scanning is not new, they don't want to be pedo hosting dumping grounds, but the incredible nuances in pushing this technology stack to the client-side is where Apple's change is what's being discussed. I'm still watching discourse. h…
@swiftonsecurity SoS on x
Just to state: Apple's scanning does not detect photos of child abuse. It detects a list of known banned images added to a database, which are initially child abuse imagery found circulating elsewhere. What images are added over time is arbitrary. It doesn't know what a child is.
@zackwhittaker Zack Whittaker on x
New: Apple confirmed it will begin scanning for known child sexual abuse materials (CSAM) on iPhones and Macs starting later this year. The scanning will roll out initially in the U.S., but already faces resistance from security and privacy advocates. https://techcrunch.com/...
@mistresslienne @mistresslienne on x
Let's be real tho. This is another move towards fascist surveillance state done in the name of “protecting the children”. Most things done w the intention of “doing good” in this country are actually terrible https://twitter.com/...
@snazzyq Quinn Nelson on x
All of this sounds fantastic. I do feel bad that once an iCloud user exceeds the CSAM threshold it'll go to manual review for verification. I can't think of a worse job that having to view child sexual abuse material all day long. 😔 https://twitter.com/...
@katebevan Kate Bevan on x
This is a really useful discussion of Apple's plans to scan images to detect CSAM. It's smart and elegant, but the concern is clearly mission creep. https://twitter.com/...
@jacknicas Jack Nicas on x
Apple is aiming to prevent child sexual abuse with two new tools: 1. A pretty complex system to spot images of child sexual abuse uploaded to iCloud. 2. A feature that flags to parents when their children send or receive nude photos in texts. https://www.nytimes.com/...
@cendemtech @cendemtech on x
Our release details more of our concerns: #encryption https://cdt.org/...
@migueldeicaza @migueldeicaza on x
A better way of phrasing it: Apple likely chose to implement this system on their own terms, rather than be forced to implement a system imposed on them by legislators that might have had worst outcomes.
@dinodaizovi @dinodaizovi on x
I'm eagerly awaiting more details on this, but I'm not liking what it's looking like so far. This could set a dangerous, worldwide precedent of pushing continuous scanning down to users' devices for whatever content is considered illegal in a particular country. https://twitter.c…
@ryantate Ryan Tate on x
China forced Uyghurs to install an app that scanned for illegal files. Many ended up in concentration camps. Apple today forces this spying on Americans, declaring, “If you're storing [illegal] material, yes, this is bad for you.” https://www.vice.com/... https://www.nytimes.com/…
@koush @koush on x
iCloud and iMessage will be scanning your photos under the guise of “protecting children”, ending their claim to end to end encryption. A tried and true strategy: incite a moral panic and leverage outrage to erode user privacy. https://www.eff.org/...
@malwarejake Jake Williams on x
If your response to very legitimate privacy and abuse concerns is “but the children!”, please unfollow me now. Then read up on parallel construction (and its abuses) because it's context you'll definitely need to fully understand the situation. https://twitter.com/...
@wimremes @wimremes on x
I honestly can't believe that anyone at Apple thinks this is a great idea. They weren't “thinking differently”, they were not thinking at all. Waiting until they SWAT their first parent because they had a naked pic of their kid. https://twitter.com/...
@santiagomayer_ Santiago Mayer on x
We can all agree we must do everything we can to stop child abuse. ...But having Apple scan all my pictures and potentially send them to human reviewers is not something I'm comfortable with. https://techcrunch.com/...
@sophieladder @sophieladder on x
some thoughts on this: - they are only checking the images against known child sexual abuse material (CSAM). this makes me less concerned than if they were using AI (recall nudity AIs flagging pictures of sand dunes) which could incorrectly identify unrelated media https://twitte…
@billt Bill Thompson on x
This should be apparent to everyone. However, this is not a new idea - this from 2013 https://www.theguardian.com/ ... - and so we should be asking broader questions around how we make the net safer, and not simply reacting to the news cycle. https://twitter.com/...
@briannawu Brianna Wu on x
This is the worst idea in Apple history, and I don't say that lightly. It destroys their credibility on privacy. It will be abused by governments. It will get gay children killed and disowned. This is the worst idea ever. https://www.nytimes.com/...
@swiftonsecurity SoS on x
Previous discussion on possibilities of false-positive detections when scanning for CSAM. Apple's approach is even more guarded and requires a threshold to be met to eliminate individual detections. https://twitter.com/...
@feross @feross on x
@Apple This @EFF write-up explains why this backdoor is a disaster: “To say that we are disappointed by Apple's plans is an understatement.” “It is a shocking about-face for users who have relied on the company's leadership in privacy and security” https://www.eff.org/...
@juliet_shen Juliet Shen on x
Regarding the Apple and CSAM scanning news, the biggest difference is that the extension of this canning to client photos that haven't been shared anywhere, AND the use of a neural network to detect unknown CSAM. They've been scanning iCloud for a while https://nakedsecurity.soph…
@dangillmor Dan Gillmor on x
That iPhone is Apple's phone, not yours. And this completely betrays the company's pious privacy assurances. This is just the beginning of what governments everywhere will demand. All of your data will be fair game. If you think otherwise, you're terminally naive. https://twitter…

Chronicles