Apple says in the next couple of months it will begin scanning photos on iOS 15 and macOS Monterey devices in the US for known child abuse images

TechCrunch 2021-08-07 Zack Whittaker

Context & Ripple Effects

Apple’s plan moved from a reported consideration of on-device hashing for known abuse material to a stated rollout for its next iOS and macOS releases. Related coverage also shows Apple pairing the image-matching system with on-device Messages warnings for sexually explicit images, broadening the child-safety initiative beyond photo libraries.

The announcement has immediately become a privacy-policy test for Apple: security experts warned that governments may seek a wider scope, while Apple’s internal response frames the feature as necessary to protect children.

First-order effects

US users of iOS 15 and macOS Monterey will face a new Apple-controlled check of photos against known child-abuse-image material, making device-level scanning part of the platform’s safety tooling.
Apple must defend the rollout against the concerns documented in security experts’ privacy critique, rather than treating the feature as a routine operating-system update.

Second-order effects

Apple’s decision raises pressure on other device and messaging platforms to explain whether they will adopt comparable on-device child-safety controls or differentiate on privacy grounds.
The simultaneous photo-scanning and Messages-warning plans make Apple’s policy debate extend from stored content to family communications, increasing scrutiny of how safety features are defined and governed.

Third-order effects

If governments act on the expanded-scope concern raised by experts, device-side matching systems could become a recurring point of negotiation between platform operators, privacy advocates, and public authorities.
The episode points toward child-safety enforcement being embedded in consumer operating systems, with legitimacy depending on whether providers can constrain such tools to their stated purpose.

The trend: Consumer platforms are moving child-safety enforcement onto devices, turning the limits of local scanning into a central privacy and governance question.

Discussion

@snowden Edward Snowden on x
No matter how well-intentioned, @Apple is rolling out mass surveillance to the entire world with this. Make no mistake: if they can scan for kiddie porn today, they can scan for anything tomorrow. They turned a trillion dollars of devices into iNarcs—*without asking.* https://twi…
@snowden Edward Snowden on x
🚨🚨 Apple says to “protect children,” they're updating every iPhone to continuously compare your photos and cloud storage against a secret blacklist. If it finds a hit, they call the cops. iOS will also tell your parents if you view a nude in iMessage. https://www.eff.org/...
@swiftonsecurity SoS on x
Just to state: Apple's scanning does not detect photos of child abuse. It detects a list of known banned images added to a database, which are initially child abuse imagery found circulating elsewhere. What images are added over time is arbitrary. It doesn't know what a child is.
@bennypinkas Benny Pinkas on x
I reviewed the Apple PSI system that will be used for detecting photos with CSAM (child sexual abuse) content while keeping the contents of all non-CSAM photos encrypted and private. This system is only used in iCloud Photos — not iMessage.
@marcan42 Hector Martin on x
Good article on how this becomes a problem, even when not intentionally targeted. https://areoform.wordpress.com/ ...
@wagatwe Wagatwe Wanjuki on x
There's already been a lot of smart commentary on why this is a terrible idea. But I also wanted to add another piece of context: Violence against women and children is often used as an excuse to expand state surveillance without actually helping. https://www.apple.com/...
@marcan42 @marcan42 on x
Any automated CSAM matching system introduces an exploitable vulnerability to completely ruin someone's life.
@matthew_d_green Matthew Green on x
So the Apple scanning system just dropped. https://www.apple.com/...
@stroughtonsmith Steve Troughton-Smith on x
Wait, initial hash database comes from the authorities in any given country and isn't verified beforehand? Apple's only way to find out is w/ manual review after a user has been flagged? Using same under-paid & overworked content moderation contractors that could be infiltrated? …
@reckless Nilay Patel on x
Just to back this out farther: Apple already scans iCloud. If they had said “we are now scanning iCloud Photos for CSAM” they would have just matched the rest of the industry. Avoiding that with a complicated local scanning and hashing system created new risks entirely. https://t…
@timmarchman Tim Marchman on x
Not the main point but Facebook's decontextualized CSAM numbers are actually a serious problem according to law enforcement people I've talked to, because they are a distorted lens and mislead the public and policymakers. https://twitter.com/...
@vickerysec Chris Vickery on x
There is legal precedent which explains why Apple should not be allowed to do this- https://arstechnica.com/... The reasoning is very simple: Apple is not law enforcement. No matter how badly a commercial tech company *wants* to wield the power of law, it doesn't. Plain and simpl…
@joewintergreen Joe Wintergreen on x
the idea that a colossal tech company might suddenly decide to pursue an altruistic agenda of abuse prevention is, no joke, the most laughably barefaced complete fucking fairytale bullshit any of us will ever hear in our lives https://twitter.com/...
@samthielman @samthielman on x
This sucks. Any student of the FBI knows what this is: violating civil rights in the name of protecting the victims of revolting edge cases. The next step is always, ALWAYS, citing those violations as precedent. https://www.washingtonpost.com/ ...
@tomgara Tom Gara on x
The child abuse stuff got most of the attention yesterday but the second part seems way more expansive: iPhones will use some kind of computer vision system to search for *any* nudity in images in iMessage, not just illegal stuff https://twitter.com/...
@senblumenthal Richard Blumenthal on x
Thank you @tim_cook for taking action to help catch sexual predators & prevent the horrific abuse of children—now it's time for others in Big Tech to step up & for Congress to pass the EARN IT Act.
@evacide Eva on x
Louder, for the people in the back: it's impossible to build a client-side scanning system that can only be used for sexually explicit images sent or received by children. https://www.eff.org/...
@senblumenthal Richard Blumenthal on x
After a long & forceful push from me & @LindseyGrahamSC, Apple's plans to combat child sexual exploitation are a welcome, innovative, & bold step. This shows that we can both protect children & our fundamental privacy rights. https://www.reuters.com/...
@nytimestech @nytimestech on x
Apple unveils changes to iPhone designed to notify parents of child sexual abuse. But some privacy watchdogs are concerned about the implications. https://www.nytimes.com/...
@jcenters Josh Centers on x
Lot of Apple pundits running out to defend Apple, not many openly saying how bizarre this move is. Hard to believe Apple just up and built an expensive surveillance system that destroys their carefully crafted privacy image on their own accord.
@mala Danny O'Brien🤖 on x
So for the last few years, intelligence agency experts, like @GCHQ, have been pushing for tech companies to put remotely-controlled software to scan for unlawful content in people's devices, rather than surveil on the network. https://www.eff.org/...
@mala Danny O'Brien🤖 on x
This is bad. Really bad. And Apple was embargoing the news, so journalist may not be talking to experts, like @ohemorange and @joncallas at @EFF, about the consequences for innocent users — not just Apple users, but anyone who stores data on their devices. https://twitter.com/...
@matthew_d_green Matthew Green on x
Apple is managing an embargo of reporters right now. It's quite a remarkable thing watching them do this, goal is to make sure everyone reports this when Apple wants and on Apple's terms.
@a_greenberg Andy Greenberg on x
Seems like Apple's idea of doing iCloud abuse detection with this partially-on-device check only makes sense in two scenarios: 1) Apple is going to expand it to non-iCloud data stored on your devices or 2) Apple is going to finally E2E encrypt iCloud? https://www.wired.com/...
@eff @eff on x
Apple's filtering of iMessage and iCloud is not a slippery slope to backdoors that suppress speech and make our communications less secure. We're already there: this is a fully-built system just waiting for external pressure to make the slightest change. https://eff.org/...
@caseynewton Casey Newton on x
I tried to sort through my feelings about Apple's child safety announcements today. The risk of the slippery slope is real — but there are also real harms taking place on iCloud today, and it's good that Apple has chosen to pay attention to them https://www.platformer.news/ ... h…
@benedictevans Benedict Evans on x
There is something quite theological in Apple's idea that doing something in your devices is private and doing exactly the same thing in the cloud is not.
@captn3m0 Nemo on x
Reminder that the Indian government has successfully bullied Apple in the past to make iOS changes (TRAI). Could they pressure Apple into adding new hashes (of political memes) in the next iOS release? I'd rather not find out. https://twitter.com/...
@caseynewton Casey Newton on x
@daiwaka for sure, but I think I'm OK with this tradeoff? FB reported 15.8M cases of CSAM to NCMEC in 2019; Apple reported 205. Seems likely that there is massive under-reporting of very bad stuff on iOS
@matthew_d_green Matthew Green on x
Because surely it will ensure this, right? You'd want to ensure that Apple (or someone who hacks Apple's servers) can't change the database selectively to target it to you — and have a normal CSAM database for everyone else.
@bzamayo Benjamin Mayo on x
If the new Apple photo scanning stuff was looking at any and all photos stored on your device, I think there'd be somewhat of an ethical debate to be had. But it's just applied to iCloud Photos, which makes it no different than any other major web service or social network.
@jsrailton John Scott-Railton on x
NEW: @Apple to start checking photos on iPhone & iCloud against a blacklist of CSAM imagery. To watch out for: Now that the tech is on the table, how soon till China & other authoritarians w/political blacklists lean on Apple to search citizens' phones? https://www.apple.com/... …
@josephfcox Joseph Cox on x
New: much more interesting than Apple checking iCloud for known child abuse images is Apple will scan all images sent and received by children in Messages app to detect nudity. This is for new images, not already known. Expert concerned could be expanded https://www.vice.com/...
@santiagomayer_ Santiago Mayer on x
We can all agree we must do everything we can to stop child abuse. ...But having Apple scan all my pictures and potentially send them to human reviewers is not something I'm comfortable with. https://techcrunch.com/...

Chronicles