Apple's plan to find CSAM should have centered around scanning images on iCloud servers, not on users' devices, where there is a greater expectation of privacy

including a number of non-obvious but critical ones. It's also why hypos as a threat assessment tool will only get you so far. https://twitter.com/... Greg Howell / @g_howell : @matthew_d_green If @Apple is going to operate on behalf of law enforcement, the 4th Amendment should apply to Apple's actions. This is no less than an end-run around the Constitution. @EFF Harry McCracken / @harrymccracken : Does seem like Apple's commitment to the philosophy that doing thing on-device is more private than doing so in the cloud led the company to misjudge how many, many others would see this. https://stratechery.com/... Boaz Barak / @boazbaraktcs : This distinction of “capability vs policy” is crucial. If you create a situation where a device is capable of scanning users' private content, and the only privacy protection is code under company's control, then code is at risk from (a) court orders or (b) Pegasus-like spyware. https://twitter.com/... Alec Muffett / @alecmuffett : My suspicion is that Apple are fibbing about their capaability or desire to resist the @ukhomeoffice rocking up to them with a #TechnicalCapabilityNotice under the Regulation of Investigatory Powers act, and thereby being forced to snoop on iPhones. https://twitter.com/... Hakan / @hatr : „It's truly disappointing that Apple got so hung up on its particular vision of privacy that it ended up betraying the fulcrum of user control: being able to trust that your device is truly yours." https://stratechery.com/... via @fubits Cathy Gellis / @cathygellis : In terms of US law, 4th Amendment law about how police can't pay the landlord to let them into a tenant's apartment should apply here to limit the parade of horribles. But (a) it might not, and (b) it's of no use elsewhere in the world there's no such limit on police. https://twitter.com/... Steve Kovach / @stevekovach : Literally the graf I copied/pasted to my colleagues covering this https://twitter.com/... Jay Cuthrell / @jaycuthrell : The author appears to suggest that privacy focused Apple should use the solutions employed by [ checks notes ] Facebook and Google. 🧐🤔😳 https://twitter.com/... Antonio García Martínez / @antoniogm : Apple's move to scan the photos *on your phone, with code running on your phone* has ignited a storm of debate. This differs from how FB does things. As PR readers know, this is a general move toward on-device everything, and Apple's plan is in line with that. https://twitter.com/... John Wilson / @johnwilson : The only photos that are scanned against the hashed database are those in iCloud Photos, which are destined for iCloud. So what exactly is he saying? https://twitter.com/... Steve Kovach / @stevekovach : This is a good one —> https://twitter.com/... Dieter Bohn / @backlon : I said this on the Vergecast too. At the end of the day it's inevitably policy. In this case in particular all the technical layers are designed to make you think it's technology that's protecting your privacy. It's not. It's just Apple policy. https://twitter.com/... Michael Gartenberg / @gartenberg : There *is* something I can do If I'm that concerned. I can choose to not do business with Apple. If enough people choose that path, Apple would be likely change their policy. That's unlikely to happen. After all, how many Apple users are ok with Facebook & Google? https://twitter.com/... @stratechery : Apple's Mistake While it's possible to understand Apple's motivations behind its decision to enable on-device scanning, the company had a better way to satisfy its societal obligations while preserving user privacy. https://stratechery.com/...

Stratechery 2021-08-10 Ben Thompson

Context & Ripple Effects

Apple’s proposed on-device scanning for iCloud Photos put its privacy architecture at the center of a child-safety enforcement debate. Critics and security experts in the related coverage argue that code operating on users’ devices creates a new point of potential compulsion or abuse, rather than keeping the function within Apple’s cloud service.

The backlash was not confined to Apple: WhatsApp called the approach a privacy setback, while later coverage characterized Apple’s handling of CSAM and child safety as an effort undertaken without expert input. The dispute matters because Apple’s usual on-device privacy rationale is being tested against an enforcement function users may view differently.

First-order effects

Apple’s iCloud Photos users face a proposed shift in which CSAM detection occurs on their devices, making the device itself part of Apple’s enforcement system.
Apple must defend why on-device processing is more appropriate than server-side scanning as critics raise privacy, security, and Fourth Amendment concerns.

Second-order effects

WhatsApp and other privacy-focused services gain a clear contrast with Apple’s model, increasing pressure on Apple to explain the safeguards and limits of its approach.
The controversy makes client-side enforcement code a focal point for security researchers and legal critics, because court-compelled changes or spyware exploitation are identified risks in the related coverage.

Third-order effects

If platforms increasingly move safety enforcement onto personal devices, the privacy boundary will be defined less by where data is stored and more by what code a provider can run locally.
The episode highlights a durable policy conflict: measures framed as privacy-preserving or safety-enhancing can also reinforce the control large platforms exert over their ecosystems, a tension raised in the privacy-versus-competition debate.

The trend: Child-safety enforcement is becoming a test of whether on-device privacy designs can retain public trust when they also create a local enforcement surface.

Discussion

@matthew_d_green Matthew Green on x
I think these two paragraphs get to the heart of what is so disturbing about Apple's photo scanning initiative. https://stratechery.com/... https://twitter.com/...
@matthew_d_green Matthew Green on x
Somebody proposed the following scenario to me, and I'm curious what the law is. 1. US DoJ approaches NCMEC, asks them to add non-CSAM photos to the hash database. 2. When these photos trigger against Apple users, DoJ sends a preservation order to Apple to obtain customer IDs.
@pwnallthethings @pwnallthethings on x
This is a good question. The short answer is “DOJ would not ask, and if asked NCMEC would not comply”. But for those wanting a longer answer, here we go... https://twitter.com/...
@sjmurdoch Steven Murdoch on x
Apple commit to challenging requests to expand their CSAM detection to other material. So did UK ISPs, but they lost in court and did it anyway. Will Apple leave a market if put in the same position? https://www.apple.com/... h/t @AlexMartin https://twitter.com/...
@ngleicher Nathaniel Gleicher on x
This is a very good walkthrough of all the soft, hard, legal, technical, societal, and institutional protections at play — including a number of non-obvious but critical ones. It's also why hypos as a threat assessment tool will only get you so far. https://twitter.com/...
@normative Julian Sanchez on x
While I suspect they'd push back on this for institutional credibility reasons, it's worth noting how weird the status of NCMEC is. They're a nominally private 501(c)(3) mostly funded by DOJ & designated by statute to receive CSAM reports from telecoms & platforms. https://twitte…
@reckless Nilay Patel on x
This, from @benthompson, is the crux of the matter. https://stratechery.com/... https://twitter.com/...
@matthew_d_green Matthew Green on x
Now obviously NCMEC might say “no” to the request, but they might not, especially if it could be posed as protecting children. And Apple might only see hashes, so wouldn't know this happened. Preservation orders don't require judicial review. https://papers.ssrn.com/...
@alecmuffett Alec Muffett on x
My suspicion is that Apple are fibbing about their capaability or desire to resist the @ukhomeoffice rocking up to them with a #TechnicalCapabilityNotice under the Regulation of Investigatory Powers act, and thereby being forced to snoop on iPhones. https://twitter.com/...
@howelloneill Patrick Howell O'Neill on x
Epic is 40% owned by Tencent, one of the chief architects of the Great Firewall of China. https://gizmodo.com/... This isn't a “and what about” at all, for what it's worth, just a relevant and uncomfortable fact worth repeating as we talk about governments strong arming big tech …
@missquickstep Layla Mah on x
Let's be real, Apple abandoned the idea that their customers actually “owned” their devices in the traditional sense long ago. From locked down boot loaders to app store monopoly, there's a long history of “Apple knows best” and what's best for “Apple is best for it's users” http…
@harrymccracken Harry McCracken on x
Does seem like Apple's commitment to the philosophy that doing thing on-device is more private than doing so in the cloud led the company to misjudge how many, many others would see this. https://stratechery.com/...
@boazbaraktcs Boaz Barak on x
This distinction of “capability vs policy” is crucial. If you create a situation where a device is capable of scanning users' private content, and the only privacy protection is code under company's control, then code is at risk from (a) court orders or (b) Pegasus-like spyware. …
@g_howell Greg Howell on x
@matthew_d_green If @Apple is going to operate on behalf of law enforcement, the 4th Amendment should apply to Apple's actions. This is no less than an end-run around the Constitution. @EFF
@pwnallthethings @pwnallthethings on x
@normative For the conspiracy to work, it'd need Apple, NCMEC and DOJ working together to pull it off *voluntarily* and it to never leak. If that's your threat model, OK, but that's a huge conspiracy with enormous risk to all participants, two of which existentially.
@hatr Hakan on x
„It's truly disappointing that Apple got so hung up on its particular vision of privacy that it ended up betraying the fulcrum of user control: being able to trust that your device is truly yours." https://stratechery.com/... via @fubits
@cathygellis Cathy Gellis on x
In terms of US law, 4th Amendment law about how police can't pay the landlord to let them into a tenant's apartment should apply here to limit the parade of horribles. But (a) it might not, and (b) it's of no use elsewhere in the world there's no such limit on police. https://twi…
@stevekovach Steve Kovach on x
Literally the graf I copied/pasted to my colleagues covering this https://twitter.com/...
@jaycuthrell Jay Cuthrell on x
The author appears to suggest that privacy focused Apple should use the solutions employed by [ checks notes ] Facebook and Google. 🧐🤔😳 https://twitter.com/...
@antoniogm Antonio García Martínez on x
Apple's move to scan the photos *on your phone, with code running on your phone* has ignited a storm of debate. This differs from how FB does things. As PR readers know, this is a general move toward on-device everything, and Apple's plan is in line with that. https://twitter.com…
@johnwilson John Wilson on x
The only photos that are scanned against the hashed database are those in iCloud Photos, which are destined for iCloud. So what exactly is he saying? https://twitter.com/...
@stevekovach Steve Kovach on x
This is a good one —> https://twitter.com/...
@backlon Dieter Bohn on x
I said this on the Vergecast too. At the end of the day it's inevitably policy. In this case in particular all the technical layers are designed to make you think it's technology that's protecting your privacy. It's not. It's just Apple policy. https://twitter.com/...
@gartenberg Michael Gartenberg on x
There *is* something I can do If I'm that concerned. I can choose to not do business with Apple. If enough people choose that path, Apple would be likely change their policy. That's unlikely to happen. After all, how many Apple users are ok with Facebook & Google? https://twitter…
@stratechery @stratechery on x
Apple's Mistake While it's possible to understand Apple's motivations behind its decision to enable on-device scanning, the company had a better way to satisfy its societal obligations while preserving user privacy. https://stratechery.com/...
@lindellyehuda Yehuda Lindell on x
One thing that really bothers me about the Apple CSAM solution is that it is really easily bypassed. So it will only work for the dumb criminals, but will potentially compromise everyone else. In that sense, it's like the encryption backdoor situation.
@matthew_d_green Matthew Green on x
@ncweaver @migueldeicaza @benadida @dinodaizovi @alexstamos The hash function will leak out. And that will leave “the secrecy of NCMEC's database” as the only remaining technical measure securing Apple's encryption. But I'm going to tell you a secret: the really bad guys already …
@alexmartin Alexander Martin on x
New: Apple has responded to concerns that its CSAM detection system could be used to detect things other than abuse images. Source, Page 5: https://www.apple.com/... cc @rossjanderson @sjmurdoch @pwnallthethings https://twitter.com/...
@jonathanmayer Jonathan Mayer on x
Apple's answer about government demands is difficult to reconcile with its litigation position against FBI & DOJ. Just 5 years ago, Apple swore in court filings that if it built a capability to access encrypted data, that capability would be used far beyond its original context. …
@josephfcox Joseph Cox on x
In a call today with Apple, we asked if China demanded Apple deploy its CSAM or a similar system to detect images other than CSAM (political, etc), would Apple pull out of that market? Apple speaker said that would be above their pay grade, and system not launching in China.
@josephfcox Joseph Cox on x
Also said system has things in place such as Apple having the inability to add hashes itself to the hash list, the sourcing of the hash list (NCMEC), Apple having one operating system globally and not per country.
@jasonaten Jason Aten on x
@stephenrobles @reckless @jcenters They've now made it explicit: “The system does not work for users who have iCloud Photos disabled. This feature does not work on your private iPhone photo library on the device.” https://www.apple.com/...
@stephenwarwick9 Stephen Warwick on x
@reckless They did it just doesn't fit your narrative 😂
@reckless Nilay Patel on x
@StephenWarwick9 Where does that say local hashing is disabled?
@secparam Ian Miers on x
How would Apple not be able to add things to the hash list/ change which list they use? NMEC would need to publish some root hash of their list and Apple would have to bind it into their client software in a way even they couldn't change. Thats a tall order. https://twitter.com/.…
@joshbal4 Josh on x
you know an announcement went well when you have to publish a 6 page follow-up pdf https://twitter.com/...
@stephenwarwick9 Stephen Warwick on x
@reckless “This feature does not work on your private iPhone photo library on the device”, not carrying water, just basic reading comprehension 😂
@reckless Nilay Patel on x
@StephenWarwick9 The system comprises a local component and a cloud component. Disabling the cloud component does not require the local component to stop generating hashes. You don't _know_ this. Don't carry their water for them by guessing.
@snowden Edward Snowden on x
Apple's new iPhone contraband-scanning system is now a national security issue. They just openly admitted they have no answer for what to do when China comes knocking. Hard to understate how disastrous this new system is for iPhone security. Tim Cook needs to intervene. https://t…
@antoniogm Antonio García Martínez on x
Apple claims a one in a trillion failure rate. I'm dubious. No false positive rate is that low, and they have no way of accurately modeling the real-world rate across billions of users. Their match-thresholding scheme helps, but that too is a knob with a net false-positive rate. …
@arossp Aaron Ross Powell on x
Apple makes a ton of money selling to the Chinese market, and so it's more likely than not that they'll rationalize opening their CSAM surveillance system to Xi Jinping if he threatens kicking them out of China if they don't. https://twitter.com/...
@stephenwarwick9 Stephen Warwick on x
@reckless If this needs to be explained more to the *checks notes* Editor in chief of The Verge I don't know what to tell you 😂😂😂
@stephenwarwick9 Stephen Warwick on x
@reckless “The system does not work for users who have iCloud Photos disabled.” https://www.apple.com/...
@stephenwarwick9 Stephen Warwick on x
@reckless The system = CSAM scanning including local hashing. Come on dude you're not even trying 😂 but as I said it doesn't fit your narrative so why would you
@riana_crypto Riana Pfefferkorn on x
Also if this is so privacy-protective and Apple's so proud of it, where's the letter from Tim Cook? Remember the open letter to Apple's customers he published at the outset of the Apple vs. FBI fight? https://www.apple.com/... Tim! Call me maybe!
@aral Aral Balkan on x
immediately and to issue a statement reaffirming their commitment to end-to-end encryption and to privacy as a fundamental human right. https://ar.al/... #apple #privacy #humanRights #personhood (2/2)
@reckless Nilay Patel on x
@StephenWarwick9 Who? Quote it.
@snowden Edward Snowden on x
The last time China stamped an inappropriate demand for access, @Apple sold out their users out. To quote them: “While we advocated against iCloud being subject to these laws, we were ultimately unsuccessful.” (LINK1: https://www.reuters.com/... LINK2: https://t.co/...) https://t…
@evacide Eva on x
@josephfcox So all that a government has to do is pressure/threaten/compromiseNCMEC instead of Apple? I do not feel better.
@malwarejake Jake Williams on x
Oh, is Apple deploying different versions of iOS in different countries now? Because otherwise, this statement seems a bit disingenuous... https://twitter.com/...
@malwarejake Jake Williams on x
First, I find it beyond belief that Apple lacks the technical capability to add hashes. But my bigger concern is NCMEC. They now hold the keys to one of the world's largest surveillance platforms. But I'm sure their security is top notch and they'll never be compromised, so... ht…
@ncweaver Nicholas Weaver on x
@josephfcox Someone needs to ask the same question of Microsoft with Windows Defender.
@ryu3824796630 @ryu3824796630 on x
@josephfcox Every iPhone user needs to do the uncomfortable thing and give up their iPhone and boycott Apple. I will.
@antoniogm Antonio García Martínez on x
The Apple system, complex as it is in the details, is well-designed and keeps user privacy top-of-mind. That said, the use of ‘perceptual hashing’ to do image matching raises the possibility of false positives, something with potentially dreadful consequences in CSAM policing. ht…
@riana_crypto Riana Pfefferkorn on x
AFAICT, some civil society folks got a briefing from Apple 1 day before Thursday's announcement, and that was it. They touted the 👍 they got from prominent cryptographers, so the lack of even the usual pat phrase “in consultation with stakeholders from civil society” stands out.
@riana_crypto Riana Pfefferkorn on x
It's clear that Apple didn't consult any civil society orgs. No civil liberties or human rights input. Privacy, freedom of expression, LGBTQI+ issues, orgs for homeless queer youth, none of it. If they had, they'd be touting that (even if they ignored everything the orgs said).
@nash076 @nash076 on x
The technology can be used for a wide range of scanning and cataloging beyond child abuse (including policing for copyright violations), but Apple pinky swears it would never do that. And as we all know, Apple always keeps its word. https://arstechnica.com/...
@jonathanmayer Jonathan Mayer on x
Apple's new FAQ on CSAM detection is disappointing. The document uses misleading phrasing to avoid explaining false positives. And the FAQ says little about how Apple will ensure the hashes are only CSAM and the same for all users. This is marketing. https://www.apple.com/...
@tim Tim Bradshaw on x
“Let us be clear, this technology is limited to detecting CSAM stored in iCloud and we will not accede to any government's request to expand it.” (2/2) One to keep for the record. https://www.apple.com/...
@dsilverman Dwight Silverman on x
Apple CSAM FAQ addresses misconceptions and concerns about photo scanning https://9to5mac.com/... via @benlovejoy
@tim Tim Bradshaw on x
Apple responds to “slippery-slope” privacy concerns over CSAM tool: “We have faced demands to build and deploy government-mandated changes that degrade the privacy of users before, and have steadfastly refused those demands. We will continue to refuse them in the future...” (1/2)
@normative Julian Sanchez on x
As I said last week, IF you just look at the system in isolation, assume it's implemented exactly as intended & frozen in stone, maybe it's fine. But I don't think that's a terribly smart way to think about it.
@bergmayer John Bergmayer on x
This will be true if and only if iCloud Photos are fully encrypted, and right now, they are not. Apple has policies and systems in place to secure your iCloud images but it does still have the ability to see them if it wants to, or is ordered to https://twitter.com/...
@carnage4life Dare Obasanjo on x
One can assume there are tens to hundreds of millions of CSAM images in iCloud based on extrapolation of how much storage they've announced is used by the service (8M terabytes) and reporting rates from other services (e.g. FotoForensics says 0.056%). So they must do something.
@carnage4life Dare Obasanjo on x
The question from a privacy perspective is whether Apple keeping the “we won't look at your content on the server” promise by creating a precedent where “we look at content on your phone instead” is better or worse for customer privacy?
@carnage4life Dare Obasanjo on x
Apple has also historically made a privacy promise that they can't (actually they won't since they can decrypt your data) look at your content in iCloud. They're between a rock and a hard place. So they've chosen to keep their privacy promise by scanning content on users phones.
@reckless Nilay Patel on x
This is still not Apple explicitly saying “you can completely turn off local hashing of your photos.” The “system” and “this feature” are intentionally vague descriptors. Don't connect the dots for them! Make them say it on the record. https://twitter.com/...
@carnage4life Dare Obasanjo on x
Developer of FotoForensics is the best I've read on the valid reasons for, yet problematic technical & legal issues with Apple's CSAM detection. Apple has chosen to die on the hill of not doing server side scanning and has chosen an invasive alternative https://www.hackerfactor.c…
@profwoodward Alan Woodward on x
This sounds a lot like “trust us, we could do it but we promise we won't”. https://twitter.com/...
@normative Julian Sanchez on x
Apple's put out a FAQ in response to backlash over their new CSAM photo scanning feature. Points out various ways their current design is privacy protective, which is great, but doesn't really alleviate my core concerns. https://www.apple.com/...
@howelloneill Patrick Howell O'Neill on x
Apple's response to ‘slippery slope’ concerns: “Let us be clear, this technology is limited to detecting CSAM stored in iCloud and we will not accede to any government's request to expand it.” https://www.apple.com/... https://twitter.com/...
@howelloneill Patrick Howell O'Neill on x
It's important, as we're discussing Apple's CSAM tech, to note that saying it's simple to disable iCloud backup skips over everything we know about the way people use software. This doesn't win or lose the argument any which way but it addresses one of Apple's points.
@howelloneill Patrick Howell O'Neill on x
It's hard to overstate the power of defaults. History shows that vanishingly few users mess with most of them: “For most users, the default value is the only value.” https://blog.codinghorror.com/ ...
@jasonaten Jason Aten on x
Apple also says that it won't add images hashes to the database, they have to come from NCMEC: “[hashes] are from known, existing images of CSAM that have been acquired and validated by child safety organizations. Apple does not add to the set of known CSAM image hashes.”
@jasonaten Jason Aten on x
On why Apple is doing this: “In most countries, including the United States, simply possessing these images is a crime and Apple is obligated to report any instances we learn of to the appropriate authorities.”
@jasonaten Jason Aten on x
That's the most explicit Apple has been on the record that if you turn off uploading to iCloud Photos, CSAM detection doesn't happen. Apple wants to be clear it isn't “scanning” your photo library on your device.
@jasonaten Jason Aten on x
On the chance it could be forced to expand the scope of the feature: “We have faced demands to build and deploy government-mandated changes that degrade the privacy of users before, and have steadfastly refused those demands. We will continue to refuse them in the future.”
@martinsfp Martin Sfp Bryant on x
Apple says it “will not accede to any government's request to expand” use of its device scanning tech. Easy for them to *say* that, and Apple has resisted such demands in the past, but it's understandable why many are wary of the genie leaving the bottle https://www.theverge.com/…
@aral Aral Balkan on x
Nothing in this FAQ (PDF: https://www.apple.com/...) that Apple has released addresses any of our concerns. It basically boils down to “trust us, don't worry, it'll be fine.” To reiterate what we're asking: Apple must halt deployment of its content monitoring technology (1/2)
@kaepora @kaepora on x
“Can the CSAM detection system in iCloud Photos be used to detect things other than CSAM?” Again, we get: “No, but actually yes.” 🤦‍♂️🤦 ‍♂️ The Electronic Frontier Foundation *has already documented* instances where CSAM lists were expanded to target non-CSAM content. https://twi…
@kaepora @kaepora on x
“Could governments force Apple to add non-CSAM images to the hash list?” “Apple will refuse any such demands.” — except, they won't. Apple *has already dropped plans for encrypting iCloud backups specifically because the FBI complained*: https://www.reuters.com/... https://twitte…
@kaepora @kaepora on x
Asking people to disable iCloud Photos in 2021 is not realistic, and Apple knows this. Everyone depends strongly on iCloud Photos not just for sync, but as a critical backup feature for what is often years and years of important photos.
@kaepora @kaepora on x
Apple just published a FAQ document regarding its content-scanning rollout. Some choice parts: “Does this mean Apple is going to scan all the photos stored on my iPhone?” “No, but actually yes.”🤦‍♂️ Full FAQ Here: https://www.apple.com/... https://twitter.com/...

Chronicles