An overview of Apple's three new “Child Safety” initiatives, what critics are getting wrong, and the completely legitimate slippery slope concerns from experts
*without asking.* https://twitter.com/... Kontra / @counternotions : If you must read only one article to see the absurdity of the bombastic headline regurgitation currently being spewed about Apple's new child pron alerting system: https://daringfireball.net/... Lawrence McDonald / @convertbond : How did Apple go from refusing to open locked iPhones for law enforcement in 2017, to scanning users photos for governments in 2021? https://www.google.com/... Matt Blaze / @mattblaze : It appears that the text quoted here originated from NCMEC, not Apple (but was sent on some kind of internal Apple list). In any case, how dare they? There are serious concerns being raised by serious people here, and dismissing them as a “screeching minority” is just bad faith. https://twitter.com/... Steven Aquino / @steven_aquino : I learned a helluva lot just now about Apple's upcoming child safety features by reading this brilliant @GlennF and @rmogull FAQ for TidBITS. A must-read story. (h/t @jsnell) https://tidbits.com/... John Gruber / @gruber : @BriannaWu So I *think* this is why the Messages feature gives so much control to the kids, even those 12 and under. Control meaning that they always get a clear warning upon receiving or attempting to send a flagged image that proceeding will result in a notification to their parents. Brianna Wu / @briannawu : @gruber Strongly agree, I can live with the design of iCloud as it exists - though what could happen next concerns me. It's the iMessage component that gives me far more pause. John Gruber / @gruber : @BriannaWu That said, I *think* it's very difficult to change the age of a child's iCloud account. I forget why I tried that for my son a few years ago, but I gave up. (I wanted to make him “older” for something, but don't recall the details.) Brianna Wu / @briannawu : 2/ I have a family member who enacted total lockdown on *everything* her child read. So when her child would go to the library to check out books, upon returning home she would pass them to my family member. She would then research *every single* book online and clear it. Brianna Wu / @briannawu : 3/ Any discussion of sex, certainly any discussion of homosexuality, anything promoting a secular worldview would be censored. If you don't think parents like that are going to change their child's age in iCloud to create a permanent surveillance state, you're just wrong. Brianna Wu / @briannawu : 1/ RE: Apple's plan to scan every photo in iMessage with machine learning and alert parents to nudity. I think a lot of well-meaning Silicon Valley people don't understand what life is like in the Religious Right South. Let me share so you can imagine how it will be misused. John Gruber / @gruber : @BriannaWu Good thread. It has occurred to me that while the CSAM iCloud photo detection is the feature garnering the most attention, the Messages image detection for kids has more potential to be problematic *as it exists*. Mike Masnick / @mmasnick : Worth reading. Lots of important nuance here https://twitter.com/... Rene Ritchie / @reneritchie : If you're still confused by Apple's CSAM initiative, try looking at it through this lens: Apple can't abide known CSAM images on, or trafficked through, their servers, but they don't want to scan actual private user libraries to catch them, and this system solves for that Jean-Louis Gassée / @gassee : Very useful/detailed John Gruber explanation. Also points to hasty/wrong/prejudiced WaPo article. https://twitter.com/... Brianna Wu / @briannawu : This is my point. You can't think about these systems *AT THEIR BEST.* You have to ask hard questions about how they could be misused. As an industry, we massively suck at considering the downside. In this case, the downside is outed, potentially dead children. https://twitter.com/... Dare Obasanjo / @carnage4life : @benedictevans Apple has insisted on defining privacy as “we can't see your data” then criticized Google & Facebook on privacy. However there's also “we protect your data from governments” where their track record is worse than others which is the elephant in the room for on-device scanning. Joseph Cox / @josephfcox : Re-reading all of Apple's compromises in China to cater to the government there is interesting in light of the company making a system to scan for images stored on phones https://www.nytimes.com/... https://twitter.com/... Dare Obasanjo / @carnage4life : @benedictevans @eric_seufert @TreyTitone Governments can ask them to scan for XYZ you've uploaded to iCloud today. Apple has opened the door to scanning content on your device. Even Gruber concedes this is new and now depends on how much you trust Apple to say “no” to various governments. https://daringfireball.net/... Thaddeus E. Grugq / @thegrugq : @Paxxi @pwnallthethings And yet there's only 250 cases from iPhone. Which suggests that something isn't adding up with that narrative. The CSAM database can only find known CSAM photos that have been added. So new content won't get flagged, only existing content that has to come from somewhere Thaddeus E. Grugq / @thegrugq : @Paxxi @pwnallthethings I'd like to know what those Facebook numbers represent. New CSAM? Or people uploading CSAM that gets flagged by the db? And how many users are doing this? I saw one number was 20m, but that can't be right. That's like the entire adult male population of Great Britain. Thaddeus E. Grugq / @thegrugq : @Paxxi @pwnallthethings The real number is 236 in 1yr for iPhone. Which is a bit lower than 17m a month. Pär Björklund / @paxxi : @thegrugq @pwnallthethings I meant that if Facebook detects that much, that's an indication of how prevalent it is on regular sites, no “dark web” required. Pär Björklund / @paxxi : @thegrugq @pwnallthethings These numbers suggest it's common on regular services which is easily accessible on an iPhone. https://twitter.com/... Alex Stamos / @alexstamos : @thegrugq @Paxxi @pwnallthethings The minority, but worst reports are those that involve the live abuse of a child. Those are often reported to NCMEC and also referred live to the relevant authorities by a combination of the child safety investigator (was part of my team) and law enforcement response (in legal). Antonio García Martínez / @antoniogm : @benedictevans What about the on-device nature of it makes it worse? Assuming the cloud implementation scanned the same set of photos... Alex Stamos / @alexstamos : @thegrugq @Paxxi @pwnallthethings The vast majority of Facebook NCMEC reports are hits for known CSAM using a couple of different perceptual fingerprints using both NCMEC's and FB's own hash banks. What constitutes a “report”, which can have multiple images and IPs, is one of the data challenges. Andrew Burke / @ajlburke : Here's a level-headed and detailed explanation about all the new Apple bad-image-detection features, including when one should actually start to worry, a startling revelation about existing FBI back-doors, and a good ol' take down of a sloppy journalist https://daringfireball.net/... Benedict Evans / @benedictevans : The irony of Apple's theory that any analysis done on the device is automatically private and anything analysis in the cloud violates your privacy - they built a CSAM detection system that is *potentially* much worse for your privacy *because* it happens on your device Profdeibert / @rondeibert : Correction: that memo was by MCMEC and not Apple. Still, poorly thought rollout by Apple. Rene Ritchie / @reneritchie : @charlesarthur @benedictevans My guess is that 1 is mainly Apple no longer being willing/able to tolerate CSAM on their servers and this is their engineering solution to that Charles Arthur / @charlesarthur : @reneritchie @benedictevans I agree with Gruber's take that the 2018 delay on E2E iCloud backup encryption was done in order to incorporate this. The timings all feel right. I think there's probably been CSAM but it's been hard to action (can't be seen to root through iCloud photos). Now it can. Charles Arthur / @charlesarthur : @benedictevans I doubt 2 would have much effect? Why do you think it would? Seems like 1 should have a dramatic effect of identifying offenders, if the proportion holding is anything like the FB/WApp numbers would imply. Benedict Evans / @benedictevans : What's the net effect on CSAM on iOS of both: 1: on-device scanning of images uploaded to iCloud for matches to known CSAM and 2: iOS internet traffic going through a double-blind encrypted relay, so IP addresses can't ID users (And could Apple, politically, do 2 w/out 1?) Carl Howe / @cdhowe : Unlike most of the alarmist pieces published this week, John @Gruber writes a comprehensive analysis of how Apple's new child safety initiatives actually work. Bottom line: while not perfect, it protects personal privacy a lot more than you think. https://daringfireball.net/... Fernando / @fromjrtosr : Apple saying they will “reject any such demands from a government” is laughable at best and a flat out lie at worst. Once any government makes it a law for Apple to use a different set of hashes, what then? Are they stopping business in China? In the US? In the EU? Ridiculous. https://twitter.com/... @simardcasanova : I'm a big fan of @apple and I use a ton of their products and services But on top of being creepy, we currently has zero *credible* reassurance that this system won't be weaponized by authoritarian regimes or used for unrelated purposes Maybe bc such a reassurance cannot exist https://twitter.com/... Patrick Beja / @notpatrick : Probably the best summary I've seen of Apple's recently announced child safety initiatives, including where critics are excessive or plain wrong, as well as genuine concerns for potential abuse in the future. https://daringfireball.net/... Alex Stamos / @alexstamos : Let's talk about Apple's new controversial child safety proposals! Join @Riana_Crypto, @kingjen, @matthew_d_green and me at 4pm PT / 7pm ET at this live stream. We will take questions from the live chat or tweet to us here! https://www.youtube.com/... Aral Balkan / @aral : are. I do not say this lightly or take any pleasure whatsoever in saying it. I cannot stress enough how important it is that this line does not get crossed. https://appleprivacyletter.com/ (2/2) Nadim Kobeissi / @kaepora : We are now at over 3,000 screeching voices of the minority! We tried to vet every signature. Our open letter now illustrates a very strong opposition to Apple's new content screening measures. Thanks @Snowden for keeping me and @georgionic up all night! https://appleprivacyletter.com/ Thaddeus E. Grugq / @thegrugq : @pwnallthethings Is CASM on iPhones a big problem? How do they get it there from the dark web? Don't pedos believe this capability already exists and avoid iPhones to begin with? Arnold Kim / @arnoldkim : Great overview and take on Apple CSAM stuff by @gruber - Apple's messaging/clarity on this could have been much much better, but it's not entirely a messaging issue. https://daringfireball.net/... Matt Rosoff / @mattrosoff : Amazing how many sources got this story completely abjectly wrong. The slippery slope arguments are reasonable and worth considering—but if you don't get the technology, you can't even begin to plausibly make those arguments. https://daringfireball.net/... Joseph Cox / @josephfcox : We previously obtained a sample of malware deployed in Xinjiang that scans phones for 70,000 specific files related to politics, Uighurs, etc. Why wouldn't Chinese authorities try to get Apple to leverage its new child abuse system to instead fulfil this https://www.vice.com/... Curtis Herbert / @parrots : If you're like me you might have had a hard time cutting through the noise to understand exactly what is going on. This is a very good piece breaking down exactly what Apple is doing, and what they can and cannot see. A good 101 before convos about pros and cons and slopes. https://twitter.com/... Sarah Jamie Lewis / @sarahjamielewis : Clearly a rubicon moment for privacy and end-to-end encryption. I worry if Apple faces anything other than existential annihilation for proposing continual surveillance of private messages then it won't be long before other providers feel the pressure to do the same. @migueldeicaza : The EU is actively working on legislation on this regard, so I am now convinced that this is Apple influencing the outcome so we don't end up with the equivalent of the cookie pop up for photos and messages: https://ec.europa.eu/... and https://www.consilium.europa.eu/ ... Tim Culpan / @tculpan : Apple 2021 is starting to feel like Microsoft circa 2000. Strong with overtones of bully. Use a known atrocity to push acceptance of their own tech agenda. Stand up against the agenda and you're painted as in favor of the atrocity. Back then it was anti-P2P, now pic scanning. Matt Blaze / @mattblaze : The only think preventing this scheme from scanning local photos (which Apple never previously had access to) is the policy and the integrity of the code. Both those things warrant close scrutiny. Profdeibert / @rondeibert : Apple's solution to a serious, harmful problem (child sexual exploitation) is alarming b/c of the door it opens to other surveillance, and the risks around what countries like China will do with it, once opened. As @josephfcox points out, these are not hypothetical concerns: https://twitter.com/... Sami Fathi / @samifathi_ : What isn't getting talked about is the real-world impact CSAM scanning will have. Think about the children who have been exploited and the scars they'll have for the rest of their lives. If finding pedophiles means Apple needs to do on-device processing of photos, then so be it. Julian Sanchez / @normative : If everything operates precisely as Apple intends, scanning client-side is at worst an accounting detail & at best enables greater privacy. If everything does not operate precisely as Apple intends, moving scans client-side is a dangerous Rubicon to cross. Julian Sanchez / @normative : I should add, because a couple folks have noted this: IF this were implemented only as Apple intends and for the purposes it states, then yes, this would be functionally identical to the sort of scanning that's routine on platforms & cloud storage services.... Matthew Green / @matthew_d_green : I spoke to talk radio in LA about Apple's scanning system. I was steeling myself to explain how the tech is problematic even if it might catch some bad people. I didn't need to. They asked me how to disable it. Peter Sterne🌹 / @petersterne : @SwiftOnSecurity The memo demonstrates that Apple is proud to work with NCMEC, a group that disdains privacy and security advocates. But it's a bit of a leap to say that Apple shares NCMEC's extreme views. Apple itself didn't refer to critics as a “screeching minority”. Jon Prosser / @jon_prosser : This is what the NCMEC said in a memo to Apple in response to the backlash to Apple's photo scanning 👇 Absolutely gross to frame this as “good vs. evil” and call fair criticism “screeching voices” Fuuuuuuuck that. Read the full article here: https://www.frontpagetech.com/ ... https://twitter.com/... Runa Sandvik / @runasand : The fact that we are all struggling to understand what Apple is doing on our devices and with our backups is concerning. Even more so when we don't know how this will impact our lives in the future. Julian Sanchez / @normative : And a whole bunch of the arguments Apple deployed in the San Bernardino encryption case don't obviously apply if they've already built the scan architecture & a government is just adding items to a preexisting list. Joseph Menn / @josephmenn : We have so many important fights taking place in secret courts that it is theoretically possible that Apple's surprising device-scanning system was hatched to settle some hidden legal demand. Probably not, but still. Not a good method for technology or democracies to develop.