In addition to scanning for known child abuse photos, Apple will also begin using on-device ML to warn parents and kids of sexually explicit photos in Messages
Apple later this year will roll out new tools that will warn children and parents if the child sends or receives sexually explicit photos through the Messages app.
TechCrunchSarah Perez
Context & Ripple Effects
Apple’s Messages warnings arrived alongside a separate plan for on-device scanning for known child-abuse images, a pairing that prompted Apple to acknowledge internal concern about its broader photo-safety approach. The Messages feature is narrower in function: it intervenes around sexually explicit images exchanged by children rather than identifying known abuse material.
Later coverage shows Apple adapting the model by market and audience: a UK opt-in version without parental notification and an on-device Sensitive Content Warning for adults extended the same safety-tool framework beyond the original child-account setting.
First-order effects
Children using Messages and their parents receive warnings when sexually explicit photos are sent or received, making Apple’s safety intervention part of the messaging flow.
Apple must operate two distinct on-device child-safety mechanisms at once: Messages nudity warnings and the separately announced known-image scanning system.
Second-order effects
Privacy criticism of the broader photo-scanning plan raises the stakes for Apple to distinguish image-warning features from systems that identify known child-abuse material.
The removal of parent notifications in Apple’s later UK implementation shows that parental involvement becomes a product-design and deployment constraint, not merely a technical setting.
Third-order effects
Apple’s subsequent adult warning and Australian child-reporting feature point to a modular safety stack: local processing, account-age safeguards and reporting can be combined differently by user group and market.
If that pattern persists, content safety on consumer devices will be governed less by one universal policy than by region-specific choices over consent, parental notification and escalation paths.
The trend: Apple is evolving on-device content safety from a single child-protection proposal into configurable safeguards for different users and jurisdictions.
New: much more interesting than Apple checking iCloud for known child abuse images is Apple will scan all images sent and received by children in Messages app to detect nudity. This is for new images, not already known. Expert concerned could be expanded https://www.vice.com/...
This is a good example of where the US & EU privacy philosophies diverge. US media argues big tech should police private communications in online groups which is a straight line to including iMessage. EU considers private messaging to be sacred by law. https://www.apple.com/...
I'm with @CaseyNewton and have very mixed feelings about Apple's #ChildSafety initiative. However, these computing devices open up children to enormous forms of novel abuse that can cause lifelong trauma and suffering. Taking some time with this one. https://www.apple.com/... htt…
Reminder that the Indian government has successfully bullied Apple in the past to make iOS changes (TRAI). Could they pressure Apple into adding new hashes (of political memes) in the next iOS release? I'd rather not find out. https://twitter.com/...
NEW: @Apple to start checking photos on iPhone & iCloud against a blacklist of CSAM imagery. To watch out for: Now that the tech is on the table, how soon till China & other authoritarians w/political blacklists lean on Apple to search citizens' phones? https://www.apple.com/... …
my expectation is just like everything else this will start with child porn and then get expanded to terrorism and then subpoenas will start going out and then edgy politics content etc etc usual course of affairs https://twitter.com/...
So, Apple makes its entire iOS ad campaign about “privacy”, but then starts scanning your photos, messages, and searches? Not only does this normalize this mass surveillance, but it's only a matter of time before the algorithm makes a huge mistake. https://www.apple.com/...
if i'm reading this right, it seems you can opt out of this scanning by disabling icloud backup for photos, even though all the CSAM matching happens locally https://twitter.com/...
As a parent, this might make a lot of sense. On the other hand, Apple's about to drive a whole generation of teenagers & peer groups away from iMessage, to never come back. This feels like a radioactive feature, opt-in or not https://twitter.com/... https://twitter.com/...
@slava_pestov @hadeon_eon I don't have data myself; regardless of that number, Stamos echoed my observation: Apple is likely under pressure to weaken encryption across the board with this as a core issue. Possibly the math was “better do this than be forced to weaken everything”
Been reading a lot about Apple's upcoming Child Safety protections coming to iOS 15. It's still private and Apple is not looking at all of your images physically and everything is under multiple layers of encryption. https://www.apple.com/...