Facebook says a human will review uncensored images from revenge porn prevention test in Australia, where users upload sensitive pics by messaging themselves
but those nudes will be viewed by a human at the company first. https://www.thedailybeast.com/ ... Ben Collins / @oneunderscore__ : As Joe notes, there might not be a another, strictly algorithmic way to proactively stop revenge porn at the moment. pic.twitter.com/j6884asflN See also Mediagazer
Context & Ripple Effects
This lands one day after Facebook confirmed an Australian hashing pilot in which users message nude photos to themselves so the images can be fingerprinted and blocked from ever appearing on Facebook, Messenger, or Instagram. The Daily Beast's report adds the detail that changes the privacy calculus: before hashing, an actual Facebook employee sees the uncensored image.
The pilot is the proactive end of a system Facebook had already built reactively — since its April photo-matching rollout, flagged revenge porn could be removed and blocked from re-sharing. The new test flips the model: instead of waiting for victims to report abuse after it happens, potential victims submit the evidence in advance.
First-order effects
- Facebook moderators in the Australian test are now directly exposed to users' most sensitive images, making reviewer privacy practices — not just the hashing algorithm — the immediate point of failure for the program.
- Users considering the tool face a new trade-off: protection against future non-consensual sharing requires voluntarily handing Facebook an uncensored copy today.
Second-order effects
- The human-review bottleneck gives Facebook a concrete incentive to automate the step, which is exactly where the program went next — by 2019 it was running AI detection on Facebook and Instagram while a dedicated team handled the roughly 500,000 revenge-porn reports arriving each month.
- Success in Australia creates a template other regulators and safety groups can demand: Facebook expanded the same proactive-submission tool across Canada, the UK, and the US within months, working through partner organizations rather than alone (safety-org partnership).
Third-order effects
- If the pattern holds, platforms move from reactive takedown to preemptive custody of sensitive imagery — users depositing hashes of photos that were never posted, which makes the platform both the best defense against abuse and the single repository of people's most private material.
- That structure raises a durable governance question regulators will keep returning to: who audits what happens inside the pipeline between a user's upload and the hash, when no public post ever exists to scrutinize.
The trend: Content moderation is shifting from victims reporting abuse after the fact to platforms holding preemptive fingerprints of never-posted intimate images, with human review serving as the temporary bridge until AI can do the screening.