Researchers detail how AI tools for generating deepfakes spread on Civitai before it banned deepfakes in 2025; many tools submitted before the ban remain live
Civitai—an online marketplace for buying and selling AI-generated content, backed by the venture capital firm Andreessen Horowitz …
The new research turns the question from whether Civitai announced a ban to whether it removed the pre-ban tools already circulating on the marketplace. That distinction matters because its earlier real-person-likeness policy shift was tied to legal pressure and payment-processing access.
First-order effects
Tools submitted before the ban remain accessible, so Civitai’s stated restriction does not yet fully remove the capability researchers identified from its platform.
Civitai faces an immediate enforcement and catalog-review problem: legacy listings require treatment alongside newly submitted content, rather than relying solely on an upload-date policy.
Second-order effects
The finding increases compliance pressure on Civitai from payment and platform partners, since a formal policy is less protective if prohibited tools remain distributed.
Other AI-model marketplaces are pushed to audit legacy inventories as well as new uploads; moderation workflows centered only on future submissions leave the same exposure.
Third-order effects
Marketplace governance for dual-use generative models is likely to be judged by ongoing discoverability and removal of capabilities, not by announced content rules alone.
If enforcement gaps persist, distribution platforms may face stronger incentives to build provenance, catalog auditing, and repeat-review systems for high-risk models—though the eventual mix of platform and regulatory requirements remains uncertain.
The trend: This is part of a broader shift from permissive AI-model distribution toward lifecycle governance of dual-use tools after they have entered platform catalogs.
Users requested tools to generate images of public figures like Charli D'Amelio or Gracie Abrams, often linking to their social media so their images could be grabbed. Some wanted to generate the individual's entire body, accurately capture tattoos, or change hair color. They
News comes from a new study looking at people's requests for content on the site, called “bounties.” Most bounties asked for animated content—but a significant portion were for deepfakes of real people, and 90% of these deepfake requests targeted women.
Grok's debacle was about deepfakes made on platforms, but this is about the demand for tools that let users fine-tune them elsewhere. 86% of deepfake requests were for instruction files called LoRAs that coach mainstream AI models into generating content they weren't trained to
New: Civitai—an online marketplace for buying and selling AI-generated content, backed by a16z—is letting users buy custom instruction files for generating celebrity deepfakes. Some were designed to make pornographic images the site says are banned. https://www.technologyreview.c…
Latest working paper 🧪 w/ Shalmoli Ghosh and @mdeverna2 shows that AI porn and NSFW deepfakes targeting women are being commoditized A Marketplace for AI-Generated Adult Content and Deepfakes Preprint: https://doi.org/... [image]
Civitai runs a monetized feature called Bounties: users post paid requests—via “buzz” (worth real money)—and others compete to fulfill them. We analyze all bounty requests collected over a 14-month period following the platform's launch—especially NSFW content and deepfakes. [ima…
We also find that platform governance is inconsistent. Civitai displays a ‘real person likeness’ notice on 86% of SFW deepfake bounties, but that figure drops to just 58% for NSFW bounties. [image]
Deepfakes are a non-negligible slice of the marketplace, with hundreds of bounties requesting deepfakes of real people and some that are explicitly NSFW. Targets are overwhelmingly women (90% for SFW; 96% for NSFW) and skew towards public figures (actors/actresses/influencers). […
We find this marketplace is dominated by “LoRAs” (low-rank adaptations): lightweight, reusable adapters that “steer” model outputs towards a specific “look” (e.g., a person). Also, demand for NSFW content has been growing and has made up a *majority* of requests since late 2024. …
🚨 New working paper 🚨 What happens when you add money + competition to generative AI creation? On Civitai—a prominent platform for gen-AI content/tools w. millions of users—you get a growing marketplace for NSFW requests and a nontrivial stream of deepfakes. [image]