Researchers detail how AI tools for generating deepfakes proliferated on Civitai before it banned them in 2025; many tools submitted before the ban remain live
Civitai—an online marketplace for buying and selling AI-generated content, backed by the venture capital firm Andreessen Horowitz …
Civitai’s ban leaves a legacy inventory problem: previously submitted deepfake-generation tools can remain available despite the platform’s current policy.
Users seeking such capabilities retain access through surviving listings, while Civitai faces a more demanding task of auditing and enforcing restrictions across its existing catalog.
Second-order effects
Payment, hosting, and other platform partners may place greater weight on demonstrable removal and monitoring systems rather than relying on announced content rules.
Competing AI-model marketplaces may be pushed to review whether bans apply retroactively, since grandfathered tools can undermine a platform-wide safety commitment.
Third-order effects
The case points to dual-use AI governance shifting from publication-time rules toward lifecycle controls: catalog review, version tracking, and removal processes will matter as much as a ban’s wording.
If enforcement gaps persist, marketplaces may increasingly be judged by their capacity to govern distributed model inventories rather than by their stated policies alone.
The trend: AI model marketplaces are confronting the operational reality that governing harmful uses requires continuous control of legacy tools, not one-time policy changes.
Users requested tools to generate images of public figures like Charli D'Amelio or Gracie Abrams, often linking to their social media so their images could be grabbed. Some wanted to generate the individual's entire body, accurately capture tattoos, or change hair color. They
News comes from a new study looking at people's requests for content on the site, called “bounties.” Most bounties asked for animated content—but a significant portion were for deepfakes of real people, and 90% of these deepfake requests targeted women.
Grok's debacle was about deepfakes made on platforms, but this is about the demand for tools that let users fine-tune them elsewhere. 86% of deepfake requests were for instruction files called LoRAs that coach mainstream AI models into generating content they weren't trained to
New: Civitai—an online marketplace for buying and selling AI-generated content, backed by a16z—is letting users buy custom instruction files for generating celebrity deepfakes. Some were designed to make pornographic images the site says are banned. https://www.technologyreview.c…
Latest working paper 🧪 w/ Shalmoli Ghosh and @mdeverna2 shows that AI porn and NSFW deepfakes targeting women are being commoditized A Marketplace for AI-Generated Adult Content and Deepfakes Preprint: https://doi.org/... [image]
Civitai runs a monetized feature called Bounties: users post paid requests—via “buzz” (worth real money)—and others compete to fulfill them. We analyze all bounty requests collected over a 14-month period following the platform's launch—especially NSFW content and deepfakes. [ima…
We also find that platform governance is inconsistent. Civitai displays a ‘real person likeness’ notice on 86% of SFW deepfake bounties, but that figure drops to just 58% for NSFW bounties. [image]
Deepfakes are a non-negligible slice of the marketplace, with hundreds of bounties requesting deepfakes of real people and some that are explicitly NSFW. Targets are overwhelmingly women (90% for SFW; 96% for NSFW) and skew towards public figures (actors/actresses/influencers). […
We find this marketplace is dominated by “LoRAs” (low-rank adaptations): lightweight, reusable adapters that “steer” model outputs towards a specific “look” (e.g., a person). Also, demand for NSFW content has been growing and has made up a *majority* of requests since late 2024. …
🚨 New working paper 🚨 What happens when you add money + competition to generative AI creation? On Civitai—a prominent platform for gen-AI content/tools w. millions of users—you get a growing marketplace for NSFW requests and a nontrivial stream of deepfakes. [image]