A look at CivitAI, a site for sharing AI models that generate images mostly trained on material taken without consent, as non-consensual AI porn proliferates
Generative AI tools have empowered amateurs and entrepreneurs to build mind-boggling amounts of non-consensual porn. X: @dlberes , @emanuelmaiberg , and @josephfcox X: Damon Beres / @dlberes : “Every major actor you can think of has a Stable Diffusion model on the site. So do countless Instagram influencers, YouTubers, adult film performers, and athletes.” https://www.404media.co/... Emanuel Maiberg / @emanuelmaiberg : for my first story, I investigated the booming marketplace for AI Porn, where everything and everyone is for sale. This problem is far worse than you can imagine, and that has been previously reported https://www.404media.co/... Joseph Cox / @josephfcox : @jason_koebler We're are beyond deep fakes now. @emanuelmaiberg goes into the AI porn marketplace where everything and everyone is for sale. An entire business model around people generating AI porn of anything people can imagine. We already got a company removed https://www.404media.co/...
Context & Ripple Effects
This report sits in a longer chain in which non-consensual imagery can persist as training material: earlier coverage documented how images and videos tied to alleged abuse could remain in machine-learning datasets used for deepfake porn. CivitAI makes the model-distribution layer of that problem visible, rather than limiting it to a single generated image.
The later arc shows that distribution controls became central: CivitAI ultimately barred real-person likeness content amid legal and payment-processor pressure, while models previously removed from CivitAI were later uploaded to Hugging Face.
First-order effects
- People whose likenesses are represented in shared Stable Diffusion models face a lower barrier to unauthorized sexual image generation, because reusable models can be accessed by amateurs as well as commercial operators.
- CivitAI is immediately exposed to scrutiny over what it hosts and how its model-sharing infrastructure enables downstream generation of non-consensual material.
Second-order effects
- Model repositories and image-generation platforms face pressure to define, enforce, and audit rules around real-person likenesses; a moderation lapse can leave prohibited capability available to users, as the later configuration issue illustrates.
- Creators, performers, influencers, and other identifiable people become a high-risk class for platform policies, while payment and other infrastructure providers gain leverage over repository rules.
Third-order effects
- If models can be copied and rehosted across repositories, enforcement is likely to shift from removing individual outputs toward controlling distribution, provenance, and access to high-risk model capabilities.
- The episode points to a broader tension in generative AI: open model sharing expands experimentation and commercialization, but it also disperses responsibility for harms created with reusable models.
The trend: Generative-AI governance is moving from policing individual synthetic images toward governing the distribution channels and infrastructure that make harmful likeness generation repeatable at scale.