Discussion

• @odonnell_jm James O'Donnell on x

  Users requested tools to generate images of public figures like Charli D'Amelio or Gracie Abrams, often linking to their social media so their images could be grabbed. Some wanted to generate the individual's entire body, accurately capture tattoos, or change hair color. They

• @odonnell_jm James O'Donnell on x

  News comes from a new study looking at people's requests for content on the site, called “bounties.” Most bounties asked for animated content—but a significant portion were for deepfakes of real people, and 90% of these deepfake requests targeted women.

• @odonnell_jm James O'Donnell on x

  Neither Civitai nor a16z responded to requests for comment. Excellent study led by @mdeverna2 and Shalmoli Ghosh

• @odonnell_jm James O'Donnell on x

  Grok's debacle was about deepfakes made on platforms, but this is about the demand for tools that let users fine-tune them elsewhere. 86% of deepfake requests were for instruction files called LoRAs that coach mainstream AI models into generating content they weren't trained to

• @odonnell_jm James O'Donnell on x

  New: Civitai—an online marketplace for buying and selling AI-generated content, backed by a16z—is letting users buy custom instruction files for generating celebrity deepfakes. Some were designed to make pornographic images the site says are banned. https://www.technologyreview.c…

• @osome_iu @osome_iu on x

  Latest working paper 🧪 w/ Shalmoli Ghosh and @mdeverna2 shows that AI porn and NSFW deepfakes targeting women are being commoditized A Marketplace for AI-Generated Adult Content and Deepfakes Preprint: https://doi.org/... [image]

• @mdeverna2 Matthew DeVerna on x

  Civitai runs a monetized feature called Bounties: users post paid requests—via “buzz” (worth real money)—and others compete to fulfill them. We analyze all bounty requests collected over a 14-month period following the platform's launch—especially NSFW content and deepfakes. [ima…

• @mdeverna2 Matthew DeVerna on x

  We also find that platform governance is inconsistent. Civitai displays a ‘real person likeness’ notice on 86% of SFW deepfake bounties, but that figure drops to just 58% for NSFW bounties. [image]

• @mdeverna2 Matthew DeVerna on x

  Deepfakes are a non-negligible slice of the marketplace, with hundreds of bounties requesting deepfakes of real people and some that are explicitly NSFW. Targets are overwhelmingly women (90% for SFW; 96% for NSFW) and skew towards public figures (actors/actresses/influencers). […

• @mdeverna2 Matthew DeVerna on x

  We find this marketplace is dominated by “LoRAs” (low-rank adaptations): lightweight, reusable adapters that “steer” model outputs towards a specific “look” (e.g., a person). Also, demand for NSFW content has been growing and has made up a *majority* of requests since late 2024. …

• @mdeverna2 Matthew DeVerna on x

  Participation in requesting bounties is concentrated, with the top 20% of requesters driving ~50% of all bounties. [image]

• @mdeverna2 Matthew DeVerna on x

  🚨 New working paper 🚨 What happens when you add money + competition to generative AI creation? On Civitai—a prominent platform for gen-AI content/tools w. millions of users—you get a growing marketplace for NSFW requests and a nontrivial stream of deepfakes. [image]