A look at the nascent scene of generative AI chatbots trained for phishing and malware attacks, including WormGPT, FraudGPT, DarkBART, and DarkBERT
Context & Ripple Effects
The story of criminal generative AI has moved in stages: researchers first flagged cybercriminals using ChatGPT itself to write malware and scam scripts in early 2023, and Europol followed within months with a formal warning that LLMs were already being exploited for fraud and cybercrime (Europol's assessment).
What this roundup documents is the next stage: purpose-built underground chatbots like WormGPT and FraudGPT that skip the guardrails entirely, joining the manipulated versions of Meta, OpenAI, and Anthropic models later found on sale in dark web markets (hacker-oriented chatbots sold on the dark web). The endpoint of that arc is visible in reports that US models have turbocharged Iran's state cyber operations (Iran's scaled use of ChatGPT and Gemini) — the same tooling graduating from hobbyist fraud to nation-state throughput.
First-order effects
- Low-skill criminals get subscription access to phishing-literate and malware-generating chatbots, collapsing the expertise barrier that previously separated script kiddies from capable attackers.
- OpenAI, Meta, and Anthropic face direct brand and liability exposure as their models are cloned or manipulated into hacker tools, forcing abuse teams to police an underground they don't control.
Second-order effects
- Platform defenders respond in kind: Meta was already blocking thousands of generative-AI-themed lure links, and security vendors now race to build LLM-based detection against LLM-generated attacks.
- The underground splits the market — commercial chatbots harden their refusals while dedicated criminal models monetize the gap, creating a paid tier of AI-assisted crime that consumer-model safety training can't reach.
Third-order effects
- If the pattern holds, cybercrime consolidates around AI-native service models — phishing and malware generation as metered products — shifting the industry's economics from skill-based to access-based.
- Regulators and model providers face structural pressure to treat model weights and fine-tuning pipelines as attack surface, since every aligned frontier model spawns an unaligned clone within the ecosystem.
The trend: Criminal use of generative AI is maturing from opportunistic misuse of consumer chatbots toward purpose-built underground models and, ultimately, state-scale offensive operations.