In our last blog, we explored the broader rise of AI-enabled threats across ransomware, phishing, and nation-state operations. Now, we’re narrowing in on a specific piece of that evolution: how cybercriminals are using large language models to scale and automate their tactics.
AI in cybersecurity is no longer experimental. It’s embedded in workflows, transforming everything from alert triage to vulnerability scoring. But while defenders have begun to harness its power, so have attackers and they’re scaling faster than many security teams expect.
The emergence of large language models (LLMs) designed specifically for cybercrime such as WormGPT, FraudGPT, and DarkBERT shows just how far threat actors are willing to go to scale operations. These tools are being marketed in dark web forums, used to automate phishing, generate malware, and launch socially engineered attacks that are more persuasive and harder to detect.
They don’t require skill. They don’t need infrastructure. And they’re reshaping the economics of cybercrime in real time.
Weaponized AI: What are these malicious LLMs capable of?
WormGPT, one of the most widely reported examples, is a blackhat chatbot built using the GPT-J model and trained on malware-related data. It’s been positioned as an alternative to ChatGPT, designed explicitly for criminal use such as writing flawless phishing emails, generating polymorphic malware, and constructing BEC (business email compromise) messages.
But WormGPT is only part of a larger trend. Tools like FraudGPT have been similarly created with malicious intent, while others like DarkBERT originated as legitimate academic research – in this case, to support the study of cybercrime on the dark web. As with any powerful model, however, there’s always the potential for misuse if repurposed outside of its intended scope.
-
Craft multi-language phishing content without grammatical errors
-
Maintain context through session memory for targeted follow-ups
-
Obfuscate malicious code to bypass detection
-
Assist with malware and exploit development even for low-skill users
These are not theoretical tools. They are built for scale and sold as services, reflecting a growing “cybercrime-as-a-service” model that makes advanced attacks widely accessible.
Recent research from Cato Networks shows that new WormGPT variants have emerged, built on top of commercial LLMs like xAI’s Grok and Mistral’s Mixtral. These modified agents are being promoted in cybercriminal forums, with subscription models starting around €60. They’re not standalone creations but wrappers around mainstream models, jailbroken to bypass safety controls and enable phishing, malware creation, and more.
From skill to scale: the new economics of cybercrime
At Rapid7’s Take Command Virtual Summit 2025, Vaillance Group CEO and former counterintelligence officer Shawnee Delaney shared how threat actors are already operationalizing AI in their playbooks:
“AI has transformed cybercrime from a game of skill to a game of scale. It scrapes the internet, builds personalized targeting packages, and launches phishing attacks faster than any human ever could”.
Delaney cited the example of a Ferrari executive who was nearly duped by a voice-cloned deepfake of the CEO. The impersonation was so convincing that it was only exposed when the target asked a question the model couldn’t answer. These scams are no longer just possible — they’re happening.
Laura Ellis, Rapid7’s VP of Data and AI, also pointed out during the same panel that AI has reduced the cost of phishing and social engineering by up to 95%, citing recent Harvard Business Review data. The barrier to entry has dropped, and the precision of these attacks is rising.
The latest variants of WormGPT demonstrate just how far that barrier has fallen. Cybercriminals are no longer building models from scratch, they’re modifying off-the-shelf tools, making advanced AI-driven attacks accessible to almost anyone with a budget and intent.
Beyond phishing and credential theft, LLMs are now being linked to coordinated disinformation campaigns. In recent weeks, Israeli officials reported AI-generated SMS messages and voice-cloned alerts designed to incite public panic during missile strikes, a tactic believed to be tied to Iranian-aligned actors. This blend of social engineering and psychological disruption signals a broader shift in how AI is being used to influence behavior, not just breach systems.
AI inside the enterprise: A new kind of insider risk
The external threat is clear, but there’s also risk from within. As more organizations embed AI into workflows, a new type of insider exposure is quietly growing.
Shadow AI, where employees use unapproved or unmanaged generative tools, can introduce significant risk. From pasting confidential data into public chatbots to relying on hallucinated content or outputs, even well-meaning users can cause harm.
As LLMs become common in email, writing, coding, and automation, organizations need guardrails in place to avoid missteps. Without them, the pace and scale of error can be just as dangerous as a malicious attack.
According to Gartner:
“By 2026, enterprises combining GenAI with an integrated platforms-based architecture in security behavior and culture programs will experience 40% fewer employee-driven cybersecurity incidents” Gartner, 2024
What security teams can do right now
Security leaders don’t need to boil the ocean, but they do need to understand how LLMs are shaping both attacker methods and internal risk. That means investing in both visibility and response.
At Rapid7, our approach blends AI-driven capabilities with clear governance and human oversight. Some immediate practices we recommend:
-
Red-team your AI models to uncover how they could be manipulated or leaked
-
Embed detection and response capabilities that harness AI to triage, cluster, and prioritize faster
-
Create AI usage policies that are practical, visible, and reinforced through training
-
Treat LLMs like any high-risk system, with monitoring, access controls, and containment where needed
We’ve built these principles into our platform, most recently through a major June 2025 update to Exposure Command that introduces AI Attack Coverage. This new capability is designed to detect LLM-driven attack paths and surface exposures that traditional AppSec tools often miss, such as configuration weaknesses or emerging exploit vectors tied to autonomous tools. It’s a timely addition for security teams looking to proactively close gaps before adversaries exploit them.
We’ve explored the update in more detail in this blog post, including why it matters now and how it supports a more proactive security posture.
A Broader AI Shift Is Coming – Are You Ready?
LLM misuse is one thread in a much wider narrative. As threat actors adopt AI at pace, from ransomware automation to real-time deepfakes, defenders must evolve too.
The threat is no longer theoretical. In June 2025, the NSA warned that Iranian cyber actors had gained access to U.S. critical infrastructure networks, exploiting known vulnerabilities to stage wiper attacks and reconnaissance operations. As generative tools lower the barrier to entry, nation-state groups are increasingly blending traditional tactics with AI-driven capabilities – raising the stakes for defenders everywhere.
We explored these dynamics in depth in our recent blog: Emerging Trends in AI-Related Cyberthreats in 2025: Impacts on Organizational Cybersecurity. From deepfake-enabled fraud to adaptive, AI-powered malware, the post outlines how attackers are incorporating AI across every phase of the attack chain and what organisations can do to prepare.
What’s clear is that AI isn’t just another tool. It is changing how threat actors think, behave, and attack.
Final thoughts: It’s not just about the technology
LLMs aren’t inherently good or bad. Like any powerful system, they reflect the intent of the people who use them.
The attackers of 2025 are not just writing better phishing emails. They’re weaponizing automation, scaling social engineering, and skipping the learning curve. Security teams need to respond with visibility, control, and collaboration. Because when everyone has access to the same technology, it’s the ones who use it responsibly and defensively who come out ahead.
This isn’t about outsmarting AI, it’s about using it better, building resilient security practices, and staying ahead of what’s next.