New research has revealed that artificial intelligence systems can now craft phishing emails as effectively as human experts, raising a whole new level of concern for the cybersecurity industry.
Published to Cornell’s arXiv server, the study, prepared by a team from Harvard Kennedy School and Avant Research Group highlights the growing sophistication of large language models (LLMs). These models can create hyper-personalized, persuasive phishing campaigns at a fraction of the cost and time required by traditional methods.
Phishing is a common cyberattack where malicious actors send fraudulent emails or text messages designed to trick recipients into divulging sensitive information, such as passwords or financial details, or clicking on links that install malware.
What’s worse, they cause a lot of economic damage. In 2023, phishing scams alone cost the US economy about $2.4 billion.
For phishing attacks to work, they must exploit human psychology, leverage trust, urgency, or curiosity, and manipulate victims. Good phishing scams need to feel authentic, but as more of these attacks slide into our collective DMs, we victims are becoming more clever. This forces scammers to up their fraudulent game, and AI has taken these schemes to unprecedented sophistication.
“This significantly elevates the threat landscape,” Dr. Fred Heiding from the Harvard Kennedy School and an author of the study told The Debrief. “It is rare to see a phenomenon where technology simultaneously improves quality and reduces cost, as it does here.”
The research team conducted an experiment involving 101 participants to evaluate the effectiveness of AI-generated phishing emails compared to those crafted by human experts. Participants were divided into four groups, each receiving a different type of email.
The Control Group received generic phishing emails with minimal personalization; the Human Expert Group received semi-personalized phishing emails crafted by cybersecurity professionals; the AI-Generated Email Group received fully automated emails created by the Claude 3.5 Sonnet LLM; and the AI with Human-in-the-Loop Email Group received AI-generated emails refined by humans.
The AI-generated emails were created using a custom-built tool that automated the entire phishing process. Making matters even more frightening, the AI conducted research on its targets using Open Source Intelligence (OSINT).
“Our custom-built tool automates the spear phishing process by gathering publicly available information about targets through publicly available information like personal websites, social media pages, and news articles,” Heiding explained. “The AI automatically saturates when it deems it has found sufficient content and then uses the information to craft tailored phishing emails.”
The study found that AI-generated phishing emails performed on par with those created by human experts, achieving a 54% success rate in eliciting clicks on embedded links. When humans intervened to refine the AI-generated emails, the success rate increased slightly to 56%. By contrast, generic control group emails had a much lower success rate of 12%, highlighting the importance of personalization in phishing campaigns.
Participants said personalized content was a key factor in trusting the emails. For example, hyper-personalized messages referencing specific academic interests or professional affiliations were far more convincing than generic appeals.
This makes sense, according to Matteo Tomasini, the Chief Technology Officer with District 4 Labs, a company that specializes in building security tools that monitor the dark web.
“In the past, sending so many personalized and detailed emails would have been time-prohibitive,” Tomasini told The Debrief in an email. “The hallmarks of past phishing emails, i.e., incorrect spellings and improper grammar, are less prevalent with AI-enhanced phishing emails and as a result, people need to be considerably more vigilant and wary when it comes to their inbox.”
With the efficacy of the OSINT research and writing sitting at just above a 50% success rate, the real damage comes from the cost. Fully automated campaigns require just four cents per email and no manual labor. Moreover, traditional phishing methods are often unprofitable due to high labor costs and limited scalability. There are multiple targets, and if the attackers don’t take their time to do good work and research, there isn’t usually a big payday at the end. Those attacks end up getting deleted.
AI seems to have changed all that.
“Attackers can now launch mass-scale attacks with personalized content, increasing the scale and profitability of data and identity theft operations,” Heiding said.
Those who work in cybersecurity concur that AI just makes phishing easier.
“The use of AI tools which can be trained on compromised data from the dark web and social media content has resulted in the proliferation of incredibly convincing spear-phishing emails,” Tomasini explained.
The researchers estimated that attackers could increase their profitability by up to 50 times when targeting large groups, around 10,000 people, using AI tools.
So how can we defend ourselves?
While LLMs can be used offensively for phishing, they also show promise as defensive tools. The study tested five LLMs for their ability to detect suspicious intent in emails. Claude 3.5 Sonnet emerged as the most effective model, achieving a 97.25% detection rate for phishing emails with no false positives when primed for suspicion. However, even this high detection rate highlights a troubling asymmetry: while defensive applications are improving, offensive capabilities are advancing at an even faster pace.
Heiding and Tomasini both agree that AI will play a “dual role” by enhancing attacks and defenses.
It will be “a cycle whereby advancements in defense will inform advancements in attacks and vice-versa,” Tomasini suspects.
In short, an arms race.
“The future likely involves an ‘AI-on-AI’ dynamic, where both attackers and defenders employ AI to outmaneuver each other,” Heiding predicts. “As AI-generated phishing becomes more prevalent, developing advanced AI-driven detection and prevention mechanisms will be crucial to stay ahead in this evolving cybersecurity landscape.”
So for every email or text that shows up knocking on your digital door, awareness becomes a key priority.
Tomasini says that people need to be proactive. He suggests setting up multi-factor authentication (MFA) on all possible accounts, making sure your security software is up to date, and change your passwords regularly. Moreover, users need to be vigilant, and make sure they assess the text or email before clicking the link, or sharing personal information.
In short, it seems the days of the awkwardly worded scam messages from Russian models and royalty from foreign nations with large sums of money they hope to send you may finally be ending.
MJ Banias covers security and technology with The Debrief. You can email him at mj@thedebrief.org or follow him on Twitter @mjbanias.