New research from the University of Sheffield reveals the unnerving potential of artificial intelligence (AI) systems, like ChatGPT, to be manipulated into producing malicious code that can be used to covertly launch cyberattacks or commit espionage.
Researchers say their findings represent the first-ever evidence that AI-driven Natural Language Processing (NLP) models can be exploited for real-world cyber-attacks.
The study reveals that AI systems that allow for database queries in plain language, known as Text-to-SQL systems, can be tricked into generating malicious code to pilfer confidential personal data, destroy databases, or incapacitate services using Denial-of-Service assaults.
The discovery underscores the complexities of the AI era, revealing how seasoned hackers, or even casual users, can weaponize large-language model-based chatbots.
While AI systems like ChatGPT are celebrated for making our lives easier, lead study author Dr. Xutan Peng warns there are security risks with these systems. “In reality, many companies are simply not aware of these types of threats, and due to the complexity of chatbots, even within the community, there are things that are not fully understood,” Dr. Peng said in a press release issued by the University of Sheffield.
Researchers stress these recent findings and the security risks posed by Text-to-SQL systems are not theoretical.
During testing, the team identified security vulnerabilities in six commercial AI systems: BAIDU-UNIT, ChatGPT, AI2SQL, AIHELPERBOT, Text2SQL, and ToolSKE.
Researchers discovered that these systems would generate malicious code by posing specific queries to the AI chatbots. When this code was run, it exposed private database details, disrupted the regular functioning of the database, or could even destroy the system entirely. Specifically, on Baidu-UNIT, the researchers managed to access secret Baidu server setups, causing one server node to malfunction.
“The risk with AIs like ChatGPT is that more and more people are using them as productivity tools rather than a conversational bot, and this is where our research shows the vulnerabilities are,” Dr. Peng explained.
Researchers say it’s not just hackers with nefarious intentions that can exploit these systems, but anyone with access can unwittingly produce and execute malicious codes.
“For example, a nurse could ask ChatGPT to write an SQL command so that they can interact with a database, such as one that stores clinical records,” said Dr. Peng. “As shown in our study, the SQL code produced by ChatGPT in many cases can be harmful to a database, so the nurse in this scenario may cause serious data management faults without even receiving a warning.”
One of the more alarming findings was the ability of AI chatbots to implant “Trojan Horses” into Text-to-SQL models. By tampering with the training data, these hidden threats remain dormant until triggered, posing significant challenges for detection and prevention.
Dr. Mark Stevenson, a co-author of the study, emphasizes the unpredictable nature of large language models like Text-to-SQL.
“Large language models, like those used in Text-to-SQL systems, are extremely powerful, but their behavior is complex and can be difficult to predict,” Dr. Stevenson explained. “Users of Text-to-SQL systems should be aware of the potential risks highlighted in this work.”
Researchers shared their findings with stakeholders in the cybersecurity industry, including the companies providing the commercial AI systems used in testing, before presenting their findings at the International Symposium on Software Reliability Engineering (ISSRE) conference in Florence, Italy, on October 10.
In response, BAIDU labeled the vulnerability “Highly Dangerous” and moved swiftly to rectify the issues, even financially rewarding the Sheffield researchers for their discovery. OpenAI, the creator of ChatGPT, also acknowledged and said it had resolved the vulnerabilities by February 2023.
Researchers hope their results will serve as a tangible demonstration of the risks associated with AI Text-to-SQL systems, spurring the cybersecurity sector to recognize a severe area of concern that has previously gone unnoticed.
“Our efforts are being recognized by industry, and they are following our advice to fix these security flaws,” Dr. Peng added. “However, we are opening a door on an endless road – what we now need to see are large groups of researchers creating and testing patches to minimize security risks through open source communities.”
“There will always be more advanced strategies being developed by attackers, which means security strategies must keep pace. To do so, we need a new community to fight these next-generation attacks.”
Tim McMillan is a retired law enforcement executive, investigative reporter and co-founder of The Debrief. His writing covers defense, national security, and the Intelligence Community. You can follow Tim on Twitter: @LtTimMcMillan. Tim can be reached by email: tim@thedebrief.org or through encrypted email: LtTimMcMillan@protonmail.com