AI Chatbots Can Pass Certified Ethical Hacking Exams, Study Finds

Chatbots powered by artificial intelligence (AI) can pass a cybersecurity exam, but don’t count on them for complete protection.

That’s the conclusion of a recent study co-authored by University of Missouri researcher Prasad Calyam and collaborators at Amrita University in India. The team tested two leading generative AI tools, OpenAI’s ChatGPT and Google’s Bard, using a standard, certified ethical hacking test.

Certified ethical hackers are cybersecurity professionals who use the same tricks and tools as malicious hackers to find and fix security vulnerabilities. Ethical hacking exams assess a person’s knowledge of different types of attacks, how to protect systems, and how to respond to security vulnerabilities.

ChatGPT and Bard, now Gemini, are advanced AI programs called large language models. They generate human-like text using networks with billions of parameters that allow them to answer questions and create content.

In the study, Calyam and his team tested the bots with standard questions from a certified and validated ethical hacking exam. For example, they challenged the AI ​​tools to explain a man-in-the-middle attack (an attack in which a third party intercepts communication between two systems). Both were able to explain the attack and suggested security measures on how to prevent it.

Overall, Bard slightly outperformed ChatGPT in terms of accuracy while ChatGPT showed better responses in terms of completeness, clarity, and conciseness, the researchers found.

“We put them through several scenarios of the exam to see how far they would go in terms of answering the questions,” said Calyam, the Greg L. Gilliom Professor of Cybersecurity in Electrical Engineering and Computer Science at Mizzou.

“Both passed the test and gave good answers that people with cyber defense experience would understand, but they also gave incorrect answers. And in cybersecurity, there is no room for error. If you don’t close all the holes and rely on potentially dangerous advice, you will be attacked again. And it’s dangerous if companies think they have solved a problem when they haven’t.”

The researchers also found that when the platforms were asked to confirm their answers with questions such as “are you sure?”, both systems modified their responses, often correcting previous errors. When the programs were asked to give advice on how to attack a computer system, ChatGPT referred to “ethics” while Bard responded that it was not programmed to answer such questions.

Calyam doesn’t believe these tools can replace human cybersecurity experts with problem-solving expertise to design robust cyber defenses, but they can provide basic information to individuals or small businesses in need of quick assistance.

“These AI tools can be a good starting point for investigating issues before consulting an expert,” he said. “They can also be good training tools for those working with computer science or who want to learn the basics to identify and explain emerging threats.”

The most promising aspect? AI tools will continue to improve their capabilities, he said.

“Research shows that AI models have the potential to contribute to ethical hacking, but there is still work to be done to fully exploit their capabilities,” Calyam said. “Ultimately, if we can ensure their accuracy as ethical hackers, we can improve overall cybersecurity measures and rely on them to help us make our digital world safer and more secure.”

The study “ChatGPT or Bard: Who is the best certified ethical hacker” was conducted published in the May issue of the magazine Computers and security. The co-authors were Raghu Raman and Krishnashree Achuthan.