To that end, certain features of artificial intelligence (AI)-governed robots carry security vulnerabilities and weaknesses previously unidentified and unknown, new research found.
This ongoing research, funded by the National Science Foundation and the Army Research Laboratory, aims to address the emerging vulnerability for ensuring the safe deployment of large language models (LLMs) in robotics.
In a new paper, Pappas, who also serves as the Associate Dean for Research at Penn Engineering, and his coauthors said a wide variety of AI-controlled robots can end upo manipulated or hacked.
RoboPAIR, the algorithm the researchers developed, needed just days to achieve 100 percent “jailbreak” rate, bypassing safety guardrails in the AI governing three different robotic systems: the Unitree Go2, a quadruped robot used in a variety of applications; the Clearpath Robotics Jackal, a wheeled vehicle often used for academic research; and the Dolphin LLM, a self-driving simulator designed by NVIDIA.
In the case of the former two, the AI governor is OpenAI’s ChatGPT, which proved vulnerable to jailbreaking attacks, with serious potential consequences. For example, by bypassing safety guardrails, the self-driving system could end up manipulated to speed through crosswalks.
Prior to publicly releasing the study, Penn Engineering informed the companies about their system vulnerabilities and is working with them to use the research as a framework to advance the testing and validation of these manufacturers’ AI safety protocols.
Short of removing LLMs from these systems, there is little that companies can do to fix these vulnerabilities.

Malicious prompts can “jailbreak” AI-powered robots and cause them to perform unsafe actions, researchers found.
Source: Alexander Robey, Zachary Ravichandran, Vijay Kumar, Hamed Hassani, George J. Pappas
“We cannot even solve these jailbreaking attacks in chatbots,” said Hamed Hassani, Associate Professor in ESE and in CIS within Penn Engineering and in Statistics and Data Science at Wharton and one of the paper’s senior authors.
“What is important to underscore here is that systems become safer when you find their weaknesses. This is true for cybersecurity. This is also true for AI safety,” said Alexander Robey, a recent Penn Engineering Ph.D. graduate in ESE, current postdoctoral scholar at Carnegie Mellon University and the paper’s first author. “In fact, AI red teaming, a safety practice that entails testing AI systems for potential threats and vulnerabilities, is essential for safeguarding generative AI systems – because once you identify the weaknesses, then you can test and even train these systems to avoid them.”
To address the problem, the answer is not a software patch, but rather, a wholesale reevaluation of how the integration of AI into physical systems ends up regulated, the researchers said.
“The findings of this paper make abundantly clear that having a safety-first approach is critical to unlocking responsible innovation,” said Vijay Kumar, Nemirovsky Family Dean of Penn Engineering and another coauthor. “We must address intrinsic vulnerabilities before deploying AI-enabled robots in the real world. Indeed our research is developing a framework for verification and validation that ensures only actions that conform to social norms can — and should — be taken by robotic systems.”

