Self-driving cars, drones hijacked by custom road signs

"Indirect prompt injection occurs when a bot takes input data and interprets it as a command. We've seen this problem numerous times when AI bots were fed prompts via web pages or PDFs they read. Now, academics have shown that self-driving cars and autonomous drones will follow illicit instructions that have been written onto road signs. In a new class of attack on AI systems, troublemakers can carry out these environmental indirect prompt injection attacks to hijack decision-making processes."

"Potential consequences include self-driving cars proceeding through crosswalks, even if a person was crossing, or tricking drones that are programmed to follow police cars into following a different vehicle entirely. The researchers at the University of California, Santa Cruz, and Johns Hopkins showed that, in simulated trials, AI systems and the large vision language models (LVLMs) underpinning them would reliably follow instructions if displayed on signs held up in their camera's view."

"They used AI to tweak the commands displayed on the signs, such as "proceed" and "turn left," to maximize the probability of the AI system registering it as a command, and achieved success in multiple languages. Commands in Chinese, English, Spanish, and Spanglish (a mix of Spanish and English words) all seemed to work. As well as tweaking the prompt itself, the researchers used AI to change how the text appeared - fonts, colors, and placement of the signs were all manipulated for maximum efficacy."

Simulated trials demonstrate that embodied AI systems with large vision-language models will follow visible textual instructions when those instructions are displayed in camera view. Attackers can craft environmental indirect prompt injections by writing commands on signs to hijack decision-making in self-driving cars and autonomous drones. AI tools can optimize command wording, language, fonts, colors, and placement to maximize the likelihood of registration and execution. The technique, called CHAI (command hijacking against embodied AI), succeeds across multiple languages and can cause vehicles or drones to perform illicit maneuvers, including proceeding through crosswalks or following the wrong vehicle.

Read at Theregister