light_mode

RoboMD: Uncovering Robot Vulnerabilities through Semantic Potential Fields

Som Sagar, Jiafei Duan, Sreevishakh Vasudevan, Yifan Zhou, Heni Ben Amor, Dieter Fox, Ransalu Senanayake
International Conference on Learning Representations (ICLR), 2026

RoboMD teaser figure

In one sentence. RoboMD trains a deep reinforcement learning policy to navigate a vision-language embedding space, structured as a potential field of successes and failures, to discover vulnerabilities in robot manipulation policies without real-world trials — up to 23% more unique vulnerabilities than vision-language baselines.

picture_as_pdf PDF code Code play_circle Video

Abstract

Robot manipulation policies, while central to the promise of physical AI, are highly vulnerable in the presence of external variations in the real-world. Diagnosing these vulnerabilities is hindered by two key challenges: (i) the relevant variations to test against are often unknown, and (ii) direct testing in the real world is costly and unsafe. We introduce a framework that tackles both issues by learning a separate deep reinforcement learning (deep RL) policy for vulnerability prediction through virtual runs on a continuous vision-language embedding trained with limited success-failure data. By treating this embedding space, which is rich in semantic and visual variations, as a potential field, the policy learns to move toward vulnerable regions while being repelled from success regions. This vulnerability prediction policy, trained on virtual rollouts, enables scalable and safe vulnerability analysis without expensive physical trials. By querying this policy, our framework builds a probabilistic vulnerability-likelihood map. Experiments across simulation benchmarks and a physical robot arm show that our framework uncovers up to 23% more unique vulnerabilities than state-of-the-art vision-language baselines, revealing subtle vulnerabilities overlooked by heuristic testing. Additionally, we show that fine-tuning the manipulation policy with vulnerabilities discovered by our framework improves performance with much less data.

Note. An earlier workshop version of this work appeared as “From Mystery to Mastery: Failure Diagnosis for Improving Manipulation Policies” at the RSS 2025 Workshop on Out-of-Distribution Generalization in Robotics. Both share arXiv identifier 2412.02818.

BibTeX

@inproceedings{sagar2026robomd,
  title     = {RoboMD: Uncovering Robot Vulnerabilities through Semantic Potential Fields},
  author    = {Sagar, Som and Duan, Jiafei and Vasudevan, Sreevishakh and Zhou, Yifan and Ben Amor, Heni and Fox, Dieter and Senanayake, Ransalu},
  booktitle = {International Conference on Learning Representations (ICLR)},
  year      = {2026}
}

Topics

robot manipulation, vulnerability discovery, failure diagnosis, reinforcement learning, vision-language embeddings, red teaming, robot policy evaluation

About the author. Som Sagar is a computer science PhD student at Arizona State University, advised by Ransalu Senanayake in the LENS Lab. He works on reinforcement learning for post-training and agentic LLM systems, and on failure diagnosis, red teaming, and safety evaluation of foundation models, vision-language models, and robot manipulation policies.

home Homepage library_books All publications description CV school Google Scholar