Trustworthy Explanations for Robot Behaviors

Som Sagar*, Aditya Taparia*, Harsh Mankodiya, Pranav Bidare, Yifan Zhou, Ransalu Senanayake
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2025

In one sentence. BaTCAVe is a Bayesian concept-based explainability method that produces human-interpretable explanations of robot decisions together with calibrated uncertainty scores, enabling trustworthy post-hoc diagnosis of neural network policies.

picture_as_pdf PDF code Code play_circle Video

Abstract

Black box neural networks are an indispensable part of modern robots. Nevertheless, deploying such high-stakes systems in real-world scenarios poses significant challenges when the stakeholders, such as engineers and legislative bodies, lack insights into the neural networks' decision-making process. Presently, explainable AI is primarily tailored to natural language processing and computer vision, falling short in two critical aspects when applied in robots: grounding in decision-making tasks and the ability to assess trustworthiness of their explanations. In this paper, we introduce a trustworthy explainable robotics technique based on human-interpretable, high-level concepts that attribute to the decisions made by the neural network. Our proposed technique provides explanations with associated uncertainty scores for the explanation by matching neural network's activations with human-interpretable visualizations. To validate our approach, we conducted a series of experiments with various simulated and real-world robot decision-making models, demonstrating the effectiveness of the proposed approach as a post-hoc, human-friendly robot diagnostic tool.

Note. Also appeared as “Trustworthy Conceptual Explanations for Neural Networks in Robot Decision-Making” at the NeurIPS 2024 Workshop on Safe and Trustworthy Agents.

BibTeX

@inproceedings{sagar2025trustworthy,
  title     = {Trustworthy Explanations for Robot Behaviors},
  author    = {Sagar, Som and Taparia, Aditya and Mankodiya, Harsh and Bidare, Pranav and Zhou, Yifan and Senanayake, Ransalu},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  year      = {2025}
}

Topics

explainable AI, concept-based explanation, uncertainty quantification, robot decision-making, trustworthy AI, model auditing