light_mode

Failures Are Fated, But Can Be Faded: Characterizing and Mitigating Unwanted Behaviors in Large-Scale Vision and Language Models

Som Sagar, Aditya Taparia, Ransalu Senanayake
International Conference on Machine Learning (ICML), 2024 — Spotlight (top 3.5%)

Failures Are Fated teaser figure

In one sentence. A deep reinforcement learning method that explores and maps the failure landscape of large-scale vision and language models, then restructures that landscape with limited human feedback to mitigate accuracy failures, social biases, and misalignment.

picture_as_pdf PDF code Code play_circle Video

Abstract

In large deep neural networks that seem to perform surprisingly well on many tasks, we also observe a few failures related to accuracy, social biases, and alignment with human values, among others. Therefore, before deploying these models, it is crucial to characterize this failure landscape for engineers to debug and legislative bodies to audit models. Nevertheless, it is infeasible to exhaustively test for all possible combinations of factors that could lead to a model's failure. In this paper, we introduce a post-hoc method that utilizes deep reinforcement learning to explore and construct the landscape of failure modes in pre-trained discriminative and generative models. With the aid of limited human feedback, we then demonstrate how to restructure the failure landscape to be more desirable by moving away from the discovered failure modes. We empirically show the effectiveness of the proposed method across common Computer Vision, Natural Language Processing, and Vision-Language tasks.

Note. A follow-up applying this framework to text-to-image diffusion models appeared as LLM-Assisted Red Teaming of Diffusion Models at the NeurIPS 2024 Workshop on Red Teaming GenAI.

BibTeX

@inproceedings{sagar2024failures,
  title     = {Failures Are Fated, But Can Be Faded: Characterizing and Mitigating Unwanted Behaviors in Large-Scale Vision and Language Models},
  author    = {Sagar, Som and Taparia, Aditya and Senanayake, Ransalu},
  booktitle = {Proceedings of the 41st International Conference on Machine Learning (ICML)},
  pages     = {42999--43023},
  year      = {2024}
}

Topics

failure diagnosis, red teaming, model auditing, reinforcement learning, vision-language models, AI safety evaluation, social bias

About the author. Som Sagar is a computer science PhD student at Arizona State University, advised by Ransalu Senanayake in the LENS Lab. He works on reinforcement learning for post-training and agentic LLM systems, and on failure diagnosis, red teaming, and safety evaluation of foundation models, vision-language models, and robot manipulation policies.

home Homepage library_books All publications description CV school Google Scholar