[Paper Review] fAIlureNotes: Supporting Designers in Understanding the Limits of AI Models for Computer Vision Tasks

Steven A. Moore, Q. Vera Liao+1 more|arXiv (Cornell University)|Feb 22, 2023

Ethics and Social Impacts of AI4 citations

TL;DR

fAIlureNotes is a designer-centered tool that enables UX practitioners to proactively explore and analyze failures in pre-trained computer vision models by integrating user research, scenario-based testing, and AI failure mode categorization. It outperforms interactive model cards in helping designers identify context-specific failure patterns and derive actionable UX design considerations for handling model errors.

ABSTRACT

To design with AI models, user experience (UX) designers must assess the fit between the model and user needs. Based on user research, they need to contextualize the model's behavior and potential failures within their product-specific data instances and user scenarios. However, our formative interviews with ten UX professionals revealed that such a proactive discovery of model limitations is challenging and time-intensive. Furthermore, designers often lack technical knowledge of AI and accessible exploration tools, which challenges their understanding of model capabilities and limitations. In this work, we introduced a failure-driven design approach to AI, a workflow that encourages designers to explore model behavior and failure patterns early in the design process. The implementation of fAIlureNotes, a designer-centered failure exploration and analysis tool, supports designers in evaluating models and identifying failures across diverse user groups and scenarios. Our evaluation with UX practitioners shows that fAIlureNotes outperforms today's interactive model cards in assessing context-specific model performance.

Motivation & Objective

• To address the challenge UX designers face in understanding AI model limitations due to lack of technical knowledge and accessible tools.
• To support a proactive, failure-driven design process that integrates user research with AI model evaluation.
• To enable designers to identify and contextualize model failures across diverse user groups and scenarios.
• To improve the usability of pre-trained AI models in real-world applications by surfacing context-specific failure patterns.
• To bridge the UX-AI gap by providing a shared framework and tooling for designers and engineers to collaboratively assess model reliability.

Proposed method

• fAIlureNotes integrates user personas and real-world scenarios into a guided workflow for testing pre-trained computer vision models.
• Designers annotate expected model behavior for specific input images, then compare it with actual model outputs to detect mismatches.
• The tool classifies detected mismatches into a taxonomy of 15 failure modes, such as misclassification, false positives, and sensitivity to context.
• It uses LLM-generated inputs and synthetic data to expand failure coverage across diverse user groups and edge cases.
• The system supports iterative exploration, allowing designers to refine scenarios and failure patterns over time.
• fAIlureNotes is evaluated through a study with 10 UX practitioners, comparing its effectiveness against Hugging Face’s interactive model cards.

Figure 1. Designerly understanding of AI model failures using fAIlureNotes: (a) The designer selects a user scenario based on their user research and (b) annotates the expected behavior of the AI model for an input image. Running the model for the image, (c) they compare differences in expected and

Experimental results

Research questions

• RQ1 How can UX designers effectively explore and understand the limitations of pre-trained computer vision models in context-specific user scenarios?
• RQ2 What failure patterns emerge when designers test models across diverse user groups and real-world use cases?
• RQ3 How does fAIlureNotes support the derivation of actionable UX design considerations for handling AI failures?
• RQ4 In what ways does fAIlureNotes outperform existing tools like interactive model cards in assessing contextual model performance?
• RQ5 How can a failure-driven design workflow improve collaboration between UX designers and AI engineers?

Key findings

• fAIlureNotes enabled designers to identify aggregated failure patterns across user groups and scenarios, which were not detectable using Hugging Face’s interactive model cards.
• Participants reported that fAIlureNotes helped them operationalize Human-Centered AI (HAI) guidelines by making model limitations tangible and actionable.
• The tool supported the discovery of 15 distinct failure modes in computer vision models, including context-dependent misclassifications and sensitivity to lighting or language.
• Designers were able to derive concrete design considerations for failure recovery, such as user warnings, fallback mechanisms, and human-in-the-loop handoff strategies.
• The evaluation showed that designers using fAIlureNotes were significantly more effective in assessing model fit for specific use cases than those using standard model cards.
• Participants emphasized the value of scenario-based, iterative failure exploration in building a deeper, designerly understanding of AI behavior.

Figure 2. Proactive and reactive approaches to designerly understanding of model limitations that participants discussed in the formative interview