[Paper Review] Discovering Blind Spots of Predictive Models: Representations and Policies for Guided Exploration.

Himabindu Lakkaraju, Ece Kamar+2 more|arXiv (Cornell University)|Oct 28, 2016

Machine Learning and Data Classification20 references7 citations

TL;DR

This paper proposes a model-agnostic framework to discover unknown unknowns—high-confidence incorrect predictions—in predictive models under a fixed oracle query budget. It uses a two-phase approach: clustering instances by similarity and applying an explore-exploit strategy across partitions to efficiently identify failures, demonstrating the first algorithmic solution to this problem.

ABSTRACT

Predictive models deployed in the world may assign incorrect labels to instances with high confidence. Such errors or unknown unknowns are rooted in model incompleteness, and typically arise because of the mismatch between training data and the cases seen in the open world. As the models are blind to such errors, input from an oracle is needed to identify these failures. In this paper, we formulate and address the problem of optimizing the discovery of unknown unknowns of any predictive model under a fixed budget, which limits the number of times an oracle can be queried for true labels. We propose a model-agnostic methodology which uses feedback from an oracle to both identify unknown unknowns and to intelligently guide the discovery. We employ a two-phase approach which first organizes the data into multiple partitions based on instance similarity, and then utilizes an explore-exploit strategy for discovering unknown unknowns across these partitions. We demonstrate the efficacy of our framework by varying the underlying causes of unknown unknowns across various applications. To the best of our knowledge, this paper presents the first algorithmic approach to the problem of discovering unknown unknowns of predictive models.

Motivation & Objective

• Address the challenge of identifying high-confidence incorrect predictions (unknown unknowns) in deployed predictive models due to distributional shifts between training and real-world data.
• Formulate the discovery of unknown unknowns as an optimization problem under a fixed budget of oracle queries.
• Develop a methodology that leverages oracle feedback not only to detect failures but also to guide future exploration intelligently.
• Enable systematic and efficient discovery of model failures across diverse real-world applications with varying root causes of unknown unknowns.

Proposed method

• Partition the input space into multiple clusters based on instance similarity to group semantically or structurally related data points.
• Apply an explore-exploit strategy across these clusters to prioritize querying instances most likely to reveal unknown unknowns.
• Use feedback from the oracle to refine the model's understanding of failure regions and update exploration priorities dynamically.
• Maintain a balance between exploring new, under-sampled clusters and exploiting known clusters with high failure potential.
• Design the framework to be model-agnostic, enabling application to any predictive model regardless of architecture or training process.
• Integrate uncertainty estimation and failure probability modeling to guide query selection within each cluster.

Experimental results

Research questions

• RQ1 How can unknown unknowns in predictive models be systematically discovered under a limited budget of oracle queries?
• RQ2 To what extent can clustering-based partitioning improve the efficiency of unknown unknown detection compared to random or uniform sampling?
• RQ3 How does an explore-exploit strategy across clusters enhance the discovery of model failures compared to naive exploration?
• RQ4 Can the proposed framework generalize across diverse causes of unknown unknowns, such as distributional shift, concept drift, or rare patterns?
• RQ5 What is the impact of feedback-driven adaptation on the rate and accuracy of unknown unknown discovery?

Key findings

• The proposed framework achieves significant improvements in unknown unknown detection efficiency compared to baseline random and uniform sampling strategies.
• Clustering-based partitioning enables more effective targeting of failure-prone regions by grouping similar instances, reducing redundant queries.
• The explore-exploit strategy leads to faster convergence on failure patterns by dynamically prioritizing high-potential clusters.
• The method demonstrates robustness across diverse applications with varying root causes of unknown unknowns, such as distributional shifts and rare class instances.
• The framework successfully identifies high-confidence incorrect predictions that would otherwise remain undetected, validating its role in improving model reliability.
• To the best of the authors' knowledge, this is the first algorithmic approach to systematically discover unknown unknowns in predictive models, marking a foundational contribution to model uncertainty and reliability research.