[Paper Review] Human-in-the-Loop Interpretability Prior

Isaac Lage, Andrew Slavin Ross+3 more|arXiv (Cornell University)|May 29, 2018

Fault Detection and Control Systems45 citations

TL;DR

We introduce a human-in-the-loop prior p(M) to optimize for models that are both predictive and interpretable, using a model-based optimization approach to minimize the number of user studies needed.

ABSTRACT

We often desire our models to be interpretable as well as accurate. Prior work on optimizing models for interpretability has relied on easy-to-quantify proxies for interpretability, such as sparsity or the number of operations required. In this work, we optimize for interpretability by directly including humans in the optimization loop. We develop an algorithm that minimizes the number of user studies to find models that are both predictive and interpretable and demonstrate our approach on several data sets. Our human subjects results show trends towards different proxy notions of interpretability on different datasets, which suggests that different proxies are preferred on different tasks.

Motivation & Objective

• Motivate integrating human interpretability into model training to improve safety and usefulness.
• Propose a practical prior p(M) that captures human interpretability preferences directly from user feedback.
• Develop a cost-effective pipeline to identify high-likelihood predictive models and optimize for interpretability with few user studies.
• Show that different datasets align with different interpretability proxies, underscoring context-dependent preferences.

Proposed method

• Define p(X|M) via a SILF-based likelihood to encode a thresholded performance criterion.
• Define p(M) as a human-interpretability prior (HIS) that integrates mean response time (RT) from human evaluators.
• Extend p(M) to arbitrary models by using a local-proxy that locally approximates complex models for HIS evaluation.
• Use model-based optimization (Gaussian Process with a UCB acquisition) to select which models to evaluate for p(M).
• Identify a diverse set of high-likelihood models with p(X|M) and then search among them to maximize p(X|M)p(M) with as few user studies as possible.

Experimental results

Research questions

• RQ1 How can we quantify interpretability in a way that directly reflects human preferences?
• RQ2 Can we efficiently identify models that are both accurate and interpretable with a reduced number of human evaluations?
• RQ3 Do different proxies for interpretability align with human judgments across datasets and model classes?
• RQ4 What is the impact of using local proxies versus the full model on estimating interpretability priors?

Key findings

• Different proxies for interpretability lead to different optimal models on each dataset.
• Computing the right interpretability proxy on a small sample of data points often beats using the wrong proxy on a larger sample.
• Model-based optimization outperforms random model selection in finding human-preferred interpretable models across datasets.
• Human-in-the-loop optimization yields models with faster response times and lower proxy scores, indicating alignment with human interpretability.