[Paper Review] Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language?

Peter Hase, Shiyue Zhang+2 more|arXiv (Cornell University)|Oct 8, 2020

Topic Modeling39 references4 citations

TL;DR

This paper introduces Leakage-Adjusted Simulatability (LAS), a novel metric to evaluate whether natural language explanations from AI models genuinely support their predictions, rather than merely mimicking human plausibility. Using a language model as a proxy observer, LAS measures how well explanations enable prediction of model outputs while controlling for label leakage; the proposed ST-Ra model achieves human-level LAS scores, demonstrating faithful, non-trivial explanation generation.

ABSTRACT

Data collection for natural language (NL) understanding tasks has increasingly included human explanations alongside data points, allowing past works to introduce models that both perform a task and generate NL explanations for their outputs. Yet to date, model-generated explanations have been evaluated on the basis of surface-level similarities to human explanations, both through automatic metrics like BLEU and human evaluations. We argue that these evaluations are insufficient, since they fail to indicate whether explanations support actual model behavior (faithfulness), rather than simply match what a human would say (plausibility). In this work, we address the problem of evaluating explanations from the model simulatability perspective. Our contributions are as follows: (1) We introduce a leakage-adjusted simulatability (LAS) metric for evaluating NL explanations, which measures how well explanations help an observer predict a model's output, while controlling for how explanations can directly leak the output. We use a model as a proxy for a human observer, and validate this choice with two human subject experiments. (2) Using the CoS-E and e-SNLI datasets, we evaluate two existing generative graphical models and two new approaches; one rationalizing method we introduce achieves roughly human-level LAS scores. (3) Lastly, we frame explanation generation as a multi-agent game and optimize explanations for simulatability while penalizing label leakage, which can improve LAS scores. We provide code for the experiments in this paper at https://github.com/peterbhase/LAS-NL-Explanations

Motivation & Objective

• To address the limitation of existing evaluation methods that prioritize surface-level similarity (e.g., BLEU) or plausibility over faithfulness in model-generated explanations.
• To develop a metric that measures whether explanations enable accurate prediction of model outputs (simulatability), while accounting for direct leakage of the label.
• To validate the proposed metric using human subject experiments, showing alignment with expert and crowd-sourced evaluations.
• To explore whether training objectives optimized for simulatability can improve explanation quality beyond standard language modeling.
• To investigate the trade-off between model accuracy and explanation faithfulness, especially in the context of rationalizing vs. reasoning models.

Proposed method

• Proposes Leakage-Adjusted Simulatability (LAS), a metric that computes the accuracy of a simulator (a language model) in predicting a model’s output given the input and explanation, while controlling for label leakage via a baseline model that predicts only from the explanation.
• Uses a pretrained T5 model as a simulator to automate LAS evaluation, validated through two human experiments: expert simulatability and crowdsourced ratings.
• Introduces a multi-agent game framework where explanations are optimized for simulatability while penalizing label leakage, using a proxy objective during training.
• Employs a leakage control mechanism by comparing the simulator’s accuracy when given both input and explanation ($\mathbbm{1}[\hat{y}|x,\hat{e}]$) to its accuracy when given only the explanation ($\mathbbm{1}[\hat{y}|\hat{e}]$), with the difference indicating leakage.
• Applies the LAS metric to evaluate four models—MT-Ra, ST-Re, MT-Re, and a new ST-Ra—on CoS-E and e-SNLI datasets, using both automatic and human evaluations.
• Uses regression analysis to correlate LAS scores with human ratings, revealing that label leakage is the strongest predictor of human preference.

Experimental results

Research questions

• RQ1 Can a metric be developed that evaluates whether model-generated natural language explanations actually help predict model behavior, beyond matching human explanations?
• RQ2 To what extent do existing explanation generation methods produce explanations that are faithful to model behavior rather than just plausible?
• RQ3 How does label leakage in explanations affect human perception of explanation quality, and can this be disentangled from true simulatability?
• RQ4 Can training objectives be designed to improve explanation quality by optimizing for simulatability while reducing label leakage?
• RQ5 Is there a trade-off between model accuracy and explanation faithfulness when using explanations as supervision?

Key findings

• The ST-Ra model achieves LAS scores comparable to human explanations, indicating it can generate non-trivial, behavior-supporting explanations.
• Rationalizing models (which condition on the label) outperform reasoning models (which do not), suggesting label conditioning improves simulatability.
• There is no statistically significant relationship between model accuracy and LAS scores, indicating that faithfulness and accuracy are largely decoupled.
• The strongest predictor of human explanation ratings is whether the explanation leaks the label, not its simulatability or faithfulness.
• Optimizing explanations via a multi-agent game framework improves LAS scores, though with unstable training and minimal impact on accuracy.
• The automatic LAS metric correlates strongly with expert simulation results and human ratings, validating its use as a reliable proxy for human evaluation.