[Paper Review] Learning to Scaffold: Optimizing Model Explanations for Teaching

Patrick Fernandes, Marcos Treviso+3 more|arXiv (Cornell University)|Apr 22, 2022

Explainable Artificial Intelligence (XAI)11 citations

TL;DR

This paper introduces SMaT (Scaffold-Maximizing Training), a meta-learning framework that optimizes model explanations to improve student models' ability to simulate the teacher model's predictions. By treating explanation quality as a function of student simulability, SMaT trains an attention-based explainer to generate more effective, human-aligned explanations across NLP and vision tasks, significantly outperforming static or gradient-based baselines in both simulation accuracy and plausibility.

ABSTRACT

Modern machine learning models are opaque, and as a result there is a burgeoning academic subfield on methods that explain these models' behavior. However, what is the precise goal of providing such explanations, and how can we demonstrate that explanations achieve this goal? Some research argues that explanations should help teach a student (either human or machine) to simulate the model being explained, and that the quality of explanations can be measured by the simulation accuracy of students on unexplained examples. In this work, leveraging meta-learning techniques, we extend this idea to improve the quality of the explanations themselves, specifically by optimizing explanations such that student models more effectively learn to simulate the original model. We train models on three natural language processing and computer vision tasks, and find that students trained with explanations extracted with our framework are able to simulate the teacher significantly more effectively than ones produced with previous methods. Through human annotations and a user study, we further find that these learned explanations more closely align with how humans would explain the required decisions in these tasks. Our code is available at https://github.com/coderpat/learning-scaffold

Motivation & Objective

• To address the lack of a principled, measurable criterion for evaluating and improving model explanations beyond qualitative desiderata.
• To formalize explanation quality as the ability of a student model to simulate the teacher's behavior using only the explanation, aligning with pedagogical scaffolding.
• To develop a framework that directly optimizes explanations for better student learning, rather than evaluating existing methods.
• To improve the plausibility of explanations by aligning them with human reasoning patterns in decision-making.
• To enable end-to-end training of explainers without requiring human-annotated rationales, using simulability as a proxy for interpretability.

Proposed method

• Formalize the explanation optimization problem as a bi-level optimization: the explainer is updated based on student performance in simulating the teacher model.
• Use higher-order differentiation to backpropagate through the student training process, enabling end-to-end optimization of the explainer.
• Propose a parameterized attention-based explainer module that can be trained with SMaT and adapted to any attention-based model, such as transformers.
• Train the student model to predict the teacher’s output using both input and the explanation, with the explainer's parameters updated to maximize test-time simulation accuracy.
• Integrate the framework into diverse tasks—text classification, image classification, and multilingual text regression—using pretrained transformer models.
• Evaluate explanation quality via both automated simulability metrics and human-annotated plausibility, including human studies on image classification.

Experimental results

Research questions

• RQ1 Can we optimize model explanations such that student models learn to simulate the teacher model more accurately?
• RQ2 Do explanations generated by our framework better align with human reasoning patterns compared to existing methods?
• RQ3 Can the SMaT framework be applied across diverse modalities and tasks without architectural constraints?
• RQ4 Does end-to-end training of explainers via simulability lead to more faithful and non-trivial explanations than gradient-based or static attention methods?
• RQ5 Can the framework be extended to optimize non-attention explainers, such as gradient-based or rationalization methods?

Key findings

• Students trained with explanations generated by SMaT achieved significantly higher simulation accuracy on unseen examples compared to those using static attention or gradient-based explanations.
• Human-annotated evaluations showed that SMaT-generated explanations were more plausible and closer to human rationales in text classification and regression tasks.
• In a user study on image classification, participants rated SMaT-generated explanations as more intuitive and aligned with human decision-making than baseline methods.
• The framework successfully avoided trivial explanation protocols (e.g., focusing only on punctuation or stopwords) that plagued earlier simulability-optimized explainers.
• The parameterized attention explainer discovered meaningful attention heads in transformers, demonstrating interpretability beyond standard attention visualization.
• SMaT generalized across tasks, improving simulability in both NLP and computer vision benchmarks without architectural modifications.