[Paper Review] Post Hoc Explanations of Language Models Can Improve Language Models

Satyapriya, Jiaqi Ma+4 more|arXiv (Cornell University)|May 19, 2023

Topic Modeling4 citations

TL;DR

This paper introduces AMPLIFY, a framework that improves large language models (LLMs) by automatically generating natural language rationales from post hoc explanation methods—such as gradient-based attribution—instead of relying on costly human-annotated rationales. By using smaller proxy models to compute explanations and injecting them as corrective prompts, AMPLIFY boosts LLM performance by 10–25% across diverse reasoning and understanding tasks, outperforming human-annotated methods like Chain-of-Thought prompting.

ABSTRACT

Large Language Models (LLMs) have demonstrated remarkable capabilities in performing complex tasks. Moreover, recent research has shown that incorporating human-annotated rationales (e.g., Chain-of-Thought prompting) during in-context learning can significantly enhance the performance of these models, particularly on tasks that require reasoning capabilities. However, incorporating such rationales poses challenges in terms of scalability as this requires a high degree of human involvement. In this work, we present a novel framework, Amplifying Model Performance by Leveraging In-Context Learning with Post Hoc Explanations (AMPLIFY), which addresses the aforementioned challenges by automating the process of rationale generation. To this end, we leverage post hoc explanation methods which output attribution scores (explanations) capturing the influence of each of the input features on model predictions. More specifically, we construct automated natural language rationales that embed insights from post hoc explanations to provide corrective signals to LLMs. Extensive experimentation with real-world datasets demonstrates that our framework, AMPLIFY, leads to prediction accuracy improvements of about 10-25% over a wide range of tasks, including those where prior approaches which rely on human-annotated rationales such as Chain-of-Thought prompting fall short. Our work makes one of the first attempts at highlighting the potential of post hoc explanations as valuable tools for enhancing the effectiveness of LLMs. Furthermore, we conduct additional empirical analyses and ablation studies to demonstrate the impact of each of the components of AMPLIFY, which, in turn, leads to critical insights for refining in-context learning.

Motivation & Objective

• To address the scalability and high cost of human-annotated rationales in in-context learning for LLMs.
• To explore whether post hoc explanation methods can serve as effective, automated sources of corrective signals for LLMs.
• To develop a framework that leverages attribution scores from smaller proxy models to improve LLM reasoning and prediction accuracy.
• To demonstrate that automated, explanation-driven prompting can outperform traditional human-annotated prompting techniques like Chain-of-Thought.

Proposed method

• Use a smaller, computationally feasible proxy model (e.g., GPT-2, BERT) to compute post hoc explanations for misclassified samples from the LLM’s predictions.
• Select misclassified validation samples where the proxy model is highly confident in its incorrect prediction, focusing on high-impact error cases.
• Apply gradient-based explanation methods—such as Gradient × Input or its contrastive variant—to identify the most influential input tokens for the ground truth label.
• Construct natural language rationales by embedding the top influential words into a prompt template: 'The key words: word1, word2, … and word5 are crucial clues for predicting [label]' to guide the LLM.
• Feed the resulting few-shot prompt (input, label, rationale) to the target LLM during inference to improve prediction accuracy.
• Evaluate the framework across diverse NLP tasks, including sentiment classification, commonsense reasoning, and causal judgment, using LLMs like GPT-3 and GPT-3.5.

Experimental results

Research questions

• RQ1 Can post hoc explanation methods be used to generate effective, automated rationales that improve LLM performance without human annotation?
• RQ2 How does the performance of LLMs using automatically generated rationales compare to those using human-annotated rationales like Chain-of-Thought prompting?
• RQ3 Does the choice of explanation method (e.g., Gradient × Input vs. Integrated Gradients) significantly affect the performance gain in the LLM?
• RQ4 How sensitive is the framework’s performance to variations in the rationale prompt template, such as phrasing or punctuation?
• RQ5 To what extent can post hoc explanations help LLMs recover from errors caused by overlooking critical input cues?

Key findings

• AMPLIFY improves LLM prediction accuracy by 10–25% across a wide range of tasks, significantly outperforming baseline prompting methods.
• The framework achieves consistent performance gains even on tasks where Chain-of-Thought prompting fails or underperforms, demonstrating its robustness.
• Gradient × Input and its contrastive variant produce the most effective rationales, though performance remains stable across different explanation methods.
• Tailoring the rationale template to the task context—e.g., adding 'a typical person would respond with' in causal judgment tasks—leads to measurable improvements, such as a 2.6% increase in accuracy on the Causal Judgment dataset.
• Qualitative analysis confirms that AMPLIFY helps LLMs attend to critical, previously overlooked words in input text, such as 'never supposed to touch' in a reasoning task.
• The framework is robust to minor template variations, indicating that performance gains are driven by the content of the explanation rather than superficial prompt formatting.