[Paper Review] Properties and Challenges of LLM-Generated Explanations

Jenny Kunz, Marco Kuhlmann|arXiv (Cornell University)|Feb 16, 2024

Natural Language Processing Techniques4 citations

TL;DR

This paper investigates the properties of explanations generated by large language models (LLMs), focusing on GPT-4's outputs from the Alpaca instruction fine-tuning dataset. It finds that LLM-generated explanations frequently exhibit selectivity and illustrative elements—common traits in human explanations—while subjectivity and misleading content are less prevalent, though still present. The study highlights that these properties have both positive and negative implications depending on the application context and user group.

ABSTRACT

The self-rationalising capabilities of large language models (LLMs) have been explored in restricted settings, using task/specific data sets. However, current LLMs do not (only) rely on specifically annotated data; nonetheless, they frequently explain their outputs. The properties of the generated explanations are influenced by the pre-training corpus and by the target data used for instruction fine-tuning. As the pre-training corpus includes a large amount of human-written explanations "in the wild", we hypothesise that LLMs adopt common properties of human explanations. By analysing the outputs for a multi-domain instruction fine-tuning data set, we find that generated explanations show selectivity and contain illustrative elements, but less frequently are subjective or misleading. We discuss reasons and consequences of the properties' presence or absence. In particular, we outline positive and negative implications depending on the goals and user groups of the self-rationalising system.

Motivation & Objective

• To identify and systematize typical properties of human explanations that are often considered unsuitable for explainable AI, such as selectivity, subjectivity, and illustrative elements.
• To investigate the extent to which these human-like explanation properties are reflected in LLM-generated explanations, particularly from GPT-4 using the Alpaca dataset.
• To evaluate the implications of these properties for different goals of explainable NLP, including safety, trustworthiness, troubleshooting, and knowledge discovery.
• To assess how these explanation properties affect different user groups—such as end-users, developers, and researchers—across various application contexts.
• To highlight the trade-offs between human-like naturalness and the technical requirements of faithful, accurate, and complete model explanations.

Proposed method

• Conducted a human annotation study on a subset of 100 instruction-following examples from the Alpaca dataset, which were generated by GPT-4.
• Used a structured questionnaire to assess whether generated explanations exhibit specific properties: selectivity, illustrative elements, subjectivity, and misleading content.
• Analyzed the frequency and context of each property through qualitative and quantitative coding by three raters with consistent annotation guidelines.
• Mapped the presence of each property to specific use cases (e.g., trust, troubleshooting, knowledge discovery) to evaluate functional implications.
• Evaluated the impact of alignment techniques (e.g., instruction fine-tuning and filtering) on reducing undesirable properties like subjectivity and misleadingness.
• Compared findings to prior work on human-annotated explanation datasets to contrast properties in model-generated vs. human-annotated explanations.

Figure 1: Distribution of the categories defined in Section 4.1 in the evaluation set.

Experimental results

Research questions

• RQ1 To what extent do LLM-generated explanations in the Alpaca dataset reflect common properties of human explanations, such as selectivity and illustrative elements?
• RQ2 How frequently do LLM-generated explanations contain subjective or misleading content, and what factors in the training and alignment process may reduce these properties?
• RQ3 How do the properties of LLM-generated explanations affect their utility for different goals, such as improving model trustworthiness, enabling troubleshooting, or supporting knowledge discovery?
• RQ4 In what ways do the properties of LLM-generated explanations align with or conflict with user expectations across different user groups (e.g., end-users, developers, researchers)?
• RQ5 How do the properties of explanations vary across different types of instructions and domains, and what are the limitations of generalizing findings from the Alpaca dataset?

Key findings

• LLM-generated explanations in the Alpaca dataset frequently exhibit selectivity, meaning they highlight only certain aspects of reasoning rather than providing complete justifications.
• Illustrative elements—such as analogies, examples, or metaphors—were commonly present in explanations, enhancing accessibility and rhetorical appeal.
• Subjective content was less frequent, likely due to alignment processes like instruction fine-tuning and filtering in GPT-4’s training pipeline.
• Misleading explanations were observed only rarely, though the risk may increase for low-probability or out-of-distribution inputs not covered in the Alpaca dataset.
• Despite some undesirable traits like incompleteness or rhetorical persuasion, these properties can be beneficial in contexts like education or debugging, where clarity and engagement matter.
• The study underscores that properties like selectivity and illustrative content are not inherently problematic and can enhance usability, especially for less expert users, even if they compromise full faithfulness to model reasoning.

Figure 2: Comparison of the yes -answers the three annotators (A1, A2, A3) for Questions Q1 (“Does the output contain an explanation for the prediction?”) and Q2 (“Would you give an explanation/justify your reasoning if you were asked this question by a friend?”).