[Paper Review] LLMs for XAI: Future Directions for Explaining Explanations

Alexandra Zytek, Sara Pidò+1 more|arXiv (Cornell University)|May 9, 2024

Scientific Computing and Data Management10 citations

TL;DR

The paper proposes using LLMs to convert SHAP-based ML explanations into natural narrative explanations, outlines evaluation metrics and prompt designs, and presents initial GPT-3.5 and GPT-4 results plus a pilot user study.

ABSTRACT

In response to the demand for Explainable Artificial Intelligence (XAI), we investigate the use of Large Language Models (LLMs) to transform ML explanations into natural, human-readable narratives. Rather than directly explaining ML models using LLMs, we focus on refining explanations computed using existing XAI algorithms. We outline several research directions, including defining evaluation metrics, prompt design, comparing LLM models, exploring further training methods, and integrating external data. Initial experiments and user study suggest that LLMs offer a promising way to enhance the interpretability and usability of XAI.

Motivation & Objective

• Motivate the need for more natural, human-readable ML explanations for domain experts without ML backgrounds.
• Propose research directions to enhance LLM-assisted XAI, including evaluation metrics, prompt design, model comparisons, training methods, and data integration.
• Investigate zero-shot LLM capabilities to transform SHAP explanations into narratives and assess initial performance.
• Provide early empirical and user-study insights to guide future work in narrative XAI.

Proposed method

• Review and define metrics for evaluating narrative explanations (soundness, fluency, completeness, context-awareness, length).
• Experiment with five prompts across two datasets (student performance and Ames housing) using SHAP explanations as inputs.
• Compare GPT-3.5 and GPT-4 in zero-shot prompt-based narrative generation of explanations.
• Conduct a pilot user study to compare narrative versus plot-based explanations for usability and informativeness.
• Report averaged metric scores from prompting experiments and qualitative user feedback as initial progress.

Experimental results

Research questions

• RQ1 Can LLMs convert SHAP explanations into fluent, context-aware narrative explanations without additional training?
• RQ2 How do prompt designs and different LLMs affect the quality of narrative explanations?
• RQ3 What metrics capture the quality of narrative explanations, and how do LLMs perform on them?
• RQ4 Do narrative explanations improve user understanding and trust compared to traditional explanations?
• RQ5 What are viable directions (training, data integration, model comparison) to improve LLM-based XAI narratives?

Key findings

• GPT-4 explanations are more sound, complete, and context-aware than GPT-3.5, though GPT-3.5 tends to be shorter and more fluent.
• GPT-4 produced few errors in the experiment, suggesting potential for high-stakes contexts with careful evaluation.
• Narrative-based explanations were preferred by participants in a pilot user study over plot-based explanations across usability and informativeness measures.
• Promising evidence that narrative explanations can enhance interpretability and usability of ML explanations.
• The work establishes metrics and prompts as a foundation for further exploration of LLMs in XAI, including future fine-tuning and data integration.