[論文レビュー] Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanations

Shiyuan Huang, Siddarth Mamidanna+3 more|arXiv (Cornell University)|Oct 17, 2023

Topic Modeling被引用数 19

ひとこと要約

本論文は、感情分析におけるChatGPT生成の自己説明と従来の特徴量寄与法を比較し、忠実性と同意/一致性指標を横断して、忠実性は類似している一方で大きな不一致とプロンプトの制約を示すことを明らかにする。

ABSTRACT

Large language models (LLMs) such as ChatGPT have demonstrated superior performance on a variety of natural language processing (NLP) tasks including sentiment analysis, mathematical reasoning and summarization. Furthermore, since these models are instruction-tuned on human conversations to produce "helpful" responses, they can and often will produce explanations along with the response, which we call self-explanations. For example, when analyzing the sentiment of a movie review, the model may output not only the positivity of the sentiment, but also an explanation (e.g., by listing the sentiment-laden words such as "fantastic" and "memorable" in the review). How good are these automatically generated self-explanations? In this paper, we investigate this question on the task of sentiment analysis and for feature attribution explanation, one of the most commonly studied settings in the interpretability literature (for pre-ChatGPT models). Specifically, we study different ways to elicit the self-explanations, evaluate their faithfulness on a set of evaluation metrics, and compare them to traditional explanation methods such as occlusion or LIME saliency maps. Through an extensive set of experiments, we find that ChatGPT's self-explanations perform on par with traditional ones, but are quite different from them according to various agreement metrics, meanwhile being much cheaper to produce (as they are generated along with the prediction). In addition, we identified several interesting characteristics of them, which prompt us to rethink many current model interpretability practices in the era of ChatGPT(-like) LLMs.

研究の動機と目的

• LMM生成の自己説明が感情分析におけるモデル予測に忠実であるかを評価する

提案手法

• Explain-then-predictとPredict-then-explainの2つの自己説明パラダイムをプロンプトする
• 全文語レベルの寄与またはトップ-k語に基づく説明を生成する
• 従来の説明としてOcclusionとLIMEとを比較する
• 忠実性指標（comprehensiveness, sufficiency, DF MIT, DF Frac, Rank Del）と不一致指標で評価する
• トップ-k説明とそれらの指標を検討する
• ChatGPTの説明とサリエンシー値の定性的差異を分析する

Figure 1: An overview of our investigation. Current conversational LLMs can explain their answers (e.g., by highlighting important words in the input), often automatically or at least when asked to. How should we think of these self-explanations ? In this paper, we study them in relationship to trad

実験結果

リサーチクエスチョン

• RQ1 LLM生成の自己説明は感情分析における予測を忠実に支持するか？
• RQ2 ChatGPTの自己説明は忠実性と一致性の観点で従来の寄与法（occlusion, LIME）と比較してどうか？
• RQ3 Explain-then-predictとPredict-then-explainは精度と忠実性にどのような影響を与えるか？
• RQ4 トップ-kの説明は完全な寄与と比べて忠実性とコストの面で競争力があるか？
• RQ5 LLMs時代の解釈性ワークフローに対して現実的な示唆は何か？

主な発見

完全性↑	十分性↓	DF MIT↑	DF Frac↓	Rank Del↑
E-P (Accuracy: 85%)	0.15	0.26	0.18	-0.00
LIME	0.17	0.22	0.13	-0.02
SelfExp	0.19	0.25	0.16	-0.03
P-E (Accuracy: 88%)	0.20	0.23	0.14	-0.02
LIME	0.27	0.20	0.10	-0.02
SelfExp	0.27	0.22	0.07	-0.01

• 自己説明は忠実性指標で従来法と同等の性能を示すが、方法間で一致性の差が大きい
• Explain-then-predictはpredict-then-explainと比べて精度をわずかに悪化させる（85% vs 88%）、いずれも非説明モデル（92%）より低い
• LIMEとOcclusionは特定の指標で高コストまたはLLM自己説明と整合性が低い場合がある
• トップ-k説明は常に完全な寄与を上回るとは限らず、タスクとプロンプトによって有効性が異なる
• 説明法間の高い不一致は忠実性が類似していても評価パイプラインの限界を示唆する
• ChatGPTの説明は、厳密なスコアよりもある程度丸みを帯びたレベル（例：0.5、0.75）のサリエンシー値を生成する傾向がある

Figure 2: Visualization of one explanation each for E-P and P-E model. The top- $k$ explanations