[論文レビュー] Abductive Commonsense Reasoning

Chandra Bhagavatula, Ronan Le Bras+7 more|arXiv (Cornell University)|Aug 15, 2019

Topic Modeling参考文献 46被引用数 34

ひとこと要約

この論文は abductive commonsense reasoning のための ART データセットを導入し、Abductive Natural Language Inference (alpha NLI) および Abductive Natural Language Generation (alpha NLG) を定義します。強力なベースラインを評価し、人間のパフォーマンスとの大きなギャップを示し、モデルの限界と転移学習の潜在能力を分析します。

ABSTRACT

Abductive reasoning is inference to the most plausible explanation. For example, if Jenny finds her house in a mess when she returns from work, and remembers that she left a window open, she can hypothesize that a thief broke into her house and caused the mess, as the most plausible explanation. While abduction has long been considered to be at the core of how people interpret and read between the lines in natural language (Hobbs et al., 1988), there has been relatively little research in support of abductive natural language inference and generation. We present the first study that investigates the viability of language-based abductive reasoning. We introduce a challenge dataset, ART, that consists of over 20k commonsense narrative contexts and 200k explanations. Based on this dataset, we conceptualize two new tasks -- (i) Abductive NLI: a multiple-choice question answering task for choosing the more likely explanation, and (ii) Abductive NLG: a conditional generation task for explaining given observations in natural language. On Abductive NLI, the best model achieves 68.9% accuracy, well below human performance of 91.4%. On Abductive NLG, the current best language generators struggle even more, as they lack reasoning capabilities that are trivial for humans. Our analysis leads to new insights into the types of reasoning that deep pre-trained language models fail to perform--despite their strong performance on the related but more narrowly defined task of entailment NLI--pointing to interesting avenues for future research.

研究の動機と目的

• アブダクティブ推論を人間の常識解釈の核心的側面として動機づける。
• 現実的な説明を伴う物語的文脈の大規模データセット (ART) を作成する。
• 二つの新しいタスクを定義する: abductive natural language inference (alpha NLI) および abductive natural language generation (alpha NLG)。
• state-of-the-art NLI モデルと言語生成モデルを用いて強力なベースラインを提供し、ベンチマークを確立する。

提案手法

• alpha NLI を O1 および O2 に基づき最も妥当な仮説を選択する二値の多肢選択タスクとして定義する。
• O1, O2, および H の関係を捉えるための確率モデル（完全連結、線形連鎖、依存関係）を提案する。
• alpha NLG を O1, O2 に基づく h+ の条件付き生成としてモデル化し、COMeT / ATOMIC からの背景知識を任意で利用する。
• ROCStories の物語を、群衆から得たもっともらしい/ありえない仮説と組み合わせ、アーティファクトを最小化する敵対的フィルタリングを用いて ART を構築する。
• alpha NLI には BERT ベースの分類器、alpha NLG には GPT2 ベースの生成モデルを用いてベースラインを評価し、人間の基準と比較して分析する。

実験結果

リサーチクエスチョン

• RQ1 Can language models perform abductive reasoning over narrative observations better than chance or simple entailment baselines?
• RQ2 What are the limitations of current pre-trained language models in abductive reasoning across different commonsense categories?
• RQ3 Does incorporating structured commonsense knowledge (e.g., COMeT / ATOMIC) improve abductive generation and inference?
• RQ4 Can training on ART improve performance on other commonsense tasks through transfer learning?

主な発見

モデル	GPT AF 精度（%）	ART 精度（%）
Random	50.1	50.4
Majority	50.1	50.8
Infersent (Conneau et al., 2017)	50.9	50.8
ESIM+ELMo (Chen et al., 2017)	58.2	58.8
GPT-ft	52.6 (0.9)	63.1 (0.5)
BERT-ft [h^{i} Only]	55.9 (0.7)	59.5 (0.2)
BERT-ft [O1 Only]	63.9	63.5
BERT-ft [O2 Only]	68.1	66.6
BERT-ft [Linear Chain]	65.3	68.9
BERT-ft [Fully Connected]	72.0 (0.5)	68.6 (0.5)
Human Performance	-	91.4

• Best alpha NLI baseline (BERT-based fully connected) achieves 68.9% accuracy, far below human 91.4%.
• Humans outperform models across all evaluated categories; simple entailment baselines perform near chance on ART.
• Alpha NLG is significantly harder; best generators reach about 45% vs human 96% on held-out hypotheses.
• Adversarial filtering and model architecture (fully connected vs. linear chain) impact performance, with fully connected often performing better against strong baselines.
• ART enables transfer learning benefits to smaller target datasets (e.g., WinoGrande, WSC, DPR, Hellaswag) when pre-trained on ART, especially with limited target data.