[Paper Review] Learning to Explain: Answering Why-Questions via Rephrasing

Allen Nie, Erin Bennett+1 more|arXiv (Cornell University)|Jun 4, 2019

Topic Modeling22 references4 citations

TL;DR

This paper proposes a sequence-to-sequence model that generates natural language explanations for open-domain why-questions by training on automatically collected phenomenon-explanation pairs derived from dependency parsing of 'because' clauses in text. The method achieves human-rated explanations that are fluent and plausible, outperforming baselines in generating coherent, context-aware explanations without requiring curated knowledge bases.

ABSTRACT

Providing plausible responses to why questions is a challenging but critical goal for language based human-machine interaction. Explanations are challenging in that they require many different forms of abstract knowledge and reasoning. Previous work has either relied on human-curated structured knowledge bases or detailed domain representation to generate satisfactory explanations. They are also often limited to ranking pre-existing explanation choices. In our work, we contribute to the under-explored area of generating natural language explanations for general phenomena. We automatically collect large datasets of explanation-phenomenon pairs which allow us to train sequence-to-sequence models to generate natural language explanations. We compare different training strategies and evaluate their performance using both automatic scores and human ratings. We demonstrate that our strategy is sufficient to generate highly plausible explanations for general open-domain phenomena compared to other models trained on different datasets.

Motivation & Objective

• To address the challenge of generating natural, fluent, and plausible explanations for open-domain why-questions, moving beyond ranking pre-existing explanations.
• To overcome data sparsity in existing chatbot datasets that prevent models from answering 'why' questions effectively.
• To develop a scalable, end-to-end method for explanation generation using neural sequence-to-sequence models trained on automatically constructed datasets.
• To enable models to function as single-round chitchat chatbots by rephrasing 'why' questions into statement form for input to the explanation generator.
• To evaluate the quality of generated explanations using both automatic metrics and human evaluation on grammaticality, plausibility, and helpfulness.

Proposed method

• The method uses dependency parsing via Stanford CoreNLP to extract phenomenon-explanation pairs from sentences containing 'because', splitting them into S1 (phenomenon) and S2 (explanation).
• It constructs a new dataset of self-explanations by collecting context (five preceding sentences) and concatenating it with S1 using a <SEP> token to form the input sequence for the model.
• The model is trained as a sequence-to-sequence task: given C1,...,C5 <SEP> S1, predict S2 as the explanation.
• A novel algorithm (Algorithm 1) converts 'why' questions into S1-style statements by extracting subject, auxiliary, and verb components from the dependency parse.
• The model uses a Transformer-based architecture (L2E-Seq2Seq) and is compared against LSTM-based models and language models fine-tuned on other datasets.
• Human evaluation is conducted via Amazon Mechanical Turk, with participants rating generated explanations on grammaticality, plausibility, helpfulness, and relatedness.

Experimental results

Research questions

• RQ1 Can neural sequence-to-sequence models generate fluent and plausible natural language explanations for open-domain why-questions without relying on curated knowledge bases?
• RQ2 Does incorporating contextual sentences (five prior to the phenomenon) improve the thematic relevance and quality of generated explanations?
• RQ3 Can a model trained on automatically extracted 'because' clause pairs generalize to generate explanations that are rated as plausible by humans?
• RQ4 How does the performance of the proposed model compare to baselines in terms of human-rated explanation quality and fluency?
• RQ5 To what extent can the model be adapted to function as a single-round chitchat chatbot for answering 'why' questions?

Key findings

• The L2E-Seq2Seq model achieved a human-rated grammaticality score of 0.738 (95% CI: 0.70–0.77), outperforming the original explanations (0.684) on this metric.
• On overall plausibility, the model scored 0.543 (95% CI: 0.50–0.59), which was lower than the original explanations (0.710), but still above the neutral threshold of 0.5.
• The model generated explanations that were rated more helpful (0.512) than the original explanations (0.696) in human evaluation, indicating potential for practical deployment in dialogue systems.
• Human evaluators found the generated explanations to be more grammatically correct than the original explanations, suggesting improved fluency despite lower plausibility ratings.
• The model outperformed baseline language models (e.g., LM-1B) and other Seq2Seq models trained on different datasets in generating contextually relevant and natural-sounding explanations.
• The study confirms that training on automatically collected self-explanations from 'because' clauses enables models to generate plausible explanations, though human-level quality remains unachieved.