[Paper Review] Case-Based Abductive Natural Language Inference

Marco Valentino, Mokanarangan Thayaparan+1 more|arXiv (Cornell University)|Sep 30, 2020

Topic Modeling40 references4 citations

TL;DR

This paper proposes Case-Based Abductive Natural Language Inference (CB-ANLI), a novel framework that leverages Case-Based Reasoning (CBR) to improve multi-hop abductive natural language inference by retrieving, reusing, and refining explanations from similar past cases. The method reduces semantic drift and boosts robustness and accuracy, especially on challenging explanations, by constraining reasoning with prior solutions, outperforming isolated explanation generation in commonsense and scientific reasoning tasks when integrated with pre-trained encoders.

ABSTRACT

Most of the contemporary approaches for multi-hop Natural Language Inference (NLI) construct explanations considering each test case in isolation. However, this paradigm is known to suffer from semantic drift, a phenomenon that causes the construction of spurious explanations leading to wrong conclusions. In contrast, this paper proposes an abductive framework for multi-hop NLI exploring the retrieve-reuse-refine paradigm in Case-Based Reasoning (CBR). Specifically, we present Case-Based Abductive Natural Language Inference (CB-ANLI), a model that addresses unseen inference problems by analogical transfer of prior explanations from similar examples. We empirically evaluate the abductive framework on commonsense and scientific question answering tasks, demonstrating that CB-ANLI can be effectively integrated with sparse and dense pre-trained encoders to improve multi-hop inference, or adopted as an evidence retriever for Transformers. Moreover, an empirical analysis of semantic drift reveals that the CBR paradigm boosts the quality of the most challenging explanations, a feature that has a direct impact on robustness and accuracy in downstream inference tasks.

Motivation & Objective

• To address the problem of semantic drift in abductive multi-hop NLI, where isolated explanation generation leads to spurious reasoning chains.
• To explore whether analogical transfer of prior explanations via Case-Based Reasoning (CBR) can improve the quality and robustness of explanations in abstractive NLI.
• To demonstrate that the retrieve-reuse-refine paradigm of CBR enhances performance on challenging inference problems, especially when explanations are abstract and low-lexical-overlap.
• To evaluate the integration of CB-ANLI with sparse and dense pre-trained encoders, and its effectiveness as an evidence retriever for Transformers.
• To empirically analyze how CBR mitigates semantic drift and improves downstream inference accuracy.

Proposed method

• The framework adopts the CBR retrieve-reuse-refine paradigm: for a new hypothesis, it retrieves the most similar past cases based on semantic similarity of hypotheses and explanations.
• It reuses the explanation from the most similar retrieved case as a starting point for constructing a new explanation for the current hypothesis.
• The refinement phase adapts the retrieved explanation by aligning it with the new hypothesis using contextualized sentence encoders and abstraction mechanisms to handle semantic differences.
• The system scores each hypothesis based on the quality of its generated explanation, selecting the hypothesis supported by the best explanation.
• The method integrates with both sparse (e.g., BM25) and dense (e.g., BERT) pre-trained encoders for retrieval and encoding, enabling compatibility with downstream Transformer models.
• Explanations are generated using a constrained generation process that leverages the structure and content of prior solutions to avoid spurious chains.

Experimental results

Research questions

• RQ1 Can the case-based reasoning paradigm reduce semantic drift in abductive multi-hop NLI by constraining explanation generation with prior solutions?
• RQ2 Does reusing and refining explanations from similar past cases improve the quality and accuracy of explanations on challenging, abstract reasoning problems?
• RQ3 How does CB-ANLI compare to isolated explanation generation methods in terms of robustness and performance on commonsense and scientific reasoning tasks?
• RQ4 To what extent can CB-ANLI be effectively integrated with pre-trained sentence encoders and downstream Transformer models?
• RQ5 Does the CBR framework particularly enhance the quality of explanations for the most difficult inference cases?

Key findings

• CB-ANLI significantly reduces semantic drift by leveraging prior solutions, especially improving explanation quality on the most challenging inference problems.
• The model achieves strong performance on commonsense and scientific reasoning tasks when integrated with both sparse and dense pre-trained encoders.
• Empirical results show that the CBR paradigm boosts the quality of explanations, leading to improved downstream inference accuracy and robustness.
• CB-ANLI outperforms existing abductive and multi-hop NLI approaches by effectively constraining the search space through analogical transfer of explanations.
• The framework demonstrates effectiveness as an evidence retriever for Transformers, enhancing their reasoning capabilities through retrieved, refined explanations.
• The ablation study confirms that the retrieve-reuse-refine cycle is critical for performance, with the reuse and refinement phases being particularly effective in mitigating spurious reasoning.