[Paper Review] NeuralLog: Natural Language Inference with Joint Neural and Logical Reasoning

Zeming Chen, Qiyue Gao+1 more|arXiv (Cornell University)|May 29, 2021

Topic Modeling31 references4 citations

TL;DR

NeuralLog proposes a hybrid framework that integrates neural network-based paraphrase detection with a monotonicity-driven logical inference engine to improve Natural Language Inference (NLI). By modeling NLI as a path-planning problem using beam search, it dynamically routes reasoning tasks to either neural or logical modules based on their strengths, achieving state-of-the-art accuracy on SICK (93.4%) and MED (93.4%) datasets.

ABSTRACT

Deep learning (DL) based language models achieve high performance on various benchmarks for Natural Language Inference (NLI). And at this time, symbolic approaches to NLI are receiving less attention. Both approaches (symbolic and DL) have their advantages and weaknesses. However, currently, no method combines them in a system to solve the task of NLI. To merge symbolic and deep learning methods, we propose an inference framework called NeuralLog, which utilizes both a monotonicity-based logical inference engine and a neural network language model for phrase alignment. Our framework models the NLI task as a classic search problem and uses the beam search algorithm to search for optimal inference paths. Experiments show that our joint logic and neural inference system improves accuracy on the NLI task and can achieve state-of-art accuracy on the SICK and MED datasets.

Motivation & Objective

• Address the limitations of purely deep learning or symbolic NLI systems by integrating both approaches.
• Overcome the lack of generalization in deep learning models and the poor handling of syntactic variation in symbolic systems.
• Develop a unified framework where neural and logical reasoning complement each other in a shared inference pipeline.
• Improve robustness and accuracy in NLI by leveraging neural models for paraphrase detection and logic for monotonicity-based reasoning.
• Enable explainable, precise inference on complex linguistic phenomena while maintaining performance on diverse syntactic variations.

Proposed method

• Model the NLI task as a path-planning problem, where the premise is the start state and the hypothesis is the goal state.
• Use beam search to explore and select optimal inference paths through a sequence of reasoning steps.
• Integrate three inference modules: lexical, phrasal, and syntactic variation, each handling different types of reasoning.
• Apply neural network-based paraphrase detection on phrase-level chunks to handle syntactic variations.
• Employ a monotonicity-based logical inference engine to detect entailment and contradiction using formal semantic rules.
• Use a controller mechanism to dispatch reasoning tasks to the most appropriate module (neural or logical) based on the inference type.

Experimental results

Research questions

• RQ1 Can a joint framework combining neural and symbolic reasoning improve NLI performance beyond state-of-the-art deep learning models?
• RQ2 How can neural and logical reasoning be effectively coordinated in a single inference pipeline for NLI?
• RQ3 To what extent can neural models detect paraphrases at the phrase level to resolve syntactic variations in NLI?
• RQ4 Can monotonicity-based logical inference enhance accuracy on entailment and contradiction tasks, especially in complex cases?
• RQ5 How does the integration of both reasoning types affect robustness and generalization compared to isolated approaches?

Key findings

• NeuralLog achieves state-of-the-art accuracy of 93.4% on the SICK dataset, significantly outperforming BERT+ (71.6%) and BERT (44.7%).
• On the MED dataset, NeuralLog achieves 93.4% overall accuracy, surpassing BERT+ by 21.8 percentage points and BERT by 48.7 percentage points.
• The system improves performance on both upward (entailment) and downward (contradiction) inference, with gains of +15.4 and +23.6 over BERT+, respectively.
• Ablation studies show that neither neural nor logical reasoning alone achieves optimal performance, confirming the benefit of joint reasoning.
• Error analysis reveals that dependency parsing errors and lack of background knowledge are major failure points, especially for complex inferences.
• Inconsistencies in the SICK dataset’s gold labels—such as similar syntactic structures receiving different labels (e.g., folding/unfolding vs. playing/not playing)—highlight data quality challenges.