[Paper Review] Towards Neural Theorem Proving at Scale

Pasquale Minervini, Matko Bošnjak+2 more|arXiv (Cornell University)|Jul 21, 2018

Topic Modeling19 references16 citations

TL;DR

This paper proposes NTP 2.0, a scalable neural theorem prover that accelerates inference by replacing exhaustive proof path enumeration with approximate nearest neighbors (ANNS) to retrieve only the highest-scoring candidate facts. The method enables training and inference on large knowledge bases like WordNet, achieving state-of-the-art performance on multiple benchmarks while drastically reducing computational cost.

ABSTRACT

Neural models combining representation learning and reasoning in an end-to-end trainable manner are receiving increasing interest. However, their use is severely limited by their computational complexity, which renders them unusable on real world datasets. We focus on the Neural Theorem Prover (NTP) model proposed by Rockt{ä}schel and Riedel (2017), a continuous relaxation of the Prolog backward chaining algorithm where unification between terms is replaced by the similarity between their embedding representations. For answering a given query, this model needs to consider all possible proof paths, and then aggregate results - this quickly becomes infeasible even for small Knowledge Bases (KBs). We observe that we can accurately approximate the inference process in this model by considering only proof paths associated with the highest proof scores. This enables inference and learning on previously impracticable KBs.

Motivation & Objective

• To address the computational infeasibility of neural theorem provers (NTPs) on large knowledge bases due to exhaustive proof path enumeration.
• To enable end-to-end training and inference on real-world, large-scale knowledge bases previously considered impractical for NTPs.
• To improve scalability of neural-symbolic reasoning by replacing exact similarity searches with efficient approximate nearest neighbor (ANNS) methods.
• To evaluate the performance of the proposed method on standard benchmarks and on the large WordNet KB.
• To demonstrate that ANNS-based approximation preserves predictive accuracy while enabling training on previously intractable datasets.

Proposed method

• Replace the exhaustive search over all possible proof paths in NTP with an approximate nearest neighbor (ANNS) search to retrieve only the top-k most relevant facts for each query.
• Use ANNS to efficiently identify candidate facts with the highest similarity scores to query terms in embedding space, significantly reducing computational cost.
• Retain the continuous relaxation of Prolog-style backward chaining, where unification is replaced by embedding similarity computation.
• Train the model end-to-end using link prediction objectives, maximizing scores for true facts and minimizing scores for negative facts.
• Apply ANNS during both training and inference to scale the model to large knowledge bases such as WordNet, which contains 112,581 facts.
• Use k=1 in experiments, as higher k values did not yield significant improvements, though k=1 may occasionally miss exact nearest neighbors due to approximation.

Experimental results

Research questions

• RQ1 Can approximate nearest neighbor search be used to scale neural theorem proving to large knowledge bases without sacrificing predictive performance?
• RQ2 How does the performance of the ANNS-based NTP 2.0 compare to the original NTP on standard benchmarks like Kinship, Nations, and UMLS?
• RQ3 Can NTP 2.0 be successfully trained and evaluated on large, real-world knowledge bases such as WordNet?
• RQ4 What is the impact of using k=1 in ANNS on model accuracy, especially in datasets requiring exact nearest neighbor retrieval?
• RQ5 What kind of logical rules can be induced by NTP 2.0 from large-scale knowledge bases like WordNet?

Key findings

• NTP 2.0 achieves a test accuracy of 65.72% on WordNet, which is on par with the Distance Model (68.3%) despite not using full hyperparameter tuning or regularization.
• On the Countries dataset, NTP 2.0 improves AUC-PR from 56.68% to 85.55% when k=1, demonstrating significant performance gains through efficient approximation.
• On Kinship, NTP 2.0 improves MRR from 0.60 to 0.65 and HITS@1 from 0.48 to 0.57, showing consistent gains across metrics.
• On UMLS, NTP 2.0 shows a drop in performance compared to the original NTP, suggesting that ANNS may be less suitable in datasets where exact nearest neighbor retrieval is crucial.
• The model successfully induces interpretable logical rules from WordNet, such as '_has_part(X,Y) :- _similar_to(Y,X)' with 0.998 confidence, demonstrating effective symbolic rule extraction.
• The use of ANNS enables training on WordNet—a KB previously considered out of reach for NTPs—demonstrating a major scalability breakthrough.