[Paper Review] Beta Embeddings for Multi-Hop Logical Reasoning in Knowledge Graphs

Hongyu Ren, Jure Leskovec|arXiv (Cornell University)|Oct 22, 2020

Advanced Graph Neural Networks37 references74 citations

TL;DR

BetaE embeds entities and queries as Beta distributions to enable multi-hop first-order logic reasoning over incomplete knowledge graphs, supporting full FOL operators including negation and providing state-of-the-art results on EPFO queries and beyond.

ABSTRACT

One of the fundamental problems in Artificial Intelligence is to perform complex multi-hop logical reasoning over the facts captured by a knowledge graph (KG). This problem is challenging, because KGs can be massive and incomplete. Recent approaches embed KG entities in a low dimensional space and then use these embeddings to find the answer entities. However, it has been an outstanding challenge of how to handle arbitrary first-order logic (FOL) queries as present methods are limited to only a subset of FOL operators. In particular, the negation operator is not supported. An additional limitation of present methods is also that they cannot naturally model uncertainty. Here, we present BetaE, a probabilistic embedding framework for answering arbitrary FOL queries over KGs. BetaE is the first method that can handle a complete set of first-order logical operations: conjunction ($\wedge$), disjunction ($\vee$), and negation ($ eg$). A key insight of BetaE is to use probabilistic distributions with bounded support, specifically the Beta distribution, and embed queries/entities as distributions, which as a consequence allows us to also faithfully model uncertainty. Logical operations are performed in the embedding space by neural operators over the probabilistic embeddings. We demonstrate the performance of BetaE on answering arbitrary FOL queries on three large, incomplete KGs. While being more general, BetaE also increases relative performance by up to 25.4% over the current state-of-the-art KG reasoning methods that can only handle conjunctive queries without negation.

Motivation & Objective

• Motivate and address the challenge of answering arbitrary FOL queries on large, incomplete KGs.
• Develop a probabilistic embedding framework that supports all FOL operators including negation.
• Design neural operators operating on Beta distributions to perform relation projection, intersection, and negation in a closed, scalable manner.
• Model uncertainty explicitly through Beta embeddings and quantify via differential entropy.
• Demonstrate state-of-the-art performance on standard KG benchmarks for complex queries and assess uncertainty modeling capability.

Proposed method

• Embed entities and queries as Beta distributions on [0,1], allowing probabilistic reasoning and bounded-support properties.
• Implement probabilistic projection via relation-specific MLPs: S' = MLP_r(S) to map input Beta embeddings through KG relations.
• Define probabilistic intersection as a weighted product of PDFs: p_SInter ∝ ∏ p_Si^{wi}, with weights learned via attention and summing to 1.
• Define probabilistic negation by transforming Beta parameters to approximate complement, via N([α,β]) = [1/α, 1/β], enabling closed logical operations.
• Train by minimizing a negative sampling objective using KL divergence-based distance between query embeddings and answer/entity embeddings.
• Utilize De Morgan’s laws to approximate disjunction via intersection and negation, enabling complete FOL handling while maintaining linear complexity in the number of unions.

Experimental results

Research questions

• RQ1 Can Beta embeddings represent arbitrary FOL queries (including negation) over incomplete KGs?
• RQ2 How do probabilistic operators (projection, intersection, negation) behave to reflect logical properties and closure?
• RQ3 Do BetaE embeddings accurately model uncertainty and correlate embedding entropy with query answer cardinality?
• RQ4 How does BetaE perform on standard KG benchmarks relative to methods restricted to EPFO queries?
• RQ5 Can the model handle queries with negation and still generalize to unseen query structures?

Key findings

• BetaE achieves state-of-the-art or competitive results on EPFO queries, with relative improvements up to 25.4% over prior methods that lack negation.
• BetaE successfully handles arbitrary FOL queries including negation, outperforming baselines on datasets FB15k, FB15k-237, and NELL995.
• The probabilistic Beta embeddings capture query uncertainty, showing strong correlation between differential entropy and answer set size (up to 77% relative improvement over a baselines).
• Negation and intersection operators are closed under BetaE, with N(N(S)) = S and I(S,...,S) = S when using identical inputs, ensuring logical consistency.
• The model can classify queries with/without answers using embedding entropy, achieving ROC-AUC up to 0.844 on NELL.