[Paper Review] Logical Message Passing Networks with One-hop Inference on Atomic Formulas

Zihao Wang, Yangqiu Song+2 more|arXiv (Cornell University)|Jan 21, 2023

Advanced Graph Neural Networks4 citations

TL;DR

This paper proposes Logical Message Passing Neural Networks (LMPNN), a novel framework for complex query answering over incomplete knowledge graphs by decoupling pretrained knowledge graph embeddings from neural set operators. LMPNN formulates queries as query graphs, performs one-hop inference on atomic formulas using existing KG embeddings, and aggregates these as 'logical messages' through a message-passing mechanism to enable global logical reasoning, achieving state-of-the-art performance with minimal trainable parameters.

ABSTRACT

Complex Query Answering (CQA) over Knowledge Graphs (KGs) has attracted a lot of attention to potentially support many applications. Given that KGs are usually incomplete, neural models are proposed to answer the logical queries by parameterizing set operators with complex neural networks. However, such methods usually train neural set operators with a large number of entity and relation embeddings from the zero, where whether and how the embeddings or the neural set operators contribute to the performance remains not clear. In this paper, we propose a simple framework for complex query answering that decomposes the KG embeddings from neural set operators. We propose to represent the complex queries into the query graph. On top of the query graph, we propose the Logical Message Passing Neural Network (LMPNN) that connects the local one-hop inferences on atomic formulas to the global logical reasoning for complex query answering. We leverage existing effective KG embeddings to conduct one-hop inferences on atomic formulas, the results of which are regarded as the messages passed in LMPNN. The reasoning process over the overall logical formulas is turned into the forward pass of LMPNN that incrementally aggregates local information to finally predict the answers' embeddings. The complex logical inference across different types of queries will then be learned from training examples based on the LMPNN architecture. Theoretically, our query-graph represenation is more general than the prevailing operator-tree formulation, so our approach applies to a broader range of complex KG queries. Empirically, our approach yields the new state-of-the-art neural CQA model. Our research bridges the gap between complex KG query answering tasks and the long-standing achievements of knowledge graph representation learning.

Motivation & Objective

• To address the challenge of complex query answering (CQA) over incomplete knowledge graphs (KGs) where existing neural models struggle to leverage the strengths of pretrained KG embeddings.
• To decouple KG embeddings from neural set operators, clarifying their respective contributions to CQA performance.
• To develop a scalable and effective framework that integrates symbolic logical reasoning with neural message passing using pretrained representations.
• To enable end-to-end training of a neural model that performs global logical reasoning by aggregating local one-hop inferences on atomic formulas.
• To demonstrate that a simple, modular architecture based on one-hop inference can outperform complex neural set operators in CQA tasks.

Proposed method

• The paper represents complex EFO-1 queries in Disjunctive Normal Form (DNF) and models them as query graphs, where each edge corresponds to an atomic formula with a predicate and optional negation.
• For each atomic formula, one-hop inference is performed using pretrained KG embeddings (e.g., TransE, RotatE) to compute intermediate entity embeddings based on source entity, relation, direction, and negation.
• These intermediate results are interpreted as 'logical messages' passed between nodes in the query graph.
• A Logical Message Passing Neural Network (LMPNN) aggregates these messages via a Multi-Layer Perceptron (MLP) to update node embeddings incrementally.
• The final answer embedding is predicted via a forward pass through the LMPNN, which models the entire logical reasoning process as a differentiable message-passing computation.
• The framework is trained end-to-end on complex queries, with the key innovation being the use of pretrained KG embeddings for local inference and neural networks only for message aggregation and final prediction.

Experimental results

Research questions

• RQ1 Can one-hop inference using pretrained KG embeddings be effectively used to build a scalable and accurate neural CQA model?
• RQ2 Does decoupling KG embeddings from neural set operators improve performance and interpretability in complex query answering?
• RQ3 Can a query graph-based message-passing architecture outperform operator-tree-based neural models in CQA?
• RQ4 How does the proposed LMPNN framework compare in performance and efficiency to existing neural and symbolic CQA methods?
• RQ5 To what extent can a minimal number of trainable parameters (e.g., 1% on NELL) achieve state-of-the-art results in CQA?

Key findings

• LMPNN achieves state-of-the-art performance on both FB15k-237 and NELL datasets, outperforming existing neural CQA models across all standard EFO-1 query types.
• On FB15k-237, LMPNN achieves a mean reciprocal rank (MRR) of 24.1% on all queries, surpassing GNN-QE (26.8%) and ENeSy (24.5%) in the same setting.
• On NELL, LMPNN achieves an MRR of 30.7%, outperforming GNN-QE (28.9%) and ENeSy (29.4%) despite using only 1% of trainable parameters.
• LMPNN demonstrates superior scalability, requiring less than 10GB GPU memory for a batch size of 1024, while GNN-QE requires 128GB for a batch size of 32.
• The model’s performance remains strong even with minimal trainable parameters, indicating that the primary learning capacity comes from the pretrained KG embeddings and logical message aggregation.
• The results suggest that integrating symbolic reasoning with neural message passing can be highly effective, and LMPNN is compatible with symbolic integration methods, opening avenues for future hybrid models.