[Paper Review] Canonical Tensor Decomposition for Knowledge Base Completion

Timothée Lacroix, Nicolas Usunier+1 more|arXiv (Cornell University)|Jun 19, 2018

Advanced Graph Neural Networks34 references49 citations

TL;DR

The paper re-evaluates CP for knowledge base completion, introducing a semantically invariant reciprocal predicate reformulation and a nuclear 3-norm regularizer, achieving state-of-the-art results on several benchmarks and matching ComplEx when using inverses.

ABSTRACT

The problem of Knowledge Base Completion can be framed as a 3rd-order binary tensor completion problem. In this light, the Canonical Tensor Decomposition (CP) (Hitchcock, 1927) seems like a natural solution; however, current implementations of CP on standard Knowledge Base Completion benchmarks are lagging behind their competitors. In this work, we attempt to understand the limits of CP for knowledge base completion. First, we motivate and test a novel regularizer, based on tensor nuclear $p$-norms. Then, we present a reformulation of the problem that makes it invariant to arbitrary choices in the inclusion of predicates or their reciprocals in the dataset. These two methods combined allow us to beat the current state of the art on several datasets with a CP decomposition, and obtain even better results using the more advanced ComplEx model.

Motivation & Objective

• Motivate and test CP for knowledge base completion against state-of-the-art models.
• Introduce a tensor-norm based regularizer to promote principled regularization.
• Propose a semantic invariant reformulation by modeling predicates and their inverses.
• Evaluate CP and ComplEx under reciprocal and standard settings across multiple datasets.

Proposed method

• Model knowledge bases as a third-order binary tensor and apply CP decomposition.
• Use a full multiclass log-loss for training with data-dependent weighting of factors.
• Introduce a nuclear p-norm based regularizer (especially nuclear 3-norm) as a variational form of tensor norms.
• Reformulate objective by adding inverse predicates, creating a reciprocal tensor to achieve semantic invariance.
• Compare CP and ComplEx under standard and reciprocal settings; implement weighted regularization and SGD optimization.

Experimental results

Research questions

• RQ1 Can CP achieve competitive knowledge base completion performance with appropriately chosen optimization and regularization?
• RQ2 Does a tensor-norm based regularizer improve CP performance on benchmark datasets?
• RQ3 Does modeling inverse predicates (reciprocals) improve link prediction and provide semantic invariance?
• RQ4 How do CP and ComplEx compare under reciprocal formulation across standard benchmarks?
• RQ5 What is the impact of optimization settings (batch size, learning rate, rank) on model performance?

Key findings

• CP can reach state-of-the-art performance when optimized carefully and using reciprocal predicate modeling.
• Modeling inverses yields large gains for CP and ComplEx, e.g., CP on FB15K improves MRR from 0.46 to 0.86 with reciprocals.
• The nuclear 3-norm regularizer provides small but significant improvements on harder datasets, but is not crucial for CP performance.
• Reciprocal formulation substantially boosts CP performance on WN18 and FB15K, bringing CP results close to or matching ComplEx on several datasets.
• CP and ComplEx with inverse predicates achieve higher mean reciprocal ranks across multiple datasets than their standard counterparts.
• Optimization hyper-parameters have a strong impact on final results, sometimes as large as 0.1 MRR.