[Paper Review] Tensor Decompositions for temporal knowledge base completion

Timothée Lacroix, Guillaume Obozinski+1 more|arXiv (Cornell University)|Apr 10, 2020

Advanced Graph Neural Networks26 references120 citations

TL;DR

The paper extends tensor factorization for knowledge base completion to temporal data with a temporal ComplEx-based model (TComplEx) and a TNTComplEx variant that includes a non-temporal component, introducing regularization and a large Wikidata-derived dataset for evaluation.

ABSTRACT

Most algorithms for representation learning and link prediction in relational data have been designed for static data. However, the data they are applied to usually evolves with time, such as friend graphs in social networks or user interactions with items in recommender systems. This is also the case for knowledge bases, which contain facts such as (US, has president, B. Obama, [2009-2017]) that are valid only at certain points in time. For the problem of link prediction under temporal constraints, i.e., answering queries such as (US, has president, ?, 2012), we propose a solution inspired by the canonical decomposition of tensors of order 4. We introduce new regularization schemes and present an extension of ComplEx (Trouillon et al., 2016) that achieves state-of-the-art performance. Additionally, we propose a new dataset for knowledge base completion constructed from Wikidata, larger than previous benchmarks by an order of magnitude, as a new reference for evaluating temporal and non-temporal link prediction methods.

Motivation & Objective

• Motivate temporal link prediction within knowledge bases and address how facts have temporal validity.
• Extend tensor decomposition approaches to order-4 tensors (subject, predicate, object, timestamp) for temporal KB completion.
• Introduce regularization and a non-temporal component to handle heterogeneous temporal data.
• Provide a scalable large-scale dataset derived from Wikidata for benchmarking temporal and non-temporal KB completion methods.

Proposed method

• Extend ComplEx to a temporal setting by adding a timestamp factor: X_hat(U,V,T) = Re([U, V, conj(U), T]).
• Introduce a non-temporal component to handle predicates that are not time-dependent, leading to TNTComplEx.
• Regularize with variational forms of tensor nuclear p-norms (p=3,4) and a smoothness term on temporal embeddings.
• Unfold tensors to derive regularization schemes that weight modes according to marginal statistics and accommodate sampling.
• Train with a cross-entropy loss over observed triples and sampled negatives, including reciprocal relations to stabilize learning.
• Present a large-scale Wikidata-derived dataset with 432,715 entities, 407 predicates, and 1,724 timestamps for training 7M temporal and non-temporal triples.

Experimental results

Research questions

• RQ1 Can a temporal extension of ComplEx (TComplEx) effectively model time-dependent relations in knowledge bases?
• RQ2 Does adding a non-temporal component (TNTComplEx) improve performance on heterogeneous temporal knowledge bases?
• RQ3 What regularization strategies (nuclear p-norm variational forms and temporal smoothness) improve temporal KB completion performance?
• RQ4 How does the proposed approach scale to a large, web-scale temporal KB like Wikidata for both temporal and non-temporal predicates?

Key findings

• TComplEx and TNTComplEx achieve state-of-the-art performance with a comparable number of parameters to baselines on ICEWS14, ICEWS05-15, and Yago15k.
• Regularization with nuclear p-norm variational forms and temporal smoothness substantially improves MRR, with up to 0.07 absolute gains on certain datasets.
• A large Wikidata-based dataset (7M training triples, 432K entities, 407 predicates, 1.7K timestamps) provides a scalable benchmark for temporal and non-temporal KB completion.
• TNTComplEx shows competitive temporal performance while maintaining strong non-temporal performance, justifying the shared-parameter, mixed-temporal approach.
• Training efficiency is comparable to static models for similar parameter counts, despite additional temporal modeling.