[Paper Review] Recurrent Event Network: Autoregressive Structure Inference over Temporal Knowledge Graphs

Woojeong Jin, Meng Qu+2 more|arXiv (Cornell University)|Apr 11, 2019

Advanced Graph Neural NetworksComputer Science44 references36 citations

TL;DR

RE-Net is an autoregressive model for temporal knowledge graphs that jointly models global history and local neighborhood to predict future multi-hop, multi-relational events. It enables multi-step extrapolation with competitive accuracy.

ABSTRACT

Knowledge graph reasoning is a critical task in natural language processing. The task becomes more challenging on temporal knowledge graphs, where each fact is associated with a timestamp. Most existing methods focus on reasoning at past timestamps and they are not able to predict facts happening in the future. This paper proposes Recurrent Event Network (RE-NET), a novel autoregressive architecture for predicting future interactions. The occurrence of a fact (event) is modeled as a probability distribution conditioned on temporal sequences of past knowledge graphs. Specifically, our RE-NET employs a recurrent event encoder to encode past facts and uses a neighborhood aggregator to model the connection of facts at the same timestamp. Future facts can then be inferred in a sequential manner based on the two modules. We evaluate our proposed method via link prediction at future times on five public datasets. Through extensive experiments, we demonstrate the strength of RENET, especially on multi-step inference over future timestamps, and achieve state-of-the-art performance on all five datasets. Code and data can be found at https://github.com/INK-USC/RE-Net.

Motivation & Objective

• Motivate extrapolation in temporal knowledge graphs where facts have timestamps and future events must be forecasted.
• Propose an autoregressive framework that models the joint distribution of future events conditioned on past graph sequences.
• Introduce a recurrent event encoder for temporal dependencies and a neighborhood aggregator for local graph structure.
• Demonstrate state-of-the-art performance on five public temporal knowledge graph datasets for multi-step future event prediction.

Proposed method

• Model future events as a joint distribution p(Gt|Gt−m:t−1) with an autoregressive factorization over time and events within each timestamp.
• Use a recurrent event encoder to produce a global state Ht and local representations ht(s) and ht(s,r) from past graphs.
• Employ neighborhood aggregators (mean, attentive, and multi-relational GCN) to capture local graph structure around subjects and relations.
• Define p(o t|s t,r t,Gt−m:t−1) and p(r t|s t,Gt−m:t−1) and p(s t|Gt−m:t−1) via an MLP decoder that consumes static embeddings and dynamic local/global features.
• Train with a cross-entropy style loss over predicted objects, relations, and subjects with optional weighting (λ1, λ2).
• Enable multi-step inference by sequentially sampling intermediate graphs Gt+1:Δt−1 and aggregating to estimate p(Gt+Δt|Gt).

Experimental results

Research questions

• RQ1 Can RE-Net accurately forecast unseen future events in multi-step extrapolation on temporal knowledge graphs?
• RQ2 Does incorporating both global history and local neighborhood information improve future event prediction over baselines?
• RQ3 How do different neighborhood aggregators (mean, attention, RGCN) affect predictive performance in multi-relational, temporally evolving graphs?
• RQ4 Is multi-step inference capable of producing competitive long-horizon predictions without ground-truth future events?

Key findings

• RE-Net achieves state-of-the-art results across five datasets for extrapolation link prediction.
• Multi-step inference with RE-Net consistently improves prediction accuracy for distant future timestamps.
• Multi-relational graph aggregation (RGCN) and attentive neighborhood pooling yield the best performance among aggregators.
• RE-Net substantially outperforms static baselines and temporal reasoning models on event-based and public KGs.
• Ablation shows removing the aggregator or multi-step inference degrades performance, underscoring the importance of both components.