[Paper Review] Predicting Sequences of Traversed Nodes in Graphs using Network Models with Multiple Higher Orders

Christoph Gote, Giona Casiraghi+2 more|arXiv (Cornell University)|Jul 13, 2020

Complex Network Analysis Techniques41 references4 citations

TL;DR

This paper introduces MOGen, a novel method for predicting sequences of traversed nodes in graphs by combining multiple higher-order Markov models into a unified generative framework. It outperforms state-of-the-art algorithms in next-element prediction and enables accurate out-of-sample full path prediction across six empirical datasets, with an information-based model selection algorithm that automatically determines the optimal model order without hyperparameters.

ABSTRACT

We propose a novel sequence prediction method for sequential data capturing node traversals in graphs. Our method builds on a statistical modelling framework that combines multiple higher-order network models into a single multi-order model. We develop a technique to fit such multi-order models in empirical sequential data and to select the optimal maximum order. Our framework facilitates both next-element and full sequence prediction given a sequence-prefix of any length. We evaluate our model based on six empirical data sets containing sequences from website navigation as well as public transport systems. The results show that our method out-performs state-of-the-art algorithms for next-element prediction. We further demonstrate the accuracy of our method during out-of-sample sequence prediction and validate that our method can scale to data sets with millions of sequences.

Motivation & Objective

• To address the limitations of existing sequence prediction methods that fail to account for network topology constraints and variable-length paths in sequential data.
• To develop a unified generative model that captures higher-order dependencies in node sequences while respecting the underlying graph structure.
• To introduce an information-theoretic model selection algorithm that determines the optimal maximum order for sequential patterns without manual tuning.
• To enable both next-element prediction and full path prediction from a single trained model, supporting variable-length sequences.

Proposed method

• The method combines Markov chains of multiple orders into a single multi-order generative model that captures sequential patterns constrained by the underlying graph topology.
• It formalizes the model using block adjacency and transition matrices derived from multi-layer network representations, enabling efficient computation of path likelihoods.
• The model likelihood is computed in parallel across paths, significantly accelerating training and enabling scalability to large datasets.
• An information-based model selection procedure using AIC is applied to select the optimal maximum order K, balancing model fit and complexity.
• The framework explicitly models start and end nodes of paths, as well as path length distributions, improving predictive accuracy.
• A parallelized implementation is used to scale the method to datasets with millions of variable-length sequences.

Experimental results

Research questions

• RQ1 Can a unified multi-order model outperform existing methods in predicting the next node in a sequence of traversed nodes within a network?
• RQ2 How can higher-order dependencies in sequential node traversals be effectively modeled while respecting the constraints of an underlying graph topology?
• RQ3 Can an information-theoretic model selection approach automatically identify the optimal maximum order K without requiring manual hyperparameter tuning?
• RQ4 To what extent can the proposed model generalize to out-of-sample path prediction, including full variable-length path sequences?
• RQ5 How does the model scale in terms of training time and memory usage across large-scale real-world datasets?

Key findings

• MOGen significantly outperforms state-of-the-art algorithms in next-element prediction across six empirical datasets, including website navigation and public transport systems.
• The model selection algorithm based on AIC successfully identifies the optimal maximum order K, resulting in models that generalize well without overfitting or underfitting.
• All models selected by MOGen are below 1 MB in size, indicating high compactness and efficiency despite modeling complex sequential patterns.
• Training time scales nearly linearly with the number of processing cores, demonstrating strong parallelization efficiency on large datasets.
• The method successfully enables out-of-sample prediction of full variable-length paths, such as passenger itineraries in real transportation networks.
• The model size increases super-linearly with the maximum order K, but growth is most constrained in sparsely connected networks like the TUBE dataset.