[Paper Review] HYPRO: A Hybridly Normalized Probabilistic Model for Long-Horizon Prediction of Event Sequences

Siqiao Xue, Xiaoming Shi+2 more|arXiv (Cornell University)|Oct 4, 2022

Time Series Analysis and Forecasting4 citations

TL;DR

HYPRO is a hybridly normalized probabilistic model that combines an autoregressive base model with an energy function to improve long-horizon event sequence prediction. By reweighting autoregressively generated sequences based on global realism, it reduces error cascading and achieves state-of-the-art performance on real-world datasets, outperforming autoregressive and non-autoregressive baselines.

ABSTRACT

In this paper, we tackle the important yet under-investigated problem of making long-horizon prediction of event sequences. Existing state-of-the-art models do not perform well at this task due to their autoregressive structure. We propose HYPRO, a hybridly normalized probabilistic model that naturally fits this task: its first part is an autoregressive base model that learns to propose predictions; its second part is an energy function that learns to reweight the proposals such that more realistic predictions end up with higher probabilities. We also propose efficient training and inference algorithms for this model. Experiments on multiple real-world datasets demonstrate that our proposed HYPRO model can significantly outperform previous models at making long-horizon predictions of future events. We also conduct a range of ablation studies to investigate the effectiveness of each component of our proposed methods.

Motivation & Objective

• To address the challenge of long-horizon prediction in event sequences, where autoregressive models suffer from error cascading due to sequential dependency.
• To develop a probabilistic model that improves prediction quality by reweighting candidate sequences based on their global plausibility, not just local likelihood.
• To design efficient training and inference algorithms that avoid computing intractable normalizing constants in energy-based models.
• To empirically validate the effectiveness of the proposed framework across diverse real-world datasets, including e-commerce and transportation.

Proposed method

• HYPRO combines an autoregressive base model (e.g., continuous-time Transformer) that generates candidate future event sequences.
• An energy function evaluates the global realism of each complete sequence (past + predicted future) and assigns higher scores to more plausible sequences.
• The model uses noise-contrastive estimation (NCE) with binary and multi-class variants to train the energy function without computing the intractable normalizing constant.
• A normalized importance sampling inference algorithm efficiently draws predictions from the final hybrid model distribution.
• A regularization term is introduced in the loss to encourage better separation between real and negative sequences, especially in Binary-NCE training.
• The framework is modular, allowing any autoregressive model as the base and different architectures (e.g., Transformer or LSTM) as the energy function.

Experimental results

Research questions

• RQ1 Can a hybrid model combining autoregressive generation with energy-based reweighting improve long-horizon event sequence prediction compared to purely autoregressive models?
• RQ2 Does the use of an energy function that evaluates complete sequences enhance prediction quality by correcting local errors through global consistency?
• RQ3 How effective are noise-contrastive estimation-based training objectives in learning the energy function without explicit normalization?
• RQ4 What is the impact of different energy function architectures (e.g., Transformer vs. LSTM) on model performance?
• RQ5 How does the number of negative samples and regularization affect training stability and inference quality?

Key findings

• HYPRO significantly outperforms the autoregressive baseline AttNHP across all tested prediction horizons on the Taobao dataset, with consistent improvements as the horizon increases.
• On the Taobao dataset, HYPRO with a continuous-time Transformer energy function achieved the best performance, outperforming both the autoregressive baseline and a non-autoregressive model.
• The use of distance regularization in Binary-NCE training led to statistically significant performance gains (p < 0.05), while the effect was not significant in Multi-NCE (p ≈ 0.1).
• Increasing the number of negative samples improved training and inference up to a point: performance plateaued with 10 negative samples in training and 20 in inference.
• The continuous-time Transformer energy function consistently outperformed the LSTM-based energy function across all experiments, indicating superior contextual modeling.
• Normalized importance sampling enabled efficient inference from the full HYPRO model, making long-horizon prediction tractable and scalable.