[Paper Review] Exploring Generative Neural Temporal Point Process

Haitao Lin, Lirong Wu+3 more|arXiv (Cornell University)|Aug 3, 2022

3D Shape Modeling and Analysis4 citations

TL;DR

This paper proposes GNTPP, a unified generative framework for neural temporal point processes that integrates deep generative models like denoising diffusion and score matching to improve predictive performance beyond traditional likelihood-maximization approaches. By revising the attention mechanism with adaptive reweighting based on event type relations and time intervals, GNTPP achieves superior next-event timestamp prediction and sample quality across multiple benchmarks.

ABSTRACT

Temporal point process (TPP) is commonly used to model the asynchronous event sequence featuring occurrence timestamps and revealed by probabilistic models conditioned on historical impacts. While lots of previous works have focused on `goodness-of-fit' of TPP models by maximizing the likelihood, their predictive performance is unsatisfactory, which means the timestamps generated by models are far apart from true observations. Recently, deep generative models such as denoising diffusion and score matching models have achieved great progress in image generating tasks by demonstrating their capability of generating samples of high quality. However, there are no complete and unified works exploring and studying the potential of generative models in the context of event occurence modeling for TPP. In this work, we try to fill the gap by designing a unified extbf{g}enerative framework for extbf{n}eural extbf{t}emporal extbf{p}oint extbf{p}rocess ( extsc{GNTPP}) model to explore their feasibility and effectiveness, and further improve models' predictive performance. Besides, in terms of measuring the historical impacts, we revise the attentive models which summarize influence from historical events with an adaptive reweighting term considering events' type relation and time intervals. Extensive experiments have been conducted to illustrate the improved predictive capability of extsc{GNTPP} with a line of generative probabilistic decoders, and performance gain from the revised attention. To the best of our knowledge, this is the first work that adapts generative models in a complete unified framework and studies their effectiveness in the context of TPP. Our codebase including all the methods given in Section.5.1.1 is open in \url{https://github.com/BIRD-TAO/GNTPP}. We hope the code framework can facilitate future research in Neural TPPs.

Motivation & Objective

• Address the poor predictive performance of existing neural temporal point process (TPP) models despite strong goodness-of-fit.
• Overcome limitations of likelihood-based training that restrict model expressiveness and hinder high-quality sample generation.
• Explore and unify the application of deep generative models—particularly denoising diffusion and score matching—in the context of event sequence modeling.
• Improve historical influence encoding by introducing an adaptive reweighting mechanism in attention layers based on event type similarity and temporal intervals.
• Demonstrate the feasibility and effectiveness of a complete, unified generative framework for TPPs through extensive empirical evaluation.

Proposed method

• Design a unified generative framework, GNTPP, integrating deep generative models (e.g., denoising diffusion) as probabilistic decoders in neural TPPs.
• Replace traditional likelihood-based decoders with unnormalized log-likelihood optimization via energy-based models, enabling flexible and high-quality sample generation.
• Introduce a revised attention mechanism (Rev-ATT) that adaptively reweights historical events using both event type relationships and time intervals.
• Use a learnable matrix $\bm{E}_m\bm{E}_m^T$ to model event type similarity, enhancing the encoder's expressivity in capturing temporal and categorical dependencies.
• Train the model using unnormalized log-likelihood as the objective, enabling end-to-end optimization without requiring tractable likelihoods.
• Implement multiple generative probabilistic decoders (e.g., TCDDM) within the framework to evaluate performance across diverse datasets.

Experimental results

Research questions

• RQ1 Can deep generative models such as denoising diffusion and score matching improve the predictive performance of neural temporal point processes beyond traditional likelihood maximization?
• RQ2 How does the revised attention mechanism with adaptive reweighting based on event type and time interval improve the modeling of historical influences in TPPs?
• RQ3 To what extent does the unified generative framework GNTPP enhance sample quality and predictive accuracy in next-event timestamp forecasting?
• RQ4 What is the impact of hyperparameters such as embedding size and layer number on the performance of the generative decoders in GNTPP?
• RQ5 Does the proposed framework demonstrate consistent improvements across diverse real-world datasets with varying event dynamics?

Key findings

• GNTPP significantly improves predictive performance on next-event timestamp prediction, with the TCDDM-based decoder achieving a MAPE of 4.9947 ±0.0366 on the Stack Overflow dataset, outperforming LSTM and standard attention.
• The revised attention mechanism (Rev-ATT) achieves better performance than standard attention and LSTM, with a Top1 accuracy of 0.6274 ±0.0075 on the Retweet dataset, indicating enhanced modeling of historical dependencies.
• The model shows improved CRPS and QQP_Dev scores across all datasets, with CRPS reduced to 0.4375 ±0.0163 on Stack Overflow, demonstrating better distributional fitting.
• Hyperparameter sensitivity analysis reveals that larger embedding sizes and layer numbers degrade performance on the MOOC dataset, suggesting optimal configuration is critical for generative models.
• Visualization of event type similarity matrices confirms meaningful co-occurrence patterns, such as high similarity between (Stellar Question, Great Answer), validating the model’s ability to learn semantic event relationships.
• The complete codebase is publicly available at https://github.com/BIRD-TAO/GNTPP, supporting reproducibility and future research in neural TPPs.