[Paper Review] Multi-behavioral Sequential Prediction with Recurrent Log-bilinear Model

Qiang Liu, Shu Wu+1 more|arXiv (Cornell University)|Aug 25, 2016

Recommender Systems and Techniques38 references3 citations

TL;DR

This paper proposes the Recurrent Log-Bilinear (RLBL) model and its time-aware extension, TA-RLBL, for multi-behavioral sequential prediction, leveraging behavior-specific and position- or time-specific transition matrices within a recurrent architecture to model both short- and long-term user behavior contexts. The models significantly outperform state-of-the-art methods on Movielens, Global Terrorism Database, and Tmall datasets, with RLBL achieving up to 54.94% recall@10 and TA-RLBL showing improved performance over RLBL in most metrics.

ABSTRACT

With the rapid growth of Internet applications, sequential prediction in collaborative filtering has become an emerging and crucial task. Given the behavioral history of a specific user, predicting his or her next choice plays a key role in improving various online services. Meanwhile, there are more and more scenarios with multiple types of behaviors, while existing works mainly study sequences with a single type of behavior. As a widely used approach, Markov chain based models are based on a strong independence assumption. As two classical neural network methods for modeling sequences, recurrent neural networks cannot well model short-term contexts, and the log-bilinear model is not suitable for long-term contexts. In this paper, we propose a Recurrent Log-BiLinear (RLBL) model. It can model multiple types of behaviors in historical sequences with behavior-specific transition matrices. RLBL applies a recurrent structure for modeling long-term contexts. It models several items in each hidden layer and employs position-specific transition matrices for modeling short-term contexts. Moreover, considering continuous time difference in behavioral history is a key factor for dynamic prediction, we further extend RLBL and replace position-specific transition matrices with time-specific transition matrices, and accordingly propose a Time-Aware Recurrent Log-BiLinear (TA-RLBL) model. Experimental results show that the proposed RLBL model and TA-RLBL model yield significant improvements over the competitive compared methods on three datasets, i.e., Movielens-1M dataset, Global Terrorism Database and Tmall dataset with different numbers of behavior types.

Motivation & Objective

• To address the gap in existing collaborative filtering methods that fail to model multiple behavior types in sequential user interactions.
• To overcome the limitations of Markov chains, RNNs, and log-bilinear models in capturing complex temporal dependencies in user behavior sequences.
• To develop a unified framework that models both short-term (position-specific) and long-term (recurrent) dependencies in multi-behavior sequences.
• To incorporate time difference information between user actions to improve dynamic sequential prediction.
• To evaluate the proposed models on real-world datasets with diverse behavior types and sequence lengths.

Proposed method

• The RLBL model uses a recurrent structure to model long-term dependencies in user behavior sequences, with hidden layers processing multiple items simultaneously.
• It employs behavior-specific transition matrices to model different types of user actions (e.g., click, purchase, uninstall) toward items.
• Position-specific transition matrices are used to capture short-term contextual effects from recent behaviors in the sequence.
• The TA-RLBL extension replaces position-specific matrices with time-specific matrices to model the actual time intervals between user actions.
• The models are trained end-to-end using stochastic gradient descent, with embeddings for users and items, and transition matrices parameterized per behavior type.
• Normalization and early stopping are applied to ensure convergence and prevent overfitting.

Experimental results

Research questions

• RQ1 Can a unified model effectively capture both short-term and long-term dependencies in multi-behavioral user sequences?
• RQ2 How do behavior-specific transition matrices improve prediction accuracy compared to single-type behavior modeling?
• RQ3 To what extent does incorporating time differences between actions enhance sequential prediction performance?
• RQ4 How does the proposed RLBL model compare to state-of-the-art methods like FPMC, HRM, and RNN across diverse datasets?
• RQ5 Does the TA-RLBL model achieve faster convergence and better performance than RLBL by modeling temporal dynamics?

Key findings

• On the Movielens-1M dataset, RLBL achieved a recall@10 of 54.94%, significantly outperforming FPMC (33.87%) and RNN (43.50%).
• On the Global Terrorism Database, RLBL achieved a MAP of 0.2811, surpassing RNN (0.2401) and HRM (0.1990).
• TA-RLBL achieved a recall@10 of 54.32% on Movielens, slightly outperforming RLBL (54.94%) in the medium-length window setting.
• The computational time per iteration for RLBL and TA-RLBL was less than twice that of standard RNN, indicating acceptable efficiency despite improved performance.
• Convergence curves show that recall@1 values converge faster than recall@5 and recall@10, indicating that ranking lists of increasing length require more training iterations.
• The models demonstrated consistent improvements across all three datasets—Movielens, GTD, and Tmall—regardless of sequence length or number of behavior types.