[Paper Review] D\'ej\`a vu: A Contextualized Temporal Attention Mechanism for Sequential Recommendation

Jibang Wu, Renqin Cai+1 more|arXiv (Cornell University)|Jan 29, 2020

Recommender Systems and Techniques46 references9 citations

TL;DR

This paper proposes CTA, a Contextualized Temporal Attention mechanism that enhances sequential recommendation by dynamically weighting historical actions based on their temporal proximity, contextual relevance, and content similarity. By integrating multiple learnable temporal kernels and context-aware routing, CTA outperforms state-of-the-art models on two large-scale datasets, demonstrating improved accuracy and interpretability in modeling evolving user preferences.

ABSTRACT

Predicting users' preferences based on their sequential behaviors in history is challenging and crucial for modern recommender systems. Most existing sequential recommendation algorithms focus on transitional structure among the sequential actions, but largely ignore the temporal and context information, when modeling the influence of a historical event to current prediction. In this paper, we argue that the influence from the past events on a user's current action should vary over the course of time and under different context. Thus, we propose a Contextualized Temporal Attention Mechanism that learns to weigh historical actions' influence on not only what action it is, but also when and how the action took place. More specifically, to dynamically calibrate the relative input dependence from the self-attention mechanism, we deploy multiple parameterized kernel functions to learn various temporal dynamics, and then use the context information to determine which of these reweighing kernels to follow for each input. In empirical evaluations on two large public recommendation datasets, our model consistently outperformed an extensive set of state-of-the-art sequential recommendation methods.

Motivation & Objective

• To address the limitation of existing sequential recommendation models that overlook the dynamic influence of historical events based on time and context.
• To model how the relevance of past actions to current predictions varies not only over time but also under different user engagement contexts.
• To improve recommendation accuracy by jointly learning content-based, temporal, and contextual importance scores for historical interactions.
• To enhance model interpretability through a three-stage attention weighting mechanism that separates content, temporal, and contextual influences.

Proposed method

• The model employs a three-stage attention mechanism: content-based importance (α), temporal kernel-based reweighting (β), and context-aware fusion (γ) to compute final attention scores.
• Multiple parameterized kernel functions are used to model diverse temporal dynamics, with learnable routing gates selecting the most relevant kernel for each input based on context.
• The temporal kernel component learns non-linear temporal decay patterns, allowing flexible modeling of short-term and long-term influence from historical events.
• Context information is used to gate the selection of temporal kernels, enabling dynamic adaptation to user behavior patterns such as exploration vs. exploitation.
• The final attention score γ combines content, temporal, and contextual signals, allowing the model to prioritize actions that are both relevant and timely.
• The architecture inherits the parallelization benefits of self-attention, maintaining computational efficiency while improving modeling fidelity.

Experimental results

Research questions

• RQ1 How does the influence of historical user actions on current recommendations vary over time and under different behavioral contexts?
• RQ2 Can a learnable, context-aware temporal kernel mechanism improve sequential recommendation beyond fixed or static attention mechanisms?
• RQ3 To what extent can separating content-based, temporal, and contextual importance improve model performance and interpretability?
• RQ4 How do different temporal kernel designs affect model generalization across diverse sequential patterns in real-world datasets?

Key findings

• The proposed CTA model consistently outperformed a wide range of state-of-the-art sequential recommendation methods on two large public datasets, demonstrating superior recommendation accuracy.
• Empirical results show that contextual information plays a dominant role in determining attention weights, especially for distant historical events where temporal decay alone is insufficient.
• The model maintains high computational efficiency due to its self-attention-based design, enabling parallel inference despite the added complexity of contextualized temporal modeling.
• Ablation studies confirm that the self-attention component contributes significantly to performance, but the contextualized temporal mechanism provides a key incremental gain.
• Visualizations of attention scores reveal that the model can correctly identify relevant historical actions even when they are temporally distant, provided the context supports their relevance.
• The three-stage attention mechanism enhances interpretability, allowing researchers to trace why specific items are recommended based on content, timing, and context.