[Paper Review] Deep Reinforcement Learning based Recommendation with Explicit User-Item Interactions Modeling

Feng Liu, Ruiming Tang+6 more|arXiv (Cornell University)|Oct 29, 2018

Recommender Systems and Techniques38 references82 citations

TL;DR

This paper frames recommendation as a sequential decision process and uses an Actor-Critic DRL framework (DRR) with a state representation module that explicitly models user-item interactions to optimize both immediate and long-term rewards.

ABSTRACT

Recommendation is crucial in both academia and industry, and various techniques are proposed such as content-based collaborative filtering, matrix factorization, logistic regression, factorization machines, neural networks and multi-armed bandits. However, most of the previous studies suffer from two limitations: (1) considering the recommendation as a static procedure and ignoring the dynamic interactive nature between users and the recommender systems, (2) focusing on the immediate feedback of recommended items and neglecting the long-term rewards. To address the two limitations, in this paper we propose a novel recommendation framework based on deep reinforcement learning, called DRR. The DRR framework treats recommendation as a sequential decision making procedure and adopts an "Actor-Critic" reinforcement learning scheme to model the interactions between the users and recommender systems, which can consider both the dynamic adaptation and long-term rewards. Furthermore, a state representation module is incorporated into DRR, which can explicitly capture the interactions between items and users. Three instantiation structures are developed. Extensive experiments on four real-world datasets are conducted under both the offline and online evaluation settings. The experimental results demonstrate the proposed DRR method indeed outperforms the state-of-the-art competitors.

Motivation & Objective

• Motivate modeling recommendation as a dynamic, sequential decision process rather than a static one.
• Incorporate long-term rewards into recommendation beyond immediate feedback.
• Develop an explicit state representation module to capture user-item interactions.
• Propose three instantiations of state representations to model interactions effectively.
• Demonstrate superiority of DRR over state-of-the-art baselines on real-world datasets.

Proposed method

• Model recommendation as an MDP with states as user history and actions as continuous ranking vectors.
• Use an Actor-Critic framework (DDPG) where the Actor outputs a ranking parameter to score items.
• Employ a Critic network to estimate Q-values for state-action pairs and guide Actor updates.
• Introduce a state representation module that explicitly captures user-item interactions.
• Provide three DRR state structures (DRR-p, DRR-u, DRR-ave) to model pairwise and user-item interactions.
• Train with experience replay and target networks; use offline logs or environment simulators for evaluation.

Experimental results

Research questions

• RQ1 Can DRR effectively model dynamic, sequential user preferences for recommendations?
• RQ2 Does explicitly modeling user-item interactions in the state representation improve performance over baselines?
• RQ3 How do different DRR state structures (DRR-p, DRR-u, DRR-ave) compare in capturing interactions?
• RQ4 Do DRR methods outperform traditional recommender systems and existing RL-based approaches on real-world datasets?
• RQ5 Can offline and simulator-based online evaluations reveal the benefits of long-term rewards in recommendations?

Key findings

• DRR outperforms state-of-the-art baselines in offline evaluations (Precision@k, NDCG@k) and simulated online rewards.
• Three DRR state structures (DRR-p, DRR-u, DRR-ave) explicitly model user-item interactions with varying emphasis on item ordering and user influence.
• The framework uses an Actor-Critic (DDPG) approach with prioritized experience replay and target networks.
• The state representation module learns a summarized user state from recent positive interactions and item embeddings to generate ranking actions.
• Evaluation on four real-world datasets demonstrates the proposed method yields superior performance to competing methods.