[Paper Review] Exploration in two-stage recommender systems

Jiri Hron, Karl Krauth+2 more|arXiv (Cornell University)|Sep 1, 2020

Advanced Bandit Algorithms Research7 references4 citations

TL;DR

This paper identifies that naive deployment of LinUCB in two-stage recommender systems causes linear regret due to mismatches in feature embeddings and data regularization between nominators and the ranker. It proposes a simple synchronization method that aligns posterior statistics (mean and covariance) between stages using only existing LinUCB computations, reducing regret and enabling near-optimal performance comparable to single-stage LinUCB.

ABSTRACT

Two-stage recommender systems are widely adopted in industry due to their scalability and maintainability. These systems produce recommendations in two steps: (i) multiple nominators preselect a small number of items from a large pool using cheap-to-compute item embeddings; (ii) with a richer set of features, a ranker rearranges the nominated items and serves them to the user. A key challenge of this setup is that optimal performance of each stage in isolation does not imply optimal global performance. In response to this issue, Ma et al. (2020) proposed a nominator training objective importance weighted by the ranker's probability of recommending each item. In this work, we focus on the complementary issue of exploration. Modeled as a contextual bandit problem, we find LinUCB (a near optimal exploration strategy for single-stage systems) may lead to linear regret when deployed in two-stage recommenders. We therefore propose a method of synchronising the exploration strategies between the ranker and the nominators. Our algorithm only relies on quantities already computed by standard LinUCB at each stage and can be implemented in three lines of additional code. We end by demonstrating the effectiveness of our algorithm experimentally.

Motivation & Objective

• To investigate why standard LinUCB exploration strategies fail in two-stage recommender systems.
• To identify the root causes of suboptimal performance, particularly mismatched feature mappings and data regularization between nominators and the ranker.
• To propose a minimal-complexity solution that synchronizes inferred statistics between stages to restore near-optimal exploration.
• To empirically validate that synchronization mitigates linear regret and improves regret performance in simulated environments.
• To extend insights beyond LinUCB to other 'optimism in the face of uncertainty' algorithms by highlighting the role of statistical consistency across stages.

Proposed method

• The method introduces a synchronization step that updates the nominator's posterior distribution using the ranker's uncertainty estimates, specifically matching the inverse covariance matrices and means.
• The synchronization is performed using a KL divergence-based update that penalizes overdispersion, ensuring the nominator's uncertainty does not exceed the ranker's.
• The algorithm uses only quantities already computed during LinUCB selection and update steps, requiring only three lines of additional code.
• Two variants are explored: synchronization after the standard update (post-update) and before (pre-update), both shown to perform similarly in experiments.
• The core equation (Equation 7) ensures that after each nomination, the nominator's posterior matches the ranker’s for that item, preventing divergence in uncertainty estimates.
• The method is designed to be agnostic to the specific LinUCB implementation and can be applied to any two-stage system using Bayesian linear bandits.

Experimental results

Research questions

• RQ1 Why does LinUCB exhibit linear regret when deployed naively in two-stage recommender systems?
• RQ2 How do mismatches in feature representations and training data volume between nominators and the ranker affect exploration performance?
• RQ3 Can synchronizing posterior statistics between stages restore near-optimal regret performance in two-stage LinUCB?
• RQ4 What is the computational overhead of such synchronization, and can it be implemented with minimal code extension?
• RQ5 Does the synchronization method generalize to other 'optimism in the face of uncertainty' exploration algorithms?

Key findings

• Naive two-stage LinUCB deployment leads to linear regret due to divergent uncertainty estimates between nominators and the ranker.
• The primary causes of regret are mismatched feature embeddings and differing regularization levels (i.e., amount of training data) between the two stages.
• Synchronization of posterior statistics—specifically mean and covariance—between the ranker and nominators effectively eliminates the regret gap.
• The proposed algorithm achieves regret performance comparable to single-stage LinUCB, which is known to be near-optimal.
• The pre-update and post-update synchronization variants perform nearly identically in experiments, with minimal computational overhead.
• The method is robust across varying levels of ranker pretraining (γ), with higher γ leading to better performance for synchronized systems.