[Paper Review] Exploration vs. Exploitation in the Information Filtering Problem

Xiaoting Zhao, Peter I. Frazier|arXiv (Cornell University)|Jul 30, 2014

Advanced Bandit Algorithms Research42 references3 citations

TL;DR

This paper proposes a Bayesian sequential decision-making model for information filtering that optimally balances exploration and exploitation, especially in cold-start scenarios with limited user feedback. By decomposing the problem into tractable two-armed bandit subproblems, the method outperforms pure exploitation by forwarding additional relevant items when uncertainty is high, with exploration diminishing as feedback accumulates.

ABSTRACT

We consider information filtering, in which we face a stream of items too voluminous to process by hand (e.g., scientific articles, blog posts, emails), and must rely on a computer system to automatically filter out irrelevant items. Such systems face the exploration vs. exploitation tradeoff, in which it may be beneficial to present an item despite a low probability of relevance, just to learn about future items with similar content. We present a Bayesian sequential decision-making model of this problem, show how it may be solved to optimality using a decomposition to a collection of two-armed bandit problems, and show structural results for the optimal policy. We show that the resulting method is especially useful when facing the cold start problem, i.e., when filtering items for new users without a long history of past interactions. We then present an application of this information filtering method to a historical dataset from the arXiv.org repository of scientific articles.

Motivation & Objective

• Address the cold start problem in information filtering, where new users lack sufficient historical interaction data for effective relevance prediction.
• Model the exploration vs. exploitation tradeoff in information filtering as a stochastic control problem using Bayesian statistics and dynamic programming.
• Develop an efficient, optimal policy that balances learning from uncertain items (exploration) with forwarding high-confidence relevant items (exploitation).
• Provide a scalable solution to the partially observable Markov decision process (POMDP) underlying information filtering, overcoming the curse of dimensionality.
• Demonstrate the method's effectiveness on real-world data from arXiv.org, showing significant gains over benchmark policies.

Proposed method

• Formulate the information filtering problem as a Bayesian sequential decision-making model with a belief state over item relevance, parameterized by beta distributions.
• Decompose the high-dimensional POMDP into a collection of smaller, independent two-armed bandit problems—one known arm (discard) and one unknown Bernoulli arm (forward), with state transitions based on feedback.
• Use dynamic programming to solve each two-armed bandit subproblem optimally, leveraging known results from the Gittins index literature.
• Derive upper and lower bounds on the value function that converge as the planning horizon increases, ensuring convergence to the optimal policy.
• Implement the optimal policy using recursive value function updates that depend on posterior mean relevance and a discount factor γx.
• Apply the method to a real-world dataset from arXiv.org, where it dynamically adjusts exploration based on the number of feedback-observed items.

Experimental results

Research questions

• RQ1 How can an information filtering system optimally balance exploration (forwarding uncertain items to learn) and exploitation (forwarding high-confidence relevant items) in the absence of historical data?
• RQ2 What structural properties does the optimal policy for information filtering exhibit, particularly in terms of exploration intensity as feedback accumulates?
• RQ3 Can the optimal policy be computed efficiently despite the curse of dimensionality in the full POMDP formulation?
• RQ4 How does the proposed method compare to myopic strategies and Thompson sampling in terms of cumulative reward (i.e., number of relevant items forwarded) in cold-start regimes?
• RQ5 What is the impact of exploration on long-term filtering performance when user relevance is initially unknown?

Key findings

• The optimal policy forwards every item that a myopic strategy would forward, and potentially additional items that the myopic strategy would discard, especially when feedback is scarce.
• Exploration is most aggressive when few items have been observed, and diminishes as the number of feedback-observed items increases, vanishing entirely in the limit of infinite feedback.
• The value function bounds converge to the true value function as the planning horizon M approaches infinity, with the difference between upper and lower bounds decaying geometrically as γx^M / (1 - γx).
• The method achieves optimal performance by solving a decomposition into two-armed bandit problems, each solvable via known Gittins index methods, enabling efficient computation.
• Empirical results on arXiv.org show that the optimal exploration strategy significantly outperforms both pure exploitation and Thompson sampling in terms of cumulative relevance detection.
• The optimal policy’s willingness to explore is highest in cold-start scenarios and systematically reduces as confidence in relevance estimates increases, reflecting a principled adaptation to uncertainty.