[Paper Review] On Context-Dependent Clustering of Bandits

Claudio Gentile, Shuai Li+4 more|arXiv (Cornell University)|Aug 6, 2016

Advanced Bandit Algorithms ResearchDecision Sciences8 citations

TL;DR

This paper proposes CAB (Context-Aware Bandit), a novel contextual bandit algorithm that dynamically clusters users based on item-specific contexts to enable context-dependent feedback sharing. By adapting clusterings per item and integrating collaborative effects into exploration and exploitation, CAB achieves significantly lower regret and higher prediction performance than state-of-the-art methods, especially in collaborative environments with sparse or evolving user data.

ABSTRACT

We investigate a novel cluster-of-bandit algorithm CAB for collaborative recommendation tasks that implements the underlying feedback sharing mechanism by estimating the neighborhood of users in a context-dependent manner. CAB makes sharp departures from the state of the art by incorporating collaborative effects into inference as well as learning processes in a manner that seamlessly interleaving explore-exploit tradeoffs and collaborative steps. We prove regret bounds under various assumptions on the data, which exhibit a crisp dependence on the expected number of clusters over the users, a natural measure of the statistical difficulty of the learning task. Experiments on production and real-world datasets show that CAB offers significantly increased prediction performance against a representative pool of state-of-the-art methods.

Motivation & Objective

• To address the challenge of personalized recommendation in dynamic, evolving environments with new users and items.
• To model user behavior similarity not as static groups but as context-dependent clusters that vary per item.
• To design a scalable, flexible algorithm that seamlessly integrates collaborative feedback sharing into the explore-exploit tradeoff.
• To theoretically analyze regret bounds that depend on the expected number of context-dependent clusters, reflecting the statistical difficulty of the learning task.
• To empirically validate CAB's superiority over state-of-the-art contextual bandit algorithms on production and real-world datasets.

Proposed method

• CAB uses a linear contextual bandit framework where each item's feature vector x induces a distinct clustering of users into m(x) groups based on similar responses to x.
• User clusters are estimated from feedback data using a regularized least-squares method to learn user model vectors.
• Feedback is shared within clusters, enabling collaborative learning where users in the same cluster inform each other’s policy decisions.
• The algorithm dynamically adapts to user inclusions/exclusions and evolving content, maintaining scalability and responsiveness.
• Regret bounds are derived based on the expected number of clusters over users, with tighter bounds under sparsity assumptions on user model vectors.
• CAB integrates collaborative effects directly into the inference and learning processes, blending exploration and exploitation through context-aware neighborhood sharing.

Experimental results

Research questions

• RQ1 How can collaborative effects be effectively modeled in contextual bandits when user preferences vary with item context?
• RQ2 What is the theoretical impact of context-dependent clustering on regret bounds in personalized recommendation?
• RQ3 How does CAB’s performance compare to state-of-the-art contextual bandit algorithms that use static or context-independent clustering?
• RQ4 In what settings does context-dependent feedback sharing provide a significant performance advantage over non-collaborative or context-independent methods?
• RQ5 Can CAB maintain low regret and high prediction accuracy in cold-start and dynamic environments with evolving user bases?

Key findings

• CAB achieved nearly double the Click-Through Rate (CTR) compared to baselines on the Tuenti dataset, demonstrating strong performance gains in collaborative settings.
• On the KDD Cup and Avazu datasets, CAB maintained a consistent performance lead throughout the time horizon, especially in the cold-start phase.
• On the Delicious dataset, where collaborative signals were weak, LinUCB-MULTIPLE outperformed CAB, confirming that CAB’s gains are driven by effective collaboration.
• Visual analysis of user model distances showed that user representations on Delicious were more separated than on LastFM, explaining why clustering-based methods like CAB underperformed there.
• The regret bounds for CAB depend cleanly on the expected number of context-dependent clusters, providing a natural measure of learning difficulty.
• In sparse user model scenarios, CAB achieved improved regret bounds, indicating robustness to data sparsity.