[Paper Review] Low-rank Bandits with Latent Mixtures

Aditya Gopalan, Odalric-Ambrym Maillard+1 more|arXiv (Cornell University)|Sep 6, 2016

Advanced Bandit Algorithms Research14 references15 citations

TL;DR

This paper proposes a novel bandit algorithm for recommender systems with two-sided uncertainty—unknown user preferences and item rewards—by modeling users as latent mixtures of C representative classes. It combines OFUL with a robust tensor power method to achieve a regret bound of $\tilde{O}(C\sqrt{BT})$ after T interactions, where B is the number of users, under partial feedback and low-rank reward structure.

ABSTRACT

We study the task of maximizing rewards from recommending items (actions) to users sequentially interacting with a recommender system. Users are modeled as latent mixtures of C many representative user classes, where each class specifies a mean reward profile across actions. Both the user features (mixture distribution over classes) and the item features (mean reward vector per class) are unknown a priori. The user identity is the only contextual information available to the learner while interacting. This induces a low-rank structure on the matrix of expected rewards r a,b from recommending item a to user b. The problem reduces to the well-known linear bandit when either user or item-side features are perfectly known. In the setting where each user, with its stochastically sampled taste profile, interacts only for a small number of sessions, we develop a bandit algorithm for the two-sided uncertainty. It combines the Robust Tensor Power Method of Anandkumar et al. (2014b) with the OFUL linear bandit algorithm of Abbasi-Yadkori et al. (2011). We provide the first rigorous regret analysis of this combination, showing that its regret after T user interactions is $\ ilde O(C\\sqrt{BT})$, with B the number of users. An ingredient towards this result is a novel robustness property of OFUL, of independent interest.

Motivation & Objective

• Address the challenge of online recommendation with both user and item features unknown a priori, under limited user interactions.
• Model users as stochastic mixtures over C latent classes, each with a distinct reward profile across items, forming a low-rank reward matrix.
• Develop a bandit algorithm that simultaneously learns user class mixtures and item reward profiles from partial, bandit-style feedback.
• Provide the first rigorous regret analysis for a bandit algorithm in this two-sided uncertainty setting with latent mixture models.
• Establish robustness of OFUL under estimated covariance matrices derived from noisy, partial observations.

Proposed method

• Integrate the OFUL linear bandit algorithm with a variant of the Robust Tensor Power (RTP) method to handle low-rank structure and partial feedback.
• Use an importance sampling scheme to estimate second- and third-order moments of rewards from single-action feedback in a bandit setting.
• Apply a subroutine (Algorithm 1) to recover item-side features (latent reward vectors per class) from noisy, partial observations.
• Maintain a confidence ellipsoid over user preference vectors using OFUL, with regularization to ensure invertibility of the covariance matrix.
• Ensure robustness of OFUL by proving that the estimated covariance matrix remains well-conditioned under estimation errors in moment tensors.
• Use high-probability concentration bounds and matrix perturbation theory to control the error in estimated user and item features.

Experimental results

Research questions

• RQ1 Can a bandit algorithm achieve low regret in a recommender system where both user preferences and item reward profiles are latent and unknown?
• RQ2 How can low-rank structure in the reward matrix be exploited when only partial feedback (one reward per interaction) is available?
• RQ3 What is the regret performance of a hybrid algorithm combining OFUL and tensor-based estimation under two-sided uncertainty?
• RQ4 Can the OFUL algorithm maintain regret guarantees when the covariance matrix is estimated from noisy, bandit-level observations?
• RQ5 What conditions ensure the invertibility and stability of the estimated covariance matrix in the presence of estimation errors?

Key findings

• The proposed algorithm achieves a regret bound of $\tilde{O}(C\sqrt{BT})$ after T user interactions, where B is the number of users and C is the number of latent user classes.
• The regret bound holds under the assumption that each user interacts only a small number of times, with no prior knowledge of user or item features.
• A novel robustness property of OFUL is established, showing that it maintains regret guarantees even when the covariance matrix is estimated with error.
• The recovery of latent item reward vectors via the RTP method is shown to be consistent under the proposed importance sampling scheme.
• The algorithm's performance is guaranteed with high probability $1 - \delta$, provided that the number of initial interactions $n_0$ satisfies a specific lower bound involving $A$, $C$, $\delta$, and spectral properties of the true reward matrix.
• The analysis establishes that the estimated covariance matrix remains invertible with high probability when the number of observations exceeds a threshold tied to the condition number and spectral gap of the true matrix.