[Paper Review] Nearly Dimension-Independent Sparse Linear Bandit over Small Action Spaces via Best Subset Selection

Yi Chen, Yining Wang+3 more|arXiv (Cornell University)|Sep 4, 2020

Advanced Bandit Algorithms Research60 references4 citations

TL;DR

This paper proposes a nearly dimension-independent sparse linear bandit algorithm using best subset selection and doubly growing epochs to achieve $ tilde{O}(s ilde{O}(s\sqrt{T})$ regret in high-dimensional, sparse linear contextual bandits with small, random action spaces. The method ensures near-optimality by balancing exploration and exploitation while maintaining computational feasibility and matching minimax lower bounds in low-dimensional regimes.

ABSTRACT

We consider the stochastic contextual bandit problem under the high dimensional linear model. We focus on the case where the action space is finite and random, with each action associated with a randomly generated contextual covariate. This setting finds essential applications such as personalized recommendations, online advertisements, and personalized medicine. However, it is very challenging to balance the exploration and exploitation tradeoff. We modify the LinUCB algorithm in doubly growing epochs and estimate the parameter using the best subset selection method, which is easy to implement in practice. This approach achieves O(sT) regret with high probability, which is nearly independent of the “ambient” regression model dimension <i>d</i>. We further attain a sharper O(sT) regret by using the SupLinUCB framework and match the minimax lower bound of the low-dimensional linear stochastic bandit problem. Finally, we conduct extensive numerical experiments to empirically demonstrate our algorithms’ applicability and robustness. Supplementary materials for this article are available online.

Motivation & Objective

• To address the challenge of high-dimensional linear contextual bandits where the ambient dimension $d$ is much larger than the sample size $T$, while maintaining computational efficiency.
• To develop a practical and implementable algorithm that achieves regret nearly independent of the ambient dimension $d$ in sparse high-dimensional settings.
• To balance exploration and exploitation effectively in finite, small, and randomly generated action spaces.
• To match the minimax lower bound of low-dimensional linear bandit problems by refining the SupLinUCB framework with best subset selection.
• To empirically validate the robustness and applicability of the proposed algorithm across diverse scenarios.

Proposed method

• The algorithm employs doubly growing epochs to systematically balance exploration and exploitation across time intervals of increasing size.
• It uses best subset selection to estimate the sparse parameter vector $\theta^*$, focusing only on the $s$ non-zero components, which reduces estimation error and improves interpretability.
• A ridge regression estimator with a confidence region is used within each epoch to bound estimation error, leveraging self-normalized martingale concentration inequalities.
• The confidence region is constructed using the empirical Gram matrix and regularization to ensure high-probability bounds on parameter estimation error.
• The SupLinUCB framework is adapted by incorporating best subset selection to refine the confidence set and achieve tighter regret bounds.
• The algorithm dynamically updates the support set $S_{\tau-1}$ of the estimated parameter at each epoch, ensuring sparsity-aware learning.

Experimental results

Research questions

• RQ1 Can a sparse linear bandit algorithm achieve regret nearly independent of the ambient dimension $d$ in high-dimensional settings?
• RQ2 How can best subset selection be effectively integrated into a contextual bandit framework to maintain computational feasibility and statistical efficiency?
• RQ3 What is the optimal trade-off between exploration and exploitation in finite, small, and randomly generated action spaces?
• RQ4 Can the proposed method achieve regret bounds matching the minimax lower bound of low-dimensional linear bandit problems?
• RQ5 How does the algorithm perform empirically under varying levels of sparsity and noise?

Key findings

• The proposed algorithm achieves a regret bound of $\widetilde{\mathcal{O}}(s\sqrt{T})$ with high probability, which is nearly independent of the ambient dimension $d$.
• By combining best subset selection with the SupLinUCB framework, the method achieves a sharper regret bound of $\widetilde{\mathcal{O}}(\sqrt{sT})$, matching the minimax lower bound for low-dimensional linear bandits.
• The algorithm maintains computational feasibility through best subset selection and epoch-wise estimation, avoiding the intractability of full sparse regression in high dimensions.
• Theoretical analysis confirms that the confidence region for the parameter estimator is tightly controlled using self-normalized martingale concentration and ridge regularization.
• Numerical experiments demonstrate the algorithm's robustness and strong empirical performance across various high-dimensional and sparse settings.
• The method effectively handles the curse of dimensionality by exploiting sparsity and leveraging finite, small action spaces to reduce regret growth.