[Paper Review] Learning Neural Contextual Bandits Through Perturbed Rewards

Yiling Jia, Weitong Zhang+3 more|arXiv (Cornell University)|Jan 24, 2022

Advanced Bandit Algorithms Research4 citations

TL;DR

This paper proposes a computationally efficient neural contextual bandit algorithm that eliminates explicit exploration by injecting Gaussian noise into rewards during model updates. By leveraging perturbed rewards, the method achieves a regret bound of $\tilde{\mathbf{O}}(\widetilde{d}\sqrt{T})$ without incurring additional computational costs, outperforming existing neural bandit methods in both efficiency and empirical performance on real-world datasets.

ABSTRACT

Thanks to the power of representation learning, neural contextual bandit algorithms demonstrate remarkable performance improvement against their classical counterparts. But because their exploration has to be performed in the entire neural network parameter space to obtain nearly optimal regret, the resulting computational cost is prohibitively high. We perturb the rewards when updating the neural network to eliminate the need of explicit exploration and the corresponding computational overhead. We prove that a $ ilde{O}( ilde{d}\sqrt{T})$ regret upper bound is still achievable under standard regularity conditions, where $T$ is the number of rounds of interactions and $ ilde{d}$ is the effective dimension of a neural tangent kernel matrix. Extensive comparisons with several benchmark contextual bandit algorithms, including two recent neural contextual bandit models, demonstrate the effectiveness and computational efficiency of our proposed neural bandit algorithm.

Motivation & Objective

• To address the high computational cost of explicit exploration in neural contextual bandit algorithms.
• To eliminate the need for expensive covariance matrix inversion in confidence set construction for deep neural networks.
• To maintain theoretical regret guarantees while significantly reducing computational overhead.
• To enable practical deployment of powerful neural bandit models in online learning settings.
• To bridge the gap between theoretical regret bounds and empirical performance in neural bandit algorithms.

Proposed method

• Inject zero-mean Gaussian noise into observed rewards during each model update to induce implicit exploration.
• Use the highest estimated reward arm after perturbation to avoid explicit exploration in parameter space.
• Leverage the neural tangent kernel (NTK) framework to analyze the effective dimension $\widetilde{d}$ of the model.
• Prove that the perturbation-based method achieves $\tilde{\mathbf{O}}(\widetilde{d}\sqrt{T})$ regret under standard regularity conditions.
• Apply stochastic gradient descent for model updates, avoiding full-batch matrix inversions.
• Use diagonal approximations only when necessary, avoiding theoretical justification gaps in prior work.

Experimental results

Research questions

• RQ1 Can reward perturbation replace explicit exploration in neural contextual bandits without sacrificing regret guarantees?
• RQ2 What is the theoretical regret bound of a neural bandit algorithm that uses perturbed rewards?
• RQ3 How does the computational cost of the proposed method compare to existing neural bandit algorithms?
• RQ4 Does the perturbation-based method maintain strong empirical performance on real-world datasets?
• RQ5 Can the method scale to large neural networks without incurring prohibitive computational costs?

Key findings

• The proposed method achieves a regret bound of $\tilde{\mathbf{O}}(\widetilde{d}\sqrt{T})$, matching theoretical guarantees of prior methods.
• The algorithm avoids explicit exploration and eliminates the need for full covariance matrix inversion, reducing computational cost.
• On the LastFM and Delicious datasets, NPR outperformed NeuralUCB and NeuralTS in cumulative normalized reward.
• NPR demonstrated significantly faster training times than NeuralUCB and NeuralTS due to no matrix inversion overhead.
• The method maintained strong empirical performance on both synthetic and real-world data, validating its effectiveness.
• The key bottleneck in online neural bandit learning lies in model update steps, which the proposed method efficiently addresses.