[Paper Review] Neural Contextual Bandits with UCB-based Exploration

Dongruo Zhou, Lihong Li+1 more|arXiv (Cornell University)|Nov 11, 2019

Advanced Bandit Algorithms Research59 citations

TL;DR

NeuralUCB uses a deep neural network with neural tangent kernel-inspired random features to construct a UCB and achieve near-optimal regret in stochastic contextual bandits without strong reward assumptions.

ABSTRACT

We study the stochastic contextual bandit problem, where the reward is generated from an unknown function with additive noise. No assumption is made about the reward function other than boundedness. We propose a new algorithm, NeuralUCB, which leverages the representation power of deep neural networks and uses a neural network-based random feature mapping to construct an upper confidence bound (UCB) of reward for efficient exploration. We prove that, under standard assumptions, NeuralUCB achieves $ ilde O(\sqrt{T})$ regret, where $T$ is the number of rounds. To the best of our knowledge, it is the first neural network-based contextual bandit algorithm with a near-optimal regret guarantee. We also show the algorithm is empirically competitive against representative baselines in a number of benchmarks.

Motivation & Objective

• Motivate the need for nonlinear, model-free reward learning in contextual bandits beyond linear assumptions.
• Introduce NeuralUCB that leverages deep nets and neural tangent random features to build a UCB for exploration.
• Provide regret guarantees showing near-optimal performance in terms of effective neural tangent kernel dimension.
• Demonstrate empirical competitiveness of NeuralUCB against representative baselines on synthetic and real datasets.

Proposed method

• Use a fully connected neural network to learn the unknown reward function with a bounded output.
• Construct an upper confidence bound U_{t,a} for each action using f(x_{t,a}; θ_{t-1}) and a term involving the gradient g(x_{t,a}; θ_{t-1}) and a matrix Z_{t-1}.
• Update the network parameters θ_t by approximately minimizing a regularized least-squares objective on observed data (TrainNN).
• Define and utilize the neural tangent kernel (NTK) based effective dimension  d~ to derive regret bounds that scale with  d~ rather than parameter count p.
• Show that the regret bound is ~O( d~ sqrt(T)) under standard assumptions, recovering linear-bandit guarantees as a special case.
• Provide empirical comparisons against baselines on synthetic and real datasets to validate practical performance.

Experimental results

Research questions

• RQ1 Can a neural network-based contextual bandit with UCB-based exploration achieve near-optimal regret without strong reward-model assumptions?
• RQ2 How does the effective dimension of the neural tangent kernel govern the regret, and how does it compare to linear and kernel-based approaches?
• RQ3 Does NeuralUCB perform competitively against established baselines in both synthetic and real-world contextual bandit benchmarks?
• RQ4 What is the impact of optimization error in neural training on the overall regret bound and how can it be controlled?

Key findings

• NeuralUCB achieves a regret bound of ~O( d~ sqrt(T)) where  d~ is the effective NTK dimension, matching near-optimal rates for nonlinear rewards.
• The regret bound recovers the ~O(d sqrt(T)) rate of linear contextual bandits when the reward is linear in context.
• Empirical results show NeuralUCB is competitive with representative baselines on synthetic benchmarks and real-world datasets.
• An appendix variant NeuralUCB_0 aligns with kernel/UCB and shows comparable regret, though NeuralUCB tends to perform better in practice due to richer representations.
• The analysis relies on neural network optimization and generalization results, including NTK-basedGram matrices and gradient-based feature mappings.