[Paper Review] Nonparametric Contextual Bandits in an Unknown Metric Space

Nirandika Wanigasekara, Christina Lee Yu|arXiv (Cornell University)|Aug 3, 2019

Advanced Bandit Algorithms Research22 references4 citations

TL;DR

This paper proposes a novel nonparametric contextual bandit algorithm that learns latent similarities among arms in an unknown metric space, enabling adaptive partitioning of the context-arm space to improve learning efficiency. It achieves regret bounds scaling with the local geometry of reward functions, showing $O(\sqrt{KT})$ regret when the measure of $\delta$-optimal arms scales linearly with $\delta$, without requiring prior knowledge of the arm metric or reward structure.

ABSTRACT

Consider a nonparametric contextual multi-arm bandit problem where each arm $a \in [K]$ is associated to a nonparametric reward function $f_a: [0,1] o \mathbb{R}$ mapping from contexts to the expected reward. Suppose that there is a large set of arms, yet there is a simple but unknown structure amongst the arm reward functions, e.g. finite types or smooth with respect to an unknown metric space. We present a novel algorithm which learns data-driven similarities amongst the arms, in order to implement adaptive partitioning of the context-arm space for more efficient learning. We provide regret bounds along with simulations that highlight the algorithm's dependence on the local geometry of the reward functions.

Motivation & Objective

• To address the challenge of efficient learning in nonparametric contextual bandits with a large number of arms and unknown structural relationships among them.
• To develop an algorithm that learns arm similarities from data rather than relying on pre-specified metrics or kernels.
• To provide regret bounds that reflect the local geometry of reward functions, particularly the scaling of $\delta$-optimal arms as $\delta \to 0$.
• To demonstrate that data-driven similarity learning improves long-term performance compared to independent arm learning or suboptimal metric assumptions.

Proposed method

• Adapts Slivkin’s Zooming algorithm to dynamically partition the context-arm space based on pairwise arm similarities estimated from observed rewards.
• Uses a data-driven distance metric $\mathcal{D}_u^v(a,a')$ derived from reward function differences, avoiding reliance on pre-specified metrics.
• Employs a flagging rule based on $n_t(\rho) \geq 4\ln(T)/\Delta^2$ to trigger subpartitioning when sufficient data is collected in a region.
• Introduces a clustering step to group arms with similar reward functions, enabling shared learning across similar arms.
• Applies adaptive partitioning that increases resolution in regions near optimal policies, improving estimation precision where it matters most.
• Uses a parameter $k=26$ for similarity estimation, with sensitivity to $k$-selection affecting sampling efficiency.

Experimental results

Research questions

• RQ1 Can an algorithm learn useful arm similarities in a nonparametric contextual bandit setting without prior knowledge of the underlying metric space?
• RQ2 How does the regret of a contextual bandit algorithm scale when the number of arms is large and the reward functions are smooth with respect to an unknown metric?
• RQ3 Does data-driven similarity learning outperform methods that assume a fixed metric or learn arms independently?
• RQ4 How does the algorithm’s performance depend on the local geometry of the reward functions, particularly the measure of $\delta$-optimal arms?

Key findings

• The proposed algorithm achieves $O(\sqrt{KT})$ regret when the measure of $\delta$-optimal arms scales linearly with $\delta$, indicating efficient adaptation to local reward geometry.
• Simulations show the algorithm initially incurs a high cost due to similarity estimation but significantly outperforms independent arm learning and metric-based baselines in long-horizon settings.
• The algorithm performs robustly under measure-preserving transformations of the arm space, unlike metric-dependent algorithms that degrade with poor metric choices.
• The frequency plots demonstrate that the algorithm learns and sharpens its policy over time, with the shape of arm selection becoming increasingly aligned with the optimal zigzag policy.
• The oracle variant using true distances performs best, but the algorithm using data-driven similarities closely approximates it, showing the effectiveness of self-learned metrics.
• Using a fixed metric representation (e.g., $|\theta_a - \theta_{a'}|$) leads to slower convergence than the true distance, highlighting the difficulty of choosing an appropriate metric a priori.