[Paper Review] Strategy-Driven Limit Theorems Associated Bandit Problems

Zengjing Chen, Shui Feng+1 more|arXiv (Cornell University)|Apr 9, 2022

Advanced Bandit Algorithms Research4 citations

TL;DR

This paper introduces strategy-driven limit theorems for two-armed bandit problems, establishing a law of large numbers, large deviation principle, and central limit theorem that explicitly depend on sampling strategies. The key contribution is identifying strategy-dependent limiting distributions—non-normal and set-dependent—that reveal the learning structure and enable estimation of maximal/minimal rewards and avoidance of Parrondo’s paradox.

ABSTRACT

Motivated by the study of asymptotic behaviour of the bandit problems, we obtain several strategy-driven limit theorems including the law of large numbers, the large deviation principle, and the central limit theorem. Different from the classical limit theorems, we develop sampling strategy-driven limit theorems that generate the maximum or minimum average reward. The law of large numbers identifies all possible limits that are achievable under various strategies. The large deviation principle provides the maximum decay probabilities for deviations from the limiting domain. To describe the fluctuations around averages, we obtain strategy-driven central limit theorems under optimal strategies. The limits in these theorem are identified explicitly, and depend heavily on the structure of the events or the integrating functions and strategies. This demonstrates the key signature of the learning structure. Our results can be used to estimate the maximal (minimal) rewards, and to identify the conditions of avoiding the Parrondo's paradox in the two-armed bandit problem. It also lays the theoretical foundation for statistical inference in determining the arm that offers the higher mean reward.

Motivation & Objective

• To develop a framework of strategy-driven limit theorems that characterize the asymptotic behavior of sampling strategies in two-armed bandit problems.
• To identify all achievable limits of average rewards under various sampling strategies, extending classical laws of large numbers.
• To provide a large deviation principle that quantifies the maximal decay probabilities of deviations from optimal reward limits.
• To derive a strategic central limit theorem with non-normal limiting distributions that depend on the structure of the sampling strategy and integrating functions.
• To apply the framework to estimate maximal/minimal rewards and to determine conditions for avoiding Parrondo’s paradox in bandit settings.

Proposed method

• Proposes a strategy-driven weak and strong law of large numbers, generalizing Robbins' result by incorporating sampling strategy dependence.
• Establishes a large deviation principle (LDP) with rate function $ I(x) = \inf_{\alpha \in [0,1]} I_\alpha(x) $, where $ I_\alpha(x) $ is derived from mixture of moment generating functions under strategy $ \theta^\alpha $.
• Introduces a strategic central limit theorem where the limiting distribution is explicitly identified as non-normal and set-dependent, depending on the strategy and reward structure.
• Uses Cramér's theorem and the contraction principle to derive the LDP rate function from the empirical measure of rewards under a sequence of strategies $ \theta^\alpha $.
• Constructs a family of strategies $ \theta^\alpha $ that control the long-run proportion of pulls on each arm, enabling the derivation of strategy-dependent limits.
• Employs nonlinear probability and Legendre-Fenchel transforms to characterize the rate functions $ \Lambda^* $, linking moment generating functions to large deviation rates.

Experimental results

Research questions

• RQ1 What are the set of all possible limiting average rewards achievable under different sampling strategies in a two-armed bandit problem?
• RQ2 What is the maximal decay rate of probabilities for deviations from the optimal reward limit under any strategy?
• RQ3 How do fluctuations around the average reward behave under optimal sampling strategies, and what is the form of the limiting distribution?
• RQ4 Can the framework be used to estimate the maximum or minimum expected reward in sequential sampling with unknown arm means?
• RQ5 Under what conditions can Parrondo’s paradox be avoided in the two-armed bandit setting using this strategy-driven approach?

Key findings

• The strategy-driven law of large numbers identifies all possible limits of the average reward as the limit of the empirical mean, which are explicitly determined by the strategy and the underlying reward distributions.
• The large deviation principle provides a rate function $ I(x) = \inf_{\alpha \in [0,1]} I_\alpha(x) $, with $ I_\alpha(x) $ derived from the mixture of moment generating functions under strategy $ \theta^\alpha $, and shows that the decay rate is minimized at the optimal mean.
• The strategic central limit theorem yields a limiting distribution that is generally non-normal and depends on the structure of the strategy and the integrating functions, with explicit form $ I_\alpha(x) = \inf\{ \alpha \Lambda^*_{\mu_L}(y) + (1-\alpha) \Lambda^*_{\mu_R}(z) : \alpha y + (1-\alpha) z = x \} $.
• The framework enables estimation of the maximal and minimal expected rewards by identifying the optimal strategy that achieves the extreme limits.
• The results provide a theoretical basis for statistical inference in hypothesis testing to identify the arm with the higher mean reward, under optimal sampling strategies.
• The analysis shows that Parrondo’s paradox can be avoided when the strategy-dependent limits and rate functions are carefully analyzed, particularly when the optimal strategy does not lead to a paradoxical win from two losing components.