[Paper Review] Bounds on Query Convergence

Barak A. Pearlmutter|ArXiv.org|Nov 25, 2005

Bayesian Modeling and Causal Inference8 references3 citations

TL;DR

This paper establishes fundamental lower bounds on the convergence rate of query sequences in noisy, gradient-free optimization of a quadratic cost function. It proves that the expected squared error $ E[(x_t - x^*)^2] $ decays no faster than $ O(t^{-1/2}) $, and total regret grows as $ O(t^{1/2}) $, showing that even optimal query strategies are fundamentally limited by noise and lack of gradient feedback.

ABSTRACT

The problem of finding an optimum using noisy evaluations of a smooth cost function arises in many contexts, including economics, business, medicine, experiment design, and foraging theory. We derive an asymptotic bound E[ (x_t - x*)^2 ] >= O(1/sqrt(t)) on the rate of convergence of a sequence (x_0, x_1, >...) generated by an unbiased feedback process observing noisy evaluations of an unknown quadratic function maximised at x*. The bound is tight, as the proof leads to a simple algorithm which meets it. We further establish a bound on the total regret, E[ sum_{i=1..t} (x_i - x*)^2 ] >= O(sqrt(t)) These bounds may impose practical limitations on an agent's performance, as O(eps^-4) queries are made before the queries converge to x* with eps accuracy.

Motivation & Objective

• To characterize the fundamental limits of convergence speed in optimization problems where only noisy function evaluations are available and no gradient information is accessible.
• To identify the minimum rate at which queries can converge to the true optimum under these constraints, given that the cost function is locally quadratic and measurements are corrupted by i.i.d. noise.
• To quantify the total regret incurred by such query processes, showing that it grows sublinearly but non-trivially with time.
• To demonstrate that naive or greedy strategies—especially those with insufficient query variance—can converge to suboptimal solutions without statistical warning signs.
• To argue that the absence of gradient feedback imposes an intrinsic performance ceiling, even for optimal feedback mechanisms.

Proposed method

• The analysis models the optimization problem as estimating the maximum of a known quadratic function $ f(x) = -a(x - x^*)^2 + f(x^*) $, with $ a > 0 $ known and $ x^* $ unknown.
• It uses the Cramér-Rao bound to derive a lower bound on the variance of the estimated slope $ \hat{b}_t $, which is linked to the estimate of $ x^* $ via $ \hat{x}^* = \hat{b}/(2a) $.
• Leverage of the query points $ x_\tau $ about the true optimum $ x^* $ is used to bound the precision of the slope estimate, leading to a lower bound on the variance of $ \hat{x}^*_t $.
• The bound $ \text{var}(\hat{b}_t) \geq \sigma^2 / \sum_{\tau < t} (x_\tau - x^*)^2 $ is derived, which in turn limits the convergence rate of $ x_t $ to $ x^* $.
• The proof establishes that faster query convergence implies higher leverage, but leverage is constrained by the spread of queries, leading to the $ O(t^{-1/2}) $ lower bound on error decay.
• The paper constructs a simple algorithm that achieves this bound by injecting query noise proportional to the standard error of the estimate, balancing exploration and exploitation.

Experimental results

Research questions

• RQ1 What is the fastest possible rate at which query sequences can converge to the true optimum in a gradient-free, noisy optimization setting?
• RQ2 How does the total regret—defined as the cumulative squared deviation from the optimum—scale with the number of queries?
• RQ3 Can a feedback process that only observes noisy function values achieve faster convergence than the derived $ O(t^{-1/2}) $ bound?
• RQ4 What happens when query variance is too low, and how does this affect convergence to a suboptimal solution?
• RQ5 How does the presence or absence of gradient information affect the fundamental limits of optimization performance?

Key findings

• The expected squared error $ E[(x_t - x^*)^2] $ is bounded below by $ \frac{\sigma}{\sqrt{8}a} t^{-1/2} $, proving that convergence cannot be faster than $ O(t^{-1/2}) $.
• The total regret $ E\left[\sum_{\tau=1}^t (x_\tau - x^*)^2\right] $ grows at least as fast as $ O(t^{1/2}) $, indicating a non-trivial cumulative cost of suboptimal queries.
• An $ O(\epsilon^{-4}) $ number of queries are required to achieve $ \epsilon $-accuracy in the query sequence, highlighting a fundamental performance ceiling.
• The bound is tight, as a simple algorithm injecting noise proportional to the standard error of the estimate achieves the theoretical limit.
• Greedy strategies with zero query noise fail to converge to the optimum and instead converge to incorrect estimates, even when the model is correct.
• The results are robust to prior information and small model perturbations, and the asymptotic behavior remains unchanged under smooth priors or local quadratic approximations.