[Paper Review] Improve Single-Point Zeroth-Order Optimization Using High-Pass and Low-Pass Filters

Xin Chen, Yujie Tang+1 more|arXiv (Cornell University)|Nov 2, 2021

Extremum Seeking Control Systems24 references4 citations

TL;DR

This paper proposes HLF-SZO, a novel single-point zeroth-order optimization method that integrates high-pass and low-pass filters to reduce variance and accelerate convergence. By interpreting the high-pass filter as residual feedback and the low-pass filter as momentum, HLF-SZO achieves an improved iteration complexity of 𝒪(d^{3/2}/ε^{3/2}) in both convex and non-convex settings, outperforming vanilla and residual-feedback SZO methods in numerical experiments.

ABSTRACT

Single-point zeroth-order optimization (SZO) is useful in solving online black-box optimization and control problems in time-varying environments, as it queries the function value only once at each time step. However, the vanilla SZO method is known to suffer from a large estimation variance and slow convergence, which seriously limits its practical application. In this work, we borrow the idea of high-pass and low-pass filters from extremum seeking control (continuous-time version of SZO) and develop a novel SZO method called HLF-SZO by integrating these filters. It turns out that the high-pass filter coincides with the residual feedback method, and the low-pass filter can be interpreted as the momentum method. As a result, the proposed HLF-SZO achieves a much smaller variance and much faster convergence than the vanilla SZO method and empirically outperforms the residual-feedback SZO method, which is verified via extensive numerical experiments.

Motivation & Objective

• To address the high estimation variance and slow convergence of vanilla single-point zeroth-order optimization (SZO) in online and time-varying environments.
• To bridge the gap between zeroth-order optimization and extremum seeking control by borrowing filtering techniques from the latter.
• To develop a new SZO method that achieves faster convergence and lower variance than existing approaches.
• To theoretically establish improved iteration complexity for the proposed method under standard smoothness and Lipschitz conditions.
• To empirically validate the superiority of the method over vanilla SZO and residual-feedback SZO in challenging optimization tasks.

Proposed method

• The method integrates a high-pass filter into the SZO framework, which is mathematically equivalent to the residual feedback scheme used in recent work.
• A low-pass filter is incorporated to emulate the momentum (heavy-ball) method, enhancing convergence speed.
• The combined filtering mechanism reduces gradient estimation variance and improves stability in noisy or non-stationary environments.
• The algorithm uses a single function evaluation per iteration, preserving the online and adaptive nature of SZO.
• The filtering process is applied in a time-discretized form, with parameters tuned to balance noise rejection and responsiveness.
• The method is designed to be compatible with online and black-box optimization, especially in control and reinforcement learning settings.

Experimental results

Research questions

• RQ1 Can high-pass and low-pass filtering techniques from extremum seeking control be effectively adapted to improve single-point zeroth-order optimization?
• RQ2 Does integrating filtering mechanisms reduce the variance and accelerate convergence in SZO compared to vanilla and residual-feedback approaches?
• RQ3 What is the theoretical iteration complexity of the proposed HLF-SZO method under standard smoothness and Lipschitz conditions?
• RQ4 How does HLF-SZO perform in practice on non-stationary or noisy optimization problems, such as online LQR control?
• RQ5 Can the filtering-based approach achieve performance comparable to two-point ZO methods while retaining the single-evaluation advantage of SZO?

Key findings

• The proposed HLF-SZO method achieves an iteration complexity of 𝒪(d^{3/2}/ε^{3/2}) for both convex and non-convex problems, improving upon the vanilla SZO's 𝒪(d²/ε³) complexity.
• The high-pass filter component in HLF-SZO is equivalent to the residual feedback mechanism, which significantly reduces gradient estimation variance.
• The low-pass filter component corresponds to the momentum method, contributing to faster convergence rates.
• In numerical experiments on the LQR problem, HLF-SZO converged faster than residual-feedback SZO and achieved performance comparable to two-point ZO methods.
• Under varying system noise levels (δ = 0, 0.005, 0.01), HLF-SZO maintained faster convergence and better accuracy than residual-feedback SZO across all cases.
• The method empirically outperforms vanilla SZO in terms of both convergence speed and variance reduction, demonstrating practical advantages in online and noisy settings.