[Paper Review] Graph Oracle Models, Lower Bounds, and Gaps for Parallel Stochastic Optimization

Blake Woodworth, Jialei Wang+3 more|OpenBU/Boston University Institutional Repository (Boston University)|May 25, 2018

Stochastic Gradient Optimization Techniques55 citations

TL;DR

This paper introduces an oracle-based framework with an oracle-graph to capture parallel stochastic optimization settings, derives lower bounds in terms of graph depth and size, and analyzes tightness and gaps for several parallel configurations.

ABSTRACT

We suggest a general oracle-based framework that captures different parallel stochastic optimization settings described by a dependency graph, and derive generic lower bounds in terms of this graph. We then use the framework and derive lower bounds for several specific parallel optimization settings, including delayed updates and parallel processing with intermittent communication. We highlight gaps between lower and upper bounds on the oracle complexity, and cases where the "natural" algorithms are not known to be optimal.

Motivation & Objective

• Formulate a general, graph-based oracle model for parallel stochastic optimization.
• Derive lower bounds on oracle complexity in terms of graph depth and size.
• Analyze how these bounds apply to specific parallel settings (delays, intermittent communication, layered parallelism).
• Identify gaps between lower bounds and existing algorithms to highlight potential for optimal or improved methods.

Proposed method

• Define an oracle graph G with nodes as stochastic oracle queries and edges encoding ancestral information accessible for each query.
• Use stochastic gradient and stochastic prox oracles to model gradient/prox information exchange under graph constraints.
• Construct hard function families via randomized reductions to prove lower bounds that scale with L (Lipschitz), H (smoothness), B (domain bound), D (depth), and N (size).
• Show how lower bounds combine an optimization term (dependent on D) and a statistical term (dependent on N).
• Apply Yao’s minimax principle and utilize Moreau envelope smoothing for prox-oracle lower bounds.

Experimental results

Research questions

• RQ1 What are the fundamental limits (lower bounds) on oracle complexity for parallel stochastic optimization under a dependency graph?
• RQ2 How do these limits scale with problem parameters (L, H, B) and with graph properties (depth D and size N) across gradient and prox oracles?
• RQ3 Do existing parallel optimization algorithms attain these bounds, and where are there gaps indicating potential for improvement?
• RQ4 How do specific parallel structures (e.g., layer graphs, delays, intermittent communication) compare in terms of achievable rates and gaps between lower and upper bounds?
• RQ5 Can smoothing or active querying close gaps between natural algorithms and optimal rates in certain graph settings?

Key findings

• The paper provides lower bounds that scale with the depth D and size N of the oracle graph for convex, L-Lipschitz, H-smooth functions, with an optimization term and a statistical term.
• For gradient oracles, the lower bound includes a term Ω(min{LB/√D, HB²/D²} + LB/√N).
• For prox oracles, the lower bound includes a term Ω(min{LB/D, HB²/D²} + LB/√N).
• These bounds are tight for many common graphs, and there are gaps between lower bounds and the performance of natural algorithms in several settings (e.g., delayed updates, intermittent communication).
• In simple layer graphs, smoothed A-MB-SGD can match the prox-based lower bound, and SVRG-based methods can improve when active queries are allowed; for delay graphs, wait-and-collect can be optimal but not necessarily natural SGD.
• The results extend beyond fixed-dimension assumptions and apply to non-smooth cases via smoothing of the Moreau envelope, showing the interplay between Lipschitz and smoothness in parallel settings.