[Paper Review] Semi-Cyclic Stochastic Gradient Descent

Hubert Eichner, Tomer Koren+3 more|arXiv (Cornell University)|Apr 23, 2019

Stochastic Gradient Optimization Techniques18 citations

TL;DR

This paper proposes a pluralistic semi-cyclic stochastic gradient descent (SC-SGD) method that maintains the same convergence guarantees as i.i.d. SGD despite block-cyclic data sampling, where data is drawn in fixed-order cycles across heterogeneous blocks. By training separate models per block and combining them via a simple averaging strategy, the method achieves optimal error bounds without degradation, even with only two blocks, and outperforms consensus models in practice.

ABSTRACT

We consider convex SGD updates with a block-cyclic structure, i.e. where each cycle consists of a small number of blocks, each with many samples from a possibly different, block-specific, distribution. This situation arises, e.g., in Federated Learning where the mobile devices available for updates at different times during the day have different characteristics. We show that such block-cyclic structure can significantly deteriorate the performance of SGD, but propose a simple approach that allows prediction with the same performance guarantees as for i.i.d., non-cyclic, sampling.

Motivation & Objective

• To analyze the performance degradation of standard SGD when data is sampled in block-cyclic order, especially in federated learning settings with diurnal device availability.
• To demonstrate that standard SGD fails under block-cyclic sampling, even with only two blocks, due to persistent distributional shifts across cycles.
• To propose a pluralistic approach that maintains the same convergence guarantees as i.i.d. SGD, regardless of the number of blocks or cyclic structure.
• To show empirically that pluralistic models outperform both consensus models and idealized i.i.d. baselines when block distributions differ significantly.

Proposed method

• Model the data as a mixture of m block-specific distributions, with samples drawn in a fixed cyclic order across blocks.
• Introduce a pluralistic training strategy where each block is associated with a separate model, trained via independent SGD chains.
• Use a simple averaging mechanism over the most recent model iterates from each block to form a final predictor, ensuring robustness and convergence guarantees.
• Prove that this pluralistic approach achieves the same O(√(B²/T)) error bound as i.i.d. SGD, even under worst-case block-cyclic sampling.
• Extend the method to a hedged variant that balances performance across blocks and matches or exceeds i.i.d. baselines when block differences are large.
• Validate the approach empirically on a sentiment classification task using real-world data with diurnal patterns, comparing consensus and pluralistic models.

Experimental results

Research questions

• RQ1 Can standard SGD maintain optimal convergence guarantees under block-cyclic sampling, where data is drawn from different distributions in a fixed cyclic order?
• RQ2 Does the performance of SGD degrade arbitrarily with only two blocks in a cycle, even when each block contains a large number of samples?
• RQ3 Can a pluralistic model—training a separate predictor per block—achieve the same convergence guarantees as i.i.d. SGD under block-cyclic sampling?
• RQ4 Is the pluralistic approach competitive with or superior to an idealized i.i.d. baseline when block distributions differ significantly?
• RQ5 Can a hedged pluralistic model maintain strong performance across blocks while matching or exceeding the accuracy of i.i.d. training in non-i.i.d. settings?

Key findings

• Block-cyclic sampling can cause arbitrarily bad performance degradation in standard SGD, even with only two blocks, due to persistent distributional shifts across cycles.
• The proposed pluralistic approach achieves the same O(√(B²/T)) convergence guarantee as i.i.d. SGD, regardless of the number of blocks or cyclic structure.
• Empirical results show that pluralistic models outperform consensus models trained on the same block-cyclic data, especially when block distributions differ.
• The pluralistic model matches or exceeds the performance of an idealized i.i.d. baseline, demonstrating that embracing heterogeneity improves generalization.
• A hedged variant of the pluralistic model maintains strong performance across blocks and is competitive with or better than i.i.d. training when block differences are high.