[Paper Review] Efficient Stochastic Gradient Descent for Distributionally Robust Learning

Soumyadip Ghosh, Mark S. Squillante+1 more|arXiv (Cornell University)|May 22, 2018

Stochastic Gradient Optimization Techniques8 citations

TL;DR

This paper proposes a novel stochastic gradient descent algorithm for distributionally robust learning that improves scalability by progressively expanding data support from a small subset to the full dataset. By dynamically increasing the training batch size, the method achieves faster convergence and better generalization than standard stochastic approaches, particularly on large-scale datasets.

ABSTRACT

We consider a new stochastic gradient descent algorithm for efficiently solving general min-max optimization problems that arise naturally in distributionally robust learning. By focusing on the entire dataset, current approaches do not scale well. We address this issue by initially focusing on a subset of the data and progressively increasing this support to statistically cover the entire dataset.

Motivation & Objective

• To address the poor scalability of existing stochastic gradient descent methods in distributionally robust learning due to reliance on full-batch computations.
• To develop an efficient optimization framework that maintains robustness while reducing computational overhead on large datasets.
• To enable faster convergence and improved generalization by dynamically expanding the data subset used in training.
• To bridge the gap between distributionally robust optimization and practical scalability in real-world machine learning applications.

Proposed method

• The algorithm begins training on a small, randomly selected subset of the data to reduce initial computational cost.
• It progressively increases the size of the training batch over time, ensuring broader data coverage as optimization proceeds.
• The method maintains a min-max optimization structure typical of distributionally robust learning, focusing on worst-case distributional shifts.
• Stochastic gradients are computed on the expanding data subset, balancing exploration and exploitation of the loss landscape.
• The progressive data expansion is designed to improve statistical coverage and reduce variance in gradient estimates.
• The approach integrates adaptive learning rate strategies to enhance convergence speed and stability.

Experimental results

Research questions

• RQ1 How can stochastic gradient descent be made more scalable for distributionally robust learning without sacrificing robustness?
• RQ2 What is the impact of progressively expanding the training data subset on convergence speed and generalization?
• RQ3 Can a dynamic data support strategy reduce computational cost while maintaining performance on large-scale datasets?
• RQ4 How does the proposed method compare to standard SGD in terms of robustness to distributional shifts?

Key findings

• The proposed method achieves faster convergence compared to standard stochastic gradient descent on large-scale distributionally robust learning problems.
• Progressive data expansion leads to improved generalization performance across multiple benchmark datasets.
• The algorithm reduces computational overhead by avoiding full-batch gradient computations from the start.
• The method maintains robustness to distributional shifts, even when trained on progressively expanding data subsets.