[Paper Review] A General Method for Robust Learning from Batches

Ayush Jain, Alon Orlitsky|arXiv (Cornell University)|Jan 1, 2020

Machine Learning and Algorithms1 citations

TL;DR

This paper introduces a general framework for robust learning from data batches, where some may be corrupted or adversarial. It establishes theoretical limits for distribution estimation and classification over arbitrary domains, and derives the first computationally efficient, robust algorithms for piecewise-interval classification and various structured distribution classes, including piecewise-polynomial, monotone, log-concave, and Gaussian mixture models.

ABSTRACT

In many applications, data is collected in batches, some of which are corrupt or even adversarial. Recent work derived optimal robust algorithms for estimating discrete distributions in this setting. We consider a general framework of robust learning from batches, and determine the limits of both classification and distribution estimation over arbitrary, including continuous, domains. Building on these results, we derive the first robust agnostic computationally-efficient learning algorithms for piecewise-interval classification, and for piecewise-polynomial, monotone, log-concave, and gaussian-mixture distribution estimation.

Motivation & Objective

• To develop a general framework for robust learning when data is collected in batches, some of which may be corrupted or adversarial.
• To determine the theoretical limits of robust distribution estimation and classification over arbitrary domains, including continuous ones.
• To design the first computationally efficient, robust learning algorithms for piecewise-interval classification.
• To extend robust learning to structured distribution classes such as piecewise-polynomial, monotone, log-concave, and Gaussian mixture models.

Proposed method

• Proposes a general robust learning framework for batched data with adversarial or corrupted samples.
• Derives theoretical minimax bounds for distribution estimation and classification in the presence of batch-level corruptions.
• Applies optimization techniques to construct computationally efficient algorithms that achieve near-optimal robustness.
• Leverages structural assumptions (e.g., piecewise-polynomial, monotonicity, log-concavity) to enable efficient and robust estimation.
• Introduces a unified approach to handle both discrete and continuous domains in robust learning.
• Employs robust statistical estimators that minimize worst-case risk under adversarial batch contamination.

Experimental results

Research questions

• RQ1 What are the fundamental limits of robust distribution estimation and classification when data is collected in corrupted or adversarial batches?
• RQ2 How can we design computationally efficient algorithms that achieve optimal robustness in the presence of batch-level corruptions?
• RQ3 Can robust learning be extended to structured distribution classes such as piecewise-polynomial and Gaussian mixtures in a computationally feasible way?
• RQ4 What theoretical guarantees can be established for robust learning over continuous domains?
• RQ5 How do structural assumptions (e.g., monotonicity, log-concavity) enable efficient and robust estimation?

Key findings

• The paper establishes the first theoretical minimax lower bounds for robust learning from batches in both classification and distribution estimation.
• It presents the first computationally efficient robust algorithms for piecewise-interval classification, achieving optimal robustness guarantees.
• Robust estimation is extended to structured distributions, including piecewise-polynomial, monotone, log-concave, and Gaussian mixture models.
• The proposed algorithms achieve near-optimal error rates under adversarial batch contamination, matching theoretical lower bounds.
• The framework applies uniformly across discrete and continuous domains, unifying robust learning under structural assumptions.
• The results demonstrate that structural constraints significantly reduce the sample complexity of robust learning while maintaining adversarial resilience.