[Paper Review] Efficient Robust Proper Learning of Log-concave Distributions

Ilias Diakonikolas, Daniel M. Kane+1 more|arXiv (Cornell University)|Jun 9, 2016

Machine Learning and Algorithms40 references13 citations

TL;DR

This paper presents the first computationally efficient, robust, and proper learning algorithm for univariate log-concave distributions over both continuous and discrete domains. It achieves optimal sample complexity of $ O(\epsilon^{-5/2}) $, runs in time $ \tilde{O}(\epsilon^{-4}) $, and outputs a log-concave hypothesis that is $ O(\text{OPT}) + \epsilon $-close to the target distribution in total variation distance, even under model misspecification.

ABSTRACT

A probability distribution over the Boolean cube is monotone if flipping the value of a coordinate from zero to one can only increase the probability of an element. Given samples of an unknown monotone distribution over the Boolean cube, we give (to our knowledge) the first algorithm that learns an approximation of the distribution in statistical distance using a number of samples that is sublinear in the domain. To do this, we develop a structural lemma describing monotone probability distributions. The structural lemma has further implications to the sample complexity of basic testing tasks for analyzing monotone probability distributions over the Boolean cube: We use it to give nontrivial upper bounds on the tasks of estimating the distance of a monotone distribution to uniform and of estimating the support size of a monotone distribution. In the setting of monotone probability distributions over the Boolean cube, our algorithms are the first to have sample complexity lower than known lower bounds for the same testing tasks on arbitrary (not necessarily monotone) probability distributions. One further consequence of our learning algorithm is an improved sample complexity for the task of testing whether a distribution on the Boolean cube is monotone.

Motivation & Objective

• To develop a computationally efficient, robust, and proper learning algorithm for univariate log-concave distributions over R and Z.
• To achieve optimal sample complexity (up to constant factors) for agnostic learning of log-concave families.
• To ensure the algorithm is robust to model misspecification, providing error guarantees competitive with the best approximation in the family.
• To design a method that runs in polynomial time while maintaining properness, a property often required for interpretability in statistical modeling.

Proposed method

• The algorithm uses a two-stage approach: first, it applies a non-proper agnostic learning algorithm to obtain a piecewise linear approximation of the target distribution.
• It then constructs a dynamic programming framework to approximate the piecewise linear density with a piecewise exponential function that is log-concave.
• The dynamic program operates over a discretized set of possible log-probability values and interval endpoints, using shortest path computation to find the best-fitting log-concave density.
• The method leverages an approximation theorem stating that any log-concave density can be $ \epsilon $-approximated by a piecewise linear density with $ O(\epsilon^{-1/2}) $ pieces.
• It computes the total variation distance between the approximation and the true density using a carefully designed error bound that ensures $ \|g - h\|_1 \leq O(\text{OPT} + \epsilon) $.
• The algorithm ensures that the final hypothesis is a proper log-concave density by enforcing the log-concavity condition through the structure of the dynamic program.

Experimental results

Research questions

• RQ1 Can we achieve optimal sample complexity for robust proper learning of univariate log-concave distributions?
• RQ2 Is it possible to design a polynomial-time algorithm for proper learning that is robust to model misspecification?
• RQ3 Can we efficiently compute a log-concave density that is close to the best possible approximation in the family, even when the true distribution is not log-concave?
• RQ4 What is the computational cost of maintaining properness while achieving near-optimal error guarantees?

Key findings

• The algorithm achieves sample complexity of $ O(\epsilon^{-5/2}) $, which is information-theoretically optimal up to constant factors.
• The running time is $ \tilde{O}(\epsilon^{-4}) $, or $ \tilde{O}(n^{8/5}) $ in terms of sample size $ n $, making it sub-quadratic in input size.
• The output hypothesis $ h $ satisfies $ d_{\text{TV}}(h, f) \leq O(\text{OPT}) + \epsilon $ with probability at least 9/10, where $ \text{OPT} = \inf_{g \in \text{LC}(D)} d_{\text{TV}}(f, g) $.
• The algorithm is the first to provide a proper, robust, and efficient learning solution for univariate log-concave distributions, resolving a long-standing open problem.
• The dynamic programming approach can be extended to learn mixtures of $ k $ log-concave densities, though with time complexity exponential in $ k $.