[Paper Review] A Polynomial Time Algorithm for Maximum Likelihood Estimation of Multivariate Log-concave Densities

Ilias Diakonikolas, Anastasios Sidiropoulos+1 more|arXiv (Cornell University)|Dec 13, 2018

Markov Chains and Monte Carlo Methods58 references4 citations

TL;DR

This paper presents the first polynomial-time algorithm for computing the maximum likelihood estimator (MLE) of multivariate log-concave densities. By formulating the problem as a convex optimization task and leveraging efficient stochastic subgradient methods with volume and sampling approximations for convex bodies, the algorithm achieves an $\epsilon$-approximation in time $\mathrm{poly}(n,d,1/\epsilon)$, resolving a long-standing open problem in nonparametric statistics and theoretical computer science.

ABSTRACT

We study the problem of computing the maximum likelihood estimator (MLE) of multivariate log-concave densities. Our main result is the first computationally efficient algorithm for this problem. In more detail, we give an algorithm that, on input a set of $n$ points in $\mathbb{R}^d$ and an accuracy parameter $ε>0$, it runs in time $ ext{poly}(n, d, 1/ε)$, and outputs a log-concave density that with high probability maximizes the log-likelihood up to an additive $ε$. Our approach relies on a natural convex optimization formulation of the underlying problem that can be efficiently solved by a projected stochastic subgradient method. The main challenge lies in showing that a stochastic subgradient of our objective function can be efficiently approximated. To achieve this, we rely on structural results on approximation of log-concave densities and leverage classical algorithmic tools on volume approximation of convex bodies and uniform sampling from convex sets.

Motivation & Objective

• To develop a computationally efficient algorithm for computing the maximum likelihood estimator (MLE) of multivariate log-concave densities.
• To overcome the exponential dependence on dimension $d$ present in prior methods, such as the $\mathrm{poly}(n^d)$ algorithm from [CSS10].
• To provide a rigorous, efficient, and provably correct method for learning multivariate log-concave densities in high dimensions.
• To establish the first proper learning algorithm for multivariate log-concave densities with polynomial runtime in $n$, $d$, and $1/\epsilon$.

Proposed method

• Formulate the log-concave MLE problem as a convex optimization problem over the space of log-concave densities.
• Use a projected stochastic subgradient method to solve the convex optimization problem efficiently.
• Approximate stochastic subgradients via uniform sampling from superlevel sets of the density, leveraging volume estimation algorithms.
• Apply volume approximation and uniform sampling algorithms for convex bodies to efficiently estimate subgradients and normalize densities.
• Use a rejection sampling framework with estimated normalization constants to simulate from the candidate density and estimate likelihoods.
• Combine these components into a recursive algorithm that maintains accuracy within $\epsilon$ using concentration bounds and union bounds.

Experimental results

Research questions

• RQ1 Can the multivariate log-concave MLE be computed in polynomial time in the sample size $n$, dimension $d$, and inverse accuracy $1/\epsilon$?
• RQ2 Is it possible to achieve sub-exponential dependence on dimension $d$ for the MLE computation, overcoming the $\mathrm{poly}(n^d)$ bottleneck of prior methods?
• RQ3 Can efficient sampling and volume estimation techniques be combined with convex optimization to solve nonparametric MLE problems?
• RQ4 What is the complexity of computing the MLE for mixtures of two log-concave distributions, and does the framework extend to such settings?

Key findings

• The proposed algorithm computes an $\epsilon$-approximate log-concave MLE in time $\mathrm{poly}(n,d,1/\epsilon)$, achieving the first polynomial-time solution to this problem.
• The algorithm achieves high-probability correctness with failure probability at most $\tau$, using $\mathrm{poly}(n,d,\log R,1/\delta,\log(1/\tau))$ time, where $R$ is a bound on the input points.
• The method relies on approximating stochastic subgradients via volume and sampling oracles for superlevel sets, which are efficiently implementable using known algorithms.
• The normalization constant $\alpha$ of the candidate density is estimated with error $O(\delta)$, ensuring that the output distribution is within total variation distance $O(\delta)$ of the true target.
• The algorithm ensures that the estimated density $\widetilde{\phi}$ satisfies $\|\widetilde{\phi} - \phi\|_1 \leq 10\delta$, with high probability.
• The framework provides a proper learning algorithm for multivariate log-concave densities, with runtime independent of the sample size $n$ in the logarithmic scale of $1/\epsilon$.