[Paper Review] Composite Logconcave Sampling with a Restricted Gaussian Oracle

Ruoqi Shen, Kevin Tian+1 more|arXiv (Cornell University)|Jun 10, 2020

Sparse and Compressive Sensing Techniques30 references4 citations

TL;DR

This paper presents Composite-Sample, an algorithm for approximately sampling from composite log-concave distributions of the form $\pi(x) \propto \exp(-f(x) - g(x))$, where $f$ is well-conditioned and $g$ is convex but possibly non-smooth. The method leverages a restricted Gaussian oracle for $g$ and access to the minimizer of $f+g$, achieving $\tilde{O}(\kappa^2 d \log^2(\kappa d / \epsilon))$ runtime for $\epsilon$-accuracy total variation distance, significantly improving prior composite sampling bounds.

ABSTRACT

We consider sampling from composite densities on $\mathbb{R}^d$ of the form $dπ(x) \propto \exp(-f(x) - g(x))dx$ for well-conditioned $f$ and convex (but possibly non-smooth) $g$, a family generalizing restrictions to a convex set, through the abstraction of a restricted Gaussian oracle. For $f$ with condition number $κ$, our algorithm runs in $O \left(κ^2 d \log^2 frac{κd}ε ight)$ iterations, each querying a gradient of $f$ and a restricted Gaussian oracle, to achieve total variation distance $ε$. The restricted Gaussian oracle, which draws samples from a distribution whose negative log-likelihood sums a quadratic and $g$, has been previously studied and is a natural extension of the proximal oracle used in composite optimization. Our algorithm is conceptually simple and obtains stronger provable guarantees and greater generality than existing methods for composite sampling. We conduct experiments showing our algorithm vastly improves upon the hit-and-run algorithm for sampling the restriction of a (non-diagonal) Gaussian to the positive orthant.

Motivation & Objective

• To develop a sampling algorithm for composite log-concave distributions where the potential function is the sum of a well-conditioned function $f$ and a convex, possibly non-smooth function $g$.
• To extend the toolkit of logconcave sampling beyond the well-behaved case by introducing a restricted Gaussian oracle as a natural abstraction for handling non-smooth components.
• To achieve runtime guarantees that are competitive with non-composite sampling methods while maintaining generality and robustness to inexact oracle and minimizer access.
• To demonstrate that the restricted Gaussian oracle is a practical and powerful primitive for composite sampling, enabling efficient and conceptually lightweight reductions.

Proposed method

• The algorithm uses a restricted Gaussian oracle that samples from a distribution proportional to $\exp(-\|x - v\|^2 / (2\lambda) - g(x))$, enabling efficient sampling under the composite potential.
• It reduces the composite sampling problem to a sequence of restricted Gaussian sampling and gradient queries to $f$, using a carefully designed iterative scheme with a Lyapunov function to track convergence.
• The method incorporates a time-varying temperature parameter $\alpha$ to gradually refine the sampling distribution, ensuring stability and convergence.
• A key technical component is the use of a quadratic approximation to $f$ in a local neighborhood, allowing the algorithm to simulate a Langevin-like diffusion with controlled error.
• The algorithm handles inexactness in the minimizer $x^*$ and oracle calls by showing robustness to inverse-polynomial error, with bounds affected only by constant factors.
• Theoretical analysis relies on a Lyapunov function and differential inequality arguments to bound the rate of change of the distance to the target distribution.

Experimental results

Research questions

• RQ1 Can we design an efficient sampling algorithm for composite log-concave distributions using only a restricted Gaussian oracle for the non-smooth component $g$ and access to the minimizer of $f+g$?
• RQ2 How does the restricted Gaussian oracle compare to standard first-order or zeroth-order oracles in terms of sampling efficiency and generality for composite problems?
• RQ3 What is the optimal runtime complexity for composite log-concave sampling under minimal assumptions, and can it match or approach the best non-composite sampling bounds?
• RQ4 To what extent is the algorithm robust to inexact minimization of $f+g$ and approximate implementation of the restricted Gaussian oracle?

Key findings

• The Composite-Sample algorithm achieves $\tilde{O}(\kappa^2 d \log^2(\kappa d / \epsilon))$ runtime for sampling within $\epsilon$ total variation distance, where $\kappa$ is the condition number of $f$.
• The algorithm requires only a constant expected number of queries per iteration to the gradient of $f$ and the restricted Gaussian oracle for $g$.
• The method is robust to inverse-polynomial error in both the minimizer $x^*$ and the restricted Gaussian oracle, with bounds affected only by constant factors.
• The analysis establishes a differential inequality on the distance to the target distribution using a Lyapunov function, leading to a $\tilde{O}(\kappa^2 d)$ runtime bound.
• The approach improves upon prior composite sampling methods in both generality and runtime, especially in settings where $g$ is non-smooth or structured (e.g., $\ell_1$-regularized).
• Empirical validation on $d=10$ using rejection sampling confirms the correctness of the algorithm, with visual agreement in projected 2D histograms.