[Paper Review] On Suboptimality of Least Squares with Application to Estimation of Convex Bodies

Gil Kur, Alexander Rakhlin+1 more|arXiv (Cornell University)|Jun 7, 2020

Statistical Methods and Inference4 citations

TL;DR

This paper establishes a general technique to prove lower bounds on the sample complexity of Least Squares Estimation (LSE) in non-Donsker regimes, demonstrating that LSE is minimax suboptimal for estimating convex bodies via noisy support function measurements in dimension $ d \geq 6 $. It shows LSE achieves a rate of $ \tilde{\Theta}_d(n^{-2/(d-1)}) $, while the minimax optimal rate is $ \Theta_d(n^{-4/(d+3)}) $, proving suboptimality in this setting.

ABSTRACT

We develop a technique for establishing lower bounds on the sample complexity of Least Squares (or, Empirical Risk Minimization) for large classes of functions. As an application, we settle an open problem regarding optimality of Least Squares in estimating a convex set from noisy support function measurements in dimension $d\\geq 6$. Specifically, we establish that Least Squares is mimimax sub-optimal, and achieves a rate of $\ ilde{\\Theta}_d(n^{-2/(d-1)})$ whereas the minimax rate is $\\Theta_d(n^{-4/(d+3)})$.

Motivation & Objective

• To develop a general method for proving lower bounds on the sample complexity of Least Squares Estimation (LSE) in non-Donsker regimes.
• To resolve the open question on the minimax optimality of LSE in the estimation of convex bodies from noisy support function measurements.
• To demonstrate that LSE is provably suboptimal for the class of support functions of compact convex bodies in dimension $ d \geq 6 $.
• To extend the analysis from fixed to random design settings, providing a unified framework for assessing LSE performance.

Proposed method

• Derives a general lower bound technique for LSE risk in convex, non-Donsker function classes using metric entropy and entropy numbers.
• Applies the method to the class of support functions of compact convex bodies in $ \mathbb{R}^d $, showing that the entropy of such classes grows as $ \epsilon^{-p} $ with $ p > 2 $.
• Uses radial projection and spherical caps to analyze the metric entropy of support functions on the sphere, leveraging uniform measures on spherical facets.
• Constructs $ \epsilon $-nets on spherical caps using piecewise linear approximations and applies results from Gao and Wellner (2017) to bound entropy numbers.
• Establishes that LSE cannot achieve a faster rate than $ \tilde{\Theta}_d(n^{-2/(d-1)}) $ in the non-Donsker regime for $ d \geq 6 $.
• Uses concentration inequalities and high-probability bounds on the number of design points in spherical caps to show that sets with large vertices are unlikely to be selected by LSE.

Experimental results

Research questions

• RQ1 Is the Least Squares Estimator minimax optimal for estimating convex bodies from noisy support function measurements in dimension $ d \geq 6 $?
• RQ2 What is the exact rate of convergence of the LSE in the non-Donsker regime for the class of support functions of compact convex bodies?
• RQ3 Can a general technique be developed to establish lower bounds on the LSE risk in non-Donsker regimes?
• RQ4 Does the LSE achieve the minimax rate $ n^{-2/(2+p)} $ or is it bounded by the slower $ n^{-1/p} $ in natural non-Donsker classes?

Key findings

• For $ d \geq 6 $, the Least Squares Estimator is minimax suboptimal in estimating convex bodies from noisy support function measurements.
• The LSE achieves a rate of $ \tilde{\Theta}_d(n^{-2/(d-1)}) $, which is strictly slower than the minimax optimal rate of $ \Theta_d(n^{-4/(d+3)}) $.
• The entropy of the class of support functions of compact convex bodies in $ \mathbb{R}^d $ grows as $ \epsilon^{-p} $ with $ p > 2 $, placing it in the non-Donsker regime for $ d \geq 6 $.
• The paper constructs an $ \epsilon $-net on spherical caps using radial projection and piecewise linear functions, enabling entropy estimation via metric entropy bounds.
• With high probability, LSE only considers convex sets with vertices bounded by $ O(d) $, excluding those with large vertices that would yield poor likelihood.
• The analysis confirms that the LSE risk is bounded below by $ \tilde{\Theta}_d(n^{-2/(d-1)}) $, proving it cannot achieve the minimax rate $ \Theta_d(n^{-4/(d+3)}) $.