[Paper Review] Global empirical risk minimizers with "shape constraints" are rate optimal in general dimensions

Qiyang Han|arXiv (Cornell University)|May 30, 2019

Statistical Methods and Inference41 references14 citations

TL;DR

This paper establishes that global empirical risk minimizers (ERMs) achieve rate-optimal convergence in high-dimensional models with shape constraints—even when entropy integrals diverge, closing a long-standing gap between upper and lower bounds. By leveraging shape constraints such as monotonicity or concavity, the authors derive matching upper and lower bounds for empirical processes over measurable sets, proving global ERMs are minimax optimal in settings like isotonic regression, density estimation, and classification.

ABSTRACT

Entropy integrals are widely used as a powerful tool to obtain upper bounds for the rates of convergence of global empirical risk minimizers (ERMs), in standard settings such as density estimation and regression. The upper bound for the convergence rates thus obtained typically matches the minimax lower bound when the entropy integral converges, but admits a strict gap compared with the lower bound when it diverges. [BM93] provided a striking example showing that such a gap is real with the entropy structure alone: for a variant of the natural Holder class with low regularity, the global ERM actually converges at the rate predicted by the entropy integral that substantially deviates from the lower bound. The counter-example has spawned a long-standing negative position on the use of global ERMs in the regime where the entropy integral diverges, as they are heuristically believed to converge at a sub-optimal rate in a variety of models. The present paper demonstrates that this gap can be closed if the models admit certain degree of `shape constraints' in addition to the entropy structure. In other words, the global ERMs in such `shape-constrained' models will indeed be rate-optimal, matching the lower bound even when the entropy integral diverges. The models with `shape constraints' we investigate include (i) edge estimation with additive and multiplicative errors, (ii) binary classification, (iii) multiple isotonic regression, (iv) $s$-concave density estimation, all in general dimensions when the entropy integral diverges. Here `shape constraints' are interpreted broadly in the sense that the complexity of the underlying models can be essentially captured by the size of the empirical process over certain class of measurable sets, for which matching upper and lower bounds are obtained to facilitate the derivation of sharp convergence rates for the associated global ERMs.

Motivation & Objective

• To resolve the longstanding gap between upper bounds (via entropy integrals) and minimax lower bounds for global ERMs in high-dimensional models.
• To challenge the heuristic belief that global ERMs are sub-optimal when entropy integrals diverge.
• To demonstrate that shape constraints—such as monotonicity, concavity, or isotonicity—can restore rate optimality in such regimes.
• To unify the analysis of diverse models (e.g., isotonic regression, s-concave density estimation) under a common framework based on empirical processes over measurable sets.
• To establish matching upper and lower bounds for empirical processes in shape-constrained models to achieve sharp convergence rates.

Proposed method

• The authors analyze the complexity of empirical processes over classes of measurable sets induced by shape constraints, replacing reliance on entropy integral structures alone.
• They derive matching upper and lower bounds for the supremum of the empirical process over these structured sets, enabling sharp rate analysis.
• The method leverages metric entropy and chaining techniques tailored to the geometric and order structure imposed by shape constraints.
• The framework applies to general dimensions and accommodates both additive and multiplicative error models.
• It introduces a novel decomposition of the risk that isolates the contribution of shape-constrained complexity from standard entropy-based terms.
• The core technical innovation lies in bounding the entropy of the class of sets defined by the shape constraints, enabling convergence rate matching to minimax lower bounds.

Experimental results

Research questions

• RQ1 Can global empirical risk minimizers achieve rate-optimality in high-dimensional models when entropy integrals diverge?
• RQ2 Do shape constraints—such as monotonicity or concavity—close the gap between entropy-based upper bounds and minimax lower bounds for global ERMs?
• RQ3 In what general classes of models (e.g., isotonic regression, s-concave density estimation) does the inclusion of shape constraints restore rate optimality?
• RQ4 How can empirical process theory be adapted to capture the complexity of shape-constrained models beyond standard entropy structures?
• RQ5 Can matching upper and lower bounds be derived for empirical processes over measurable sets defined by shape constraints to ensure sharp convergence rates?

Key findings

• Global ERMs achieve the minimax optimal rate of convergence in models with shape constraints, even when the entropy integral diverges.
• The paper establishes that the rate predicted by the entropy integral is not a lower bound in such models, and that shape constraints enable matching of the true minimax rate.
• For multiple isotonic regression in general dimensions, the global ERM achieves the optimal convergence rate despite divergent entropy integrals.
• In s-concave density estimation and edge estimation with additive/multiplicative errors, global ERMs are rate-optimal under shape constraints.
• The framework provides sharp upper and lower bounds for empirical processes over measurable sets defined by shape constraints, enabling precise rate analysis.
• The results resolve a long-standing open question about the sub-optimality of global ERMs in divergent entropy regimes, showing they are not inherently sub-optimal when shape constraints are present.