[Paper Review] Entropy of convex functions on $R^d$

Fuchang Gao, Jon A. Wellner|arXiv (Cornell University)|Feb 5, 2015

Point processes and geometric inequalities15 references3 citations

TL;DR

This paper establishes sharp bounds for the $\varepsilon$-entropy of convex functions on $\mathbb{R}^d$ under $L^p$ norms, showing that the universal lower bound $\varepsilon^{-d/2}$ is also an upper bound for $d$-dimensional polytopes, while the upper bound $\varepsilon^{-\frac{(d-1)}{2}\cdot\frac{pr}{r-p}}$ is attained by the unit ball when $p > \frac{dr}{d+(d-1)r}$. The results resolve the dependence of metric entropy on the geometry of the domain and have implications for nonparametric estimation rates.

ABSTRACT

Let $Ω$ be a bounded closed convex set in ${\mathbb R}^d$ with non-empty interior, and let ${\cal C}_r(Ω)$ be the class of convex functions on $Ω$ with $L^r$-norm bounded by $1$. We obtain sharp estimates of the $ε$-entropy of ${\cal C}_r(Ω)$ under $L^p(Ω)$ metrics, $1\le p\frac{dr}{d+(d-1)r}$ is attained by the closed unit ball. While a general convex body can be approximated by inscribed polytopes, the entropy rate does not carry over to the limiting body. Our results have applications to questions concerning rates of convergence of nonparametric estimators of high-dimensional shape-constrained functions.

Motivation & Objective

• To determine the exact asymptotic growth rate of the $\varepsilon$-entropy of convex functions on bounded convex domains in $\mathbb{R}^d$ under $L^p$ norms.
• To establish sharp upper and lower bounds for the metric entropy of the class $\mathcal{C}_r(\Omega)$, where $\Omega$ is a compact convex set with non-empty interior and $1 \leq p < r \leq \infty$.
• To investigate how the geometric structure of $\Omega$—such as being a polytope or the unit ball—affects the entropy rate of convex functions.
• To resolve the discrepancy between entropy rates of approximating polytopes and their limiting convex bodies, showing that entropy does not carry over in the limit.

Proposed method

• Derive a universal lower bound of $\varepsilon^{-d/2}$ for the $\varepsilon$-entropy of convex functions on any $d$-dimensional convex body $\Omega$ using a construction of disjoint squares or caps.
• Prove that this $\varepsilon^{-d/2}$ bound is also an upper bound for $\Omega$ being a $d$-dimensional polytope by triangulating the polytope and constructing a family of convex functions supported on small disjoint regions.
• Establish a matching upper bound of order $\varepsilon^{-\frac{(d-1)}{2}\cdot\frac{pr}{r-p}}$ for the unit ball by constructing $2^s$ convex functions supported on disjoint spherical caps, where $s \sim h^{-(d-1)/2}$, and using $L^p$ distance arguments.
• Use the regularized incomplete beta function to estimate the area of spherical caps and control the number of disjoint caps that can be packed on the unit sphere.
• Apply a chaining argument to construct a set of $2^{s/2}$ functions with pairwise $L^p$ distance at least $\delta_d h^{1/p}$, leading to an exponential lower bound in $h^{-(d-1)/2}$.
• Demonstrate that the entropy rate does not stabilize under approximation of a general convex body by inscribed polytopes, by showing that the entropy of the limit body does not inherit the entropy rate of the approximating polytopes.

Experimental results

Research questions

• RQ1 What is the exact asymptotic rate of the $\varepsilon$-entropy of convex functions on a $d$-dimensional convex body $\Omega$ under $L^p$ norms for $1 \leq p < r \leq \infty$?
• RQ2 Does the $\varepsilon^{-d/2}$ lower bound for metric entropy hold as an upper bound when $\Omega$ is a $d$-dimensional polytope?
• RQ3 Is the entropy rate of the class $\mathcal{C}_r(\Omega)$ maximized when $\Omega$ is the unit ball, and if so, what is the precise rate?
• RQ4 Can the entropy of a general convex body be approximated by the entropy of its inscribed polytopes, or does the entropy rate fail to converge in the limit?

Key findings

• The $\varepsilon^{-d/2}$ rate is a universal lower bound for the $\varepsilon$-entropy of convex functions on any $d$-dimensional convex body $\Omega$ with non-empty interior.
• For $d$-dimensional convex polytopes, the $\varepsilon^{-d/2}$ rate is also an upper bound, meaning it is sharp and geometrically optimal.
• For the closed unit ball $\Omega = B_d(0,1)$, the $\varepsilon$-entropy of $\mathcal{C}_\infty(\Omega)$ under $L^p$ norm is bounded below by $\exp\left(C\varepsilon^{-(d-1)p/2}\right)$, which matches the upper bound $\varepsilon^{-\frac{(d-1)}{2}\cdot\frac{pr}{r-p}}$ when $r = \infty$ and $p > \frac{d}{d+1}$.
• The entropy rate of a general convex body does not inherit the entropy rate of its approximating polytopes, indicating that the entropy behavior is not continuous under Hausdorff convergence.
• The bracketing entropy of $\mathcal{C}_\infty([0,1]^d)$ under $L^p$ norm is bounded below by $c \varepsilon^{-d/2}$ for some constant $c > 0$ depending on $d$ and $p$, due to the lower bound on the $L^1$-norm and the universal $\varepsilon^{-d/2}$ lower bound.
• The construction of $2^s$ convex functions supported on disjoint spherical caps of height $h$ yields a set of functions with pairwise $L^p$ distance at least $\delta_d h^{1/p}$, leading to a lower bound of order $\varepsilon^{-(d-1)p/2}$ when $\delta_d h^{1/p} = \varepsilon$.