[Paper Review] Rates of convergence of rho-estimators for sets of densities satisfying shape constraints

Yannick Baraud, Lucien Birgé|arXiv (Cornell University)|Mar 15, 2015

Statistical Methods and Inference19 references5 citations

TL;DR

This paper establishes sharp rates of convergence for rho-estimators in shape-restricted density estimation, showing that when the true density is close to an extremal point in a model (e.g., decreasing densities on [0,1]), the risk can be significantly smaller than the minimax bound. The key result is a superminimax behavior: the risk at non-extremal points is bounded by the sum of the risk at a nearby extremal point and the squared Hellinger distance to it, leveraging refined empirical process bounds from Baraud (2016).

ABSTRACT

The purpose of this paper is to pursue our study of rho-estimators built from i.i.d. observations that we defined in Baraud et al. (2014). For a ρ-estimator based on some model S (which means that the estimator belongs to S) and a true distribution of the observations that also belongs to S, the risk (with squared Hellinger loss) is bounded by a quantity which can be viewed as a dimension function of the model and is often related to the "metric dimension" of this model, as defined in Birgé (2006). This is a minimax point of view and it is well-known that it is pessimistic. Typically, the bound is accurate for most points in the model but may be very pessimistic when the true distribution belongs to some specific part of it. This is the situation that we want to investigate here. For some models, like the set of decreasing densities on [0,1], there exist specific points in the model that we shall call "extremal" and for which the risk is substantially smaller than the typical risk. Moreover, the risk at a non-extremal point of the model can be bounded by the sum of the risk bound at a well-chosen extremal point plus the square of its distance to this point. This implies that if the true density is close enough to an extremal point, the risk at this point may be smaller than the minimax risk on the model and this actually remains true even if the true density does not belong to the model. The result is based on some refined bounds on the suprema of empirical processes that are established in Baraud (2016).

Motivation & Objective

• To investigate the behavior of rho-estimators in shape-restricted models when the true density is close to extremal points, where standard minimax bounds are overly conservative.
• To establish that the risk of rho-estimators can be substantially smaller than the minimax risk when the true density is near extremal points in the model.
• To derive a refined risk bound that decomposes into the risk at a well-chosen extremal point and the squared Hellinger distance to it, enabling superminimax performance.
• To demonstrate robustness of rho-estimators under Hellinger deviation from the model, maintaining low risk even when the true density lies outside the model but near a well-behaved point.

Proposed method

• Uses a minimax risk bound for rho-estimators on a model $\overline{S}$, with the risk bounded by a dimension function related to metric entropy.
• Introduces the concept of extremal points in shape-restricted models (e.g., decreasing densities on [0,1]) where the risk is significantly smaller than the typical minimax bound.
• Establishes a risk decomposition: for any density $t$ in the model, $h^2(t, \widehat{s}) \leq R(\overline{s},n) + h^2(t, \overline{s})$, where $\overline{s}$ is an extremal point.
• Applies refined suprema bounds on empirical processes from Baraud (2016) to control the deviation of the estimator from the true density.
• Employs piecewise affine approximation of $\sqrt{t}$ on intervals defined by the model's structure, using a partitioning scheme with controlled variation in the derivative.
• Optimizes the number of pieces in the approximation via a trade-off between bias and complexity, leading to a final bound involving sums of $[\ell(I_j) R_j^2]^{\alpha}$ terms with $\alpha = 1/3$ or $1/5$ depending on the case.

Experimental results

Research questions

• RQ1 Can rho-estimators achieve faster-than-minimax convergence rates when the true density is close to an extremal point in a shape-restricted model?
• RQ2 How does the risk of a rho-estimator behave when the true density lies outside the model but near an extremal point within it?
• RQ3 What is the precise dependence of the risk on the Hellinger distance to the nearest extremal point in the model?
• RQ4 Can the empirical process behavior of rho-estimators be controlled tightly enough to yield sharp bounds in non-i.i.d. or non-regular settings?
• RQ5 What is the optimal trade-off between approximation complexity and estimation error in piecewise affine approximations of $\sqrt{t}$ for density estimation?

Key findings

• The risk of a rho-estimator at a non-extremal point $t$ in the model is bounded by the sum of the risk at a well-chosen extremal point $\overline{s}$ and $h^2(t, \overline{s})$, the squared Hellinger distance.
• When the true density $t$ is close to an extremal point $\overline{s}$, the risk of the rho-estimator can be substantially smaller than the minimax risk on the entire model.
• For the set of decreasing densities on $[0,1]$, extremal points exist where the risk is significantly reduced compared to the typical minimax rate.
• The final bound on the risk involves a sum of the form $\left[\sum_{j=1}^{k} (\ell(I_j) R_j^2)^{1/3}\right]^3 / (4D^2)$, where $R_j = V_{I_j}((\sqrt{t})')$, showing a non-standard dependence on the variation of the derivative.
• By optimizing the partitioning of the domain into $k$ intervals and controlling the number of pieces in the approximation, the method achieves a bound of order $\left[\sum_{j=1}^{k} (\ell(I_j)^3 R_j^2)^{1/5}\right]^5 / (16D^4)$, reflecting a refined trade-off between bias and complexity.
• The results confirm that rho-estimators are robust to small Hellinger deviations from the model, maintaining low risk even when the true density is not in the model but near a well-behaved extremal point.