[Paper Review] On Degrees of Freedom of Projection Estimators with Applications to Multivariate Shape Restricted Regression

Xi Chen, Qihang Lin+1 more|arXiv (Cornell University)|Jan 1, 2015

Statistical Methods and Inference52 references9 citations

TL;DR

This paper provides a unified characterization of degrees of freedom for projection estimators in shape-restricted regression, including bounded isotonic, multivariate convex, and penalized convex regression. It extends known results for ridge, Lasso, and generalized Lasso, enabling tuning parameter selection via Stein’s unbiased risk estimate through explicit degrees of freedom formulas.

ABSTRACT

Abstract: Consider the Gaussian sequence model y ∼ N(θ∗, σ2In), where θ ∗ is unknown but known to belong to a closed convex polyhedral set C ⊆ Rn. In this paper we provide a unified characterization of the degrees of freedom for estimators of θ ∗ obtained as the (linearly or quadratically perturbed) par-tial projection of y onto C. As special cases of our results, we derive explicit expressions for the degrees of freedom in many shape restricted regression problems, e.g., bounded isotonic regression, multivariate convex regression and penalized convex regression. Our general theory also yields, as special cases, known results on the degrees of freedom of many well-studied estima-tors in the statistics literature, such as ridge regression, Lasso and generalized Lasso. Our results can be readily used to choose the tuning parameter(s) in-volved in the estimation procedure by minimizing the Stein’s unbiased risk estimate. We illustrate this through simulation studies for bounded isotonic regression and penalized convex regression. As a by-product of our analysis we derive an interesting connection between bounded isotonic regression and isotonic regression on a general partially ordered set, which is of independent interest.

Motivation & Objective

• To develop a general theory for degrees of freedom of projection estimators in shape-restricted regression problems.
• To unify existing results on degrees of freedom for estimators like ridge, Lasso, and generalized Lasso under a common framework.
• To derive explicit degrees of freedom expressions for bounded isotonic regression, multivariate convex regression, and penalized convex regression.
• To enable data-driven tuning parameter selection using Stein’s unbiased risk estimate.
• To reveal a novel connection between bounded isotonic regression and isotonic regression on partially ordered sets.

Proposed method

• The paper models the observation vector y as a Gaussian sequence with mean θ∗ in a closed convex polyhedral set C ⊆ Rn.
• It analyzes estimators defined as (linearly or quadratically perturbed) partial projections of y onto C.
• The degrees of freedom are characterized using geometric and convex analysis tools, particularly leveraging the structure of polyhedral sets.
• The framework applies to both unconstrained and constrained estimation problems with shape restrictions.
• The theory is applied to derive explicit degrees of freedom for specific models, including isotonic and convex regression.
• The method supports tuning parameter selection via minimization of the Stein’s unbiased risk estimate (SURE).

Experimental results

Research questions

• RQ1 What is the general form of degrees of freedom for projection estimators in shape-restricted regression?
• RQ2 How can the degrees of freedom for bounded isotonic regression be explicitly characterized?
• RQ3 To what extent do existing estimators like ridge and Lasso fit into this unified framework?
• RQ4 Can the degrees of freedom formula be used to select tuning parameters via SURE in practice?
• RQ5 What is the connection between bounded isotonic regression and isotonic regression on general partially ordered sets?

Key findings

• The paper derives explicit, closed-form expressions for degrees of freedom in bounded isotonic regression, multivariate convex regression, and penalized convex regression.
• The framework recovers known degrees of freedom results for ridge regression, Lasso, and generalized Lasso as special cases.
• The degrees of freedom are shown to be computable and amenable to tuning parameter selection via Stein’s unbiased risk estimate.
• Simulation studies demonstrate the effectiveness of SURE-based tuning in bounded isotonic and penalized convex regression.
• An unexpected connection is revealed between bounded isotonic regression and isotonic regression on general partially ordered sets, which is of independent theoretical interest.
• The theory provides a principled approach to model selection and uncertainty quantification in shape-restricted estimation.