[Paper Review] Uniformly Valid Post-Regularization Confidence Regions for Many Functional Parameters in Z-Estimation Framework

Alexandre Belloni, Victor Chernozhukov+2 more|arXiv (Cornell University)|Dec 23, 2015

Statistical Methods and Inference34 references13 citations

TL;DR

This paper develops uniformly valid simultaneous confidence bands for a large number of functional parameters in high-dimensional Z-estimation models, where the number of parameters $\tilde{p}$ may greatly exceed sample size $n$. It uses a multiplier bootstrap on orthogonalized score functions to construct confidence bands without solving high-dimensional optimization problems, achieving uniform validity under weak regularity conditions and enabling inference on entire conditional distribution functions in models like functional logistic regression.

ABSTRACT

In this paper we develop procedures to construct simultaneous confidence bands for $ ilde p$ potentially infinite-dimensional parameters after model selection for general moment condition models where $ ilde p$ is potentially much larger than the sample size of available data, $n$. This allows us to cover settings with functional response data where each of the $ ilde p$ parameters is a function. The procedure is based on the construction of score functions that satisfy certain orthogonality condition. The proposed simultaneous confidence bands rely on uniform central limit theorems for high-dimensional vectors (and not on Donsker arguments as we allow for $ ilde p \gg n$). To construct the bands, we employ a multiplier bootstrap procedure which is computationally efficient as it only involves resampling the estimated score functions (and does not require resolving the high-dimensional optimization problems). We formally apply the general theory to inference on regression coefficient process in the distribution regression model with a logistic link, where two implementations are analyzed in detail. Simulations and an application to real data are provided to help illustrate the applicability of the results.

Motivation & Objective

• To address the lack of uniformly valid inference procedures for many functional parameters in high-dimensional models after model selection.
• To construct simultaneous confidence bands for a continuum of parameters, including infinite-dimensional objects, in moment condition models with $\tilde{p} \gg n$.
• To enable inference on the full conditional distribution of a response variable using functional regression models, such as distribution regression with logistic links.
• To develop computationally efficient methods that avoid repeated high-dimensional optimization by relying on resampling estimated score functions.
• To establish theoretical validity of the multiplier bootstrap under weak moment and sparsity conditions for high-dimensional nuisance parameters.

Proposed method

• Constructs orthogonalized score functions that satisfy a Neyman orthogonality condition to ensure robustness to estimation errors in nuisance parameters.
• Employs a multiplier bootstrap procedure that resamples only the estimated score functions, avoiding repeated solution of high-dimensional optimization problems.
• Uses uniform central limit theorems for high-dimensional vectors (not Donsker-type arguments) to justify asymptotic normality under $\tilde{p} \gg n$.
• Applies entropy-based complexity control via metric entropy integrals to bound the supremum of empirical processes over function classes.
• Implements a two-step estimation strategy: first estimate high-dimensional nuisance parameters via Lasso or Post-Lasso, then construct confidence bands using the residualized score functions.
• Derives uniform confidence bands for the entire parameter process $\theta_{uj}$ over $u \in \mathcal{U}$ and $j \in [\tilde{p}]$ using a multiplier bootstrap critical value.

Experimental results

Research questions

• RQ1 Can we construct uniformly valid simultaneous confidence bands for a large number of functional parameters in high-dimensional models where $\tilde{p} \gg n$?
• RQ2 How can we ensure the validity of inference after model selection when the number of parameters exceeds the sample size?
• RQ3 Can we avoid solving high-dimensional optimization problems repeatedly in post-selection inference for functional parameters?
• RQ4 What conditions ensure the asymptotic validity of the multiplier bootstrap for high-dimensional vector processes in Z-estimation frameworks?
• RQ5 How can we extend inference from scalar parameters to entire functional parameter processes in distribution regression models?

Key findings

• The proposed method constructs uniformly valid simultaneous confidence bands for $\tilde{p}$ functional parameters even when $\tilde{p} \gg n$, under weak regularity and sparsity conditions.
• The multiplier bootstrap procedure is computationally efficient, requiring only resampling of estimated score functions rather than re-solving optimization problems.
• The method achieves uniform validity over a large class of probability measures $\mathcal{P}_n$, ensuring coverage probability is uniformly close to nominal level.
• Theoretical guarantees are established via entropy integral bounds, showing that the complexity of the parameter space is controlled by $\tilde{p}$, $n$, and the sparsity level $k$.
• The approach is applicable to inference on the entire conditional distribution function in logistic distribution regression models, not just mean or quantile features.
• The key technical result is a uniform central limit theorem for high-dimensional vectors with $\tilde{p} \gg n$, derived using metric entropy and symmetrization arguments.