[Paper Review] Tuning parameter selection in econometrics

Denis Chetverikov|arXiv (Cornell University)|May 5, 2024

Forecasting Techniques and Applications18 citations

TL;DR

A selective survey of methods for tuning parameter choice in nonparametric and L1-penalized econometric estimation, detailing Mallows, Stein, Lepski, cross-validation, penalization, and aggregation with extensions to clustered, panel, and generalized models.

ABSTRACT

I review some of the main methods for selecting tuning parameters in nonparametric and $\ell_1$-penalized estimation. For the nonparametric estimation, I consider the methods of Mallows, Stein, Lepski, cross-validation, penalization, and aggregation in the context of series estimation. For the $\ell_1$-penalized estimation, I consider the methods based on the theory of self-normalized moderate deviations, bootstrap, Stein's unbiased risk estimation, and cross-validation in the context of Lasso estimation. I explain the intuition behind each of the methods and discuss their comparative advantages. I also give some extensions.

Motivation & Objective

• Clarify the main tuning parameter selection problems across nonparametric and high-dimensional settings in econometrics.
• Present and compare prominent methods (Mallows, Stein, Lepski, cross-validation, penalization, aggregation) for series estimators.
• Explain theoretical guarantees, such as oracle inequalities and asymptotic optimality, and discuss practical feasibility and extensions.
• Highlight extensions to clustered/panel data, and to quantile and generalized linear models.
• Provide guidance on when each method is advantageous and how they relate to estimation/ prediction objectives.

Proposed method

• Describe the problem setup for series estimators in nonparametric mean regression and for high-dimensional Lasso estimation.
• Explain the Mallows and Stein unbiased risk estimation approaches and their conditions (e.g., Gaussian errors for Stein).
• Outline the Lepski method and its pointwise (and extendable to other metrics) adaptation mechanism with tests based on bias-variance considerations.
• Discuss cross-validation variants (validation, V-fold, leave-one-out) and their universality and limitations.
• Present penalization and aggregation perspectives as tuning mechanisms with oracle-inequality implications.
• Note extensions to clustered/panel data and to quantile and generalized linear models.

Experimental results

Research questions

• RQ1 What are the main tuning parameter selection methods applicable to nonparametric series estimators and to high-dimensional Lasso estimation?
• RQ2 Under what regularity conditions do these methods provide (nearly) oracle or asymptotically optimal performance?
• RQ3 How do different methods compare in terms of applicability across metrics (prediction, L2, uniform, pointwise) and data structures (i.i.d., clustered, panel)?
• RQ4 What are the practical considerations for implementing these methods (feasibility, required assumptions, computational issues)?

Key findings

• Mallows and Stein provide unbiased risk estimators that yield asymptotically optimal predictors in prediction and L2 metrics (Mallows is feasible with a practical plug-in form).
• Stein’s method extends to non-linear estimators and requires Gaussian errors, often yielding identical results to Mallows in the series estimator case.
• Lepski’s method delivers adaptation and guarantees in pointwise (and extendable to uniform and L2) metrics, with an adaptation price that depends on the insensitivity region and chosen alpha/beta.
• Cross-validation is universal and practical but may not be fully efficient in some settings (notably in leave-one-out scenarios).
• Penalization and aggregation offer global performance guarantees and can be linked to oracle inequalities; they provide alternatives when unbiased risk estimators are hard to implement.
• The survey also discusses extensions to clustered/panel data and to quantile and generalized linear models, broadening applicability.