[Paper Review] Minimax Estimation of Conditional Moment Models

Nishanth Dikkala, Greg Lewis+2 more|arXiv (Cornell University)|Jun 12, 2020

Statistical Methods and Inference55 references20 citations

TL;DR

This paper introduces a minimax estimation framework for conditional moment models, formulating estimation as a zero-sum game between a modeler and an adversary over hypothesis and test function spaces. It establishes fast, localized estimation rates that scale with the critical radius of these spaces, enabling optimal rates for non-parametric models like RKHS, sparse linear models, random forests, and neural networks under minimal assumptions.

ABSTRACT

We develop an approach for estimating models described via conditional moment restrictions, with a prototypical application being non-parametric instrumental variable regression. We introduce a min-max criterion function, under which the estimation problem can be thought of as solving a zero-sum game between a modeler who is optimizing over the hypothesis space of the target model and an adversary who identifies violating moments over a test function space. We analyze the statistical estimation rate of the resulting estimator for arbitrary hypothesis spaces, with respect to an appropriate analogue of the mean squared error metric, for ill-posed inverse problems. We show that when the minimax criterion is regularized with a second moment penalty on the test function and the test function space is sufficiently rich, then the estimation rate scales with the critical radius of the hypothesis and test function spaces, a quantity which typically gives tight fast rates. Our main result follows from a novel localized Rademacher analysis of statistical learning problems defined via minimax objectives. We provide applications of our main results for several hypothesis spaces used in practice such as: reproducing kernel Hilbert spaces, high dimensional sparse linear functions, spaces defined via shape constraints, ensemble estimators such as random forests, and neural networks. For each of these applications we provide computationally efficient optimization methods for solving the corresponding minimax problem (e.g. stochastic first-order heuristics for neural networks). In several applications, we show how our modified mean squared error rate, combined with conditions that bound the ill-posedness of the inverse problem, lead to mean squared error rates. We conclude with an extensive experimental analysis of the proposed methods.

Motivation & Objective

• To develop a statistical learning theory analogue for non-parametric models defined by moment restrictions, overcoming limitations of traditional GMM.
• To enable estimation in high-dimensional and non-parametric settings using modern machine learning hypothesis classes such as neural networks and random forests.
• To derive fast, finite-sample estimation rates that adapt to the intrinsic complexity of the hypothesis and test function spaces.
• To provide computationally efficient optimization methods for solving the resulting minimax problems in practice.
• To connect the estimation error to the critical radius of function classes, achieving information-theoretically optimal rates.

Proposed method

• Formulates estimation as a min-max optimization: minimize over hypothesis space H and maximize over test function space F to identify worst-case violating moments.
• Introduces a criterion function Ψ(h,f) = E[(y−h(x))f(z)] and solves h₀ = arg inf_h sup_f Ψ(h,f), treating it as a zero-sum game.
• Applies regularization via a second moment penalty on the test function f to ensure stability and finite-sample control.
• Employs localized Rademacher complexity analysis tailored to minimax objectives to derive fast rates.
• Uses kernel approximation (Nystrom method) for scalable computation in RKHS-based models.
• Derives estimation rates by bounding the ill-posedness of the inverse problem through assumptions on instrument strength and eigenstructure of the kernel operator.

Experimental results

Research questions

• RQ1 Can we develop a statistical learning theory framework for non-parametric models defined via conditional moment restrictions, analogous to M-estimation?
• RQ2 How can we achieve fast, finite-sample estimation rates for complex hypothesis classes such as neural networks and random forests under moment restrictions?
• RQ3 What role does the critical radius of the hypothesis and test function spaces play in determining estimation error rates?
• RQ4 How can regularization and adversarial testing improve robustness and adaptivity in ill-posed inverse problems?
• RQ5 Under what conditions does the projected mean squared error rate imply a rate for the actual mean squared error?

Key findings

• The proposed minimax estimator achieves a projected mean squared error rate that scales with the critical radius of the hypothesis and test function spaces, leading to fast, optimal rates.
• When the test function space is rich and regularized with a second moment penalty, the estimation rate adapts to the intrinsic complexity of the model without prior knowledge of the true hypothesis norm.
• For reproducing kernel Hilbert spaces, the method achieves estimation rates dependent on the eigen-decay of the kernel and the strength of the instrument, with explicit bounds derived under mild assumptions.
• In high-dimensional sparse linear models, the method achieves fast rates under sparsity and restricted eigenvalue conditions, with computationally efficient first-order optimization.
• For neural networks and random forests, the framework enables efficient training via stochastic first-order heuristics, with theoretical guarantees on estimation error.
• Under sufficient conditions on instrument strength and eigenstructure (e.g. τₘ and γₘ bounds), the projected RMSE rate implies a rate for the actual RMSE, linking theoretical performance to practical estimation.