[Paper Review] A Note on "How Robust Standard Errors Expose Methodological Problems They Do Not Fix, and What to Do About It"

Peter M. Aronow|arXiv (Cornell University)|Sep 6, 2016

Advanced Causal Inference Techniques19 references3 citations

TL;DR

This paper challenges King and Roberts' (2015) claim that discrepancies between robust and classical standard errors signal model misspecification, arguing instead that such differences are common and expected in semiparametric models—especially linear models with OLS estimation—where classical errors are inconsistent under heteroskedasticity. The authors demonstrate via simulation that robust standard errors correctly reflect sampling variability while classical ones are severely biased, showing KR’s diagnostic heuristic can mislead rather than expose problems.

ABSTRACT

King and Roberts (2015, KR) claim that a disagreement between robust and classical standard errors exposes model misspecification. We emphasize that KR's claim only generally applies to parametric models: models that assume a restrictive form of the distribution of the outcome. Many common models in use in political science, including the linear model, are not necessarily parametric -- rather they may be semiparametric. Common estimators of model parameters such as ordinary least squares have both robust (corresponding to a semiparametric model) and classical (corresponding to a more restrictive model) standard error estimates. Given a properly specified semiparametric model and mild regularity conditions, the classical standard errors are not generally consistent, but the robust standard errors are. To illustrate this point, we consider the case of the regression estimate of a semiparametric linear model with no model misspecification, and show that robust standard errors may nevertheless systematically differ from classical standard errors. We show that a disagreement between robust and classical standard errors is not generally suitable as a diagnostic for regression estimators, and that KR's reanalyses of Neumayer (2003) and Büthe and Milner (2008) are predicated on strong assumptions that the original authors did not invoke nor require.

Motivation & Objective

• To challenge the generalizability of King and Roberts' (2015) diagnostic heuristic that differences between robust and classical standard errors indicate model misspecification.
• To clarify the distinction between parametric and semiparametric models, emphasizing that many widely used models in political science are semiparametric, not parametric.
• To demonstrate that in properly specified semiparametric linear models, robust standard errors may systematically differ from classical ones due to model assumptions, not misspecification.
• To show that KR’s reanalyses of Neumayer (2003) and Büthe and Milner (2008) rely on parametric assumptions not invoked by the original authors, undermining their diagnostic validity.
• To caution against data-adaptive model respecification based on standard error discrepancies, as it risks data snooping and inflated Type I error rates.

Proposed method

• Define parametric models as those assuming a finite-dimensional parametric form for the outcome distribution, and semiparametric models as those not restricting the distributional form beyond conditional mean structure.
• Use a Monte Carlo simulation with n=1000 and m=200 to compare sampling distributions of robust and classical standard errors under a heteroskedastic linear model.
• Compute expected values of robust and classical standard error estimators: E[√V_Het(β̂₁)] = 0.206 and E[√V_C(β̂₁)] = 0.141, showing classical errors are severely biased.
• Argue that classical standard errors are inconsistent under heteroskedasticity because they rely on the false assumption of homoskedasticity, while robust standard errors remain consistent.
• Apply Manski’s Law of Decreasing Credibility to argue that semiparametric models are more credible than parametric models without strong theoretical justification.
• Critique KR’s recommendation to respecify models based on standard error disagreement as a form of data snooping that undermines inferential validity.

Experimental results

Research questions

• RQ1 Under what conditions do robust and classical standard errors differ in the absence of model misspecification?
• RQ2 Why is the classical standard error inconsistent in semiparametric linear models with heteroskedasticity?
• RQ3 To what extent are King and Roberts’ reanalyses of Neumayer (2003) and Büthe and Milner (2008) valid under the original authors’ modeling assumptions?
• RQ4 Can a discrepancy between robust and classical standard errors reliably diagnose model misspecification in semiparametric models?
• RQ5 What are the implications of KR’s diagnostic heuristic for statistical inference and model credibility in quantitative political science?

Key findings

• In a properly specified semiparametric linear model with heteroskedasticity, classical standard errors are severely biased (E[√V_C(β̂₁)] = 0.141), while robust standard errors are approximately unbiased (E[√V_Het(β̂₁)] = 0.206).
• The discrepancy between robust and classical standard errors in this setting is not a sign of model misspecification but a consequence of the classical estimator’s reliance on the incorrect homoskedasticity assumption.
• KR’s diagnostic heuristic—using differences between robust and classical standard errors to flag model problems—can mislead researchers, as it may signal a problem where none exists.
• KR’s reanalyses of Neumayer (2003) and Büthe and Milner (2008) are predicated on strong parametric assumptions not present in the original studies, rendering their diagnostics inapplicable.
• Respecifying models based on standard error discrepancies risks data snooping, as the model selection process adapts to the data, inflating Type I error and undermining inferential validity.
• Semiparametric models are more credible than parametric models under Manski’s Law of Decreasing Credibility, as they make fewer restrictive assumptions about the data-generating process.