[Paper Review] Some discussions on the Read Paper "Beyond subjective and objective in statistics" by A. Gelman and C. Hennig

Gilles Celeux, Jack Jewson+3 more|arXiv (Cornell University)|May 10, 2017

Advanced Statistical Methods and Models4 references3 citations

TL;DR

This paper critically examines the debate over subjectivity versus objectivity in statistical practice, arguing that statistical inference inherently involves subjective choices in modeling, priors, and tuning parameters. It advocates for greater transparency in reporting all such choices—both conscious and implicit—to enhance reproducibility, model awareness, and epistemic rigor, while challenging the illusion of objectivity in statistical methodology.

ABSTRACT

This note is a collection of several discussions of the paper "Beyond subjective and objective in statistics", read by A. Gelman and C. Hennig to the Royal Statistical Society on April 12, 2017, and to appear in the Journal of the Royal Statistical Society, Series A.

Motivation & Objective

• To challenge the entrenched dichotomy between subjective and objective statistics, arguing it oversimplifies the complexity of statistical practice.
• To promote transparency in statistical methodology by explicitly documenting all operator-dependent choices, including priors, model assumptions, and tuning parameters.
• To highlight the limitations of current statistical frameworks—especially frequentist and Bayesian paradigms—when reduced to narrow interpretations like asymptotics or maxent priors.
• To address the epistemological gap in statistical practice, where assumptions are often hidden or unacknowledged, undermining reproducibility and critical evaluation.
• To advocate for a model-aware, critical rationalist approach to statistics, where models are treated as falsifiable and revisable, not as absolute truths.

Proposed method

• Critically analyzes the foundational assumptions of statistical inference, particularly the reliance on randomness and probabilistic generative models.
• Examines the role of subjective choices in model selection, prior specification, and parameter tuning, arguing these are unavoidable and should be made explicit.
• Proposes a shift from a false dichotomy of 'subjective vs. objective' to a framework emphasizing transparency, replicability, and model awareness.
• Introduces the concept of 'forking paths' in statistical analysis, where multiple decisions during model construction lead to divergent conclusions.
• Advocates for the inclusion of code and data sharing in publications to enable scrutiny of all methodological choices.
• Draws on critical rationalism (Popperian falsificationism) to frame statistical models as hypotheses to be tested and updated, not as fixed truths.

Experimental results

Research questions

• RQ1 How can statistical practice be reformed to make all subjective choices—explicit and implicit—transparent and accountable?
• RQ2 To what extent does the illusion of objectivity in statistics stem from unacknowledged model assumptions and tuning parameters?
• RQ3 Why is the traditional distinction between subjective and objective statistics epistemologically flawed and unhelpful in real-world applications?
• RQ4 How can statistical methodology better accommodate the realities of model uncertainty, especially in non-experimental or M-open settings?
• RQ5 What role do software tools and default settings play in perpetuating flawed but widely used statistical practices?

Key findings

• Statistical inference is inherently subjective due to unavoidable choices in model specification, prior selection, and tuning parameters, even in frequentist and Bayesian frameworks.
• The assumption of randomness in data is philosophically fragile and often unverifiable, undermining the foundation of probabilistic modeling.
• Many statistical methods, especially in machine learning and nonparametric statistics, are not inherently more objective, despite their perceived neutrality.
• The current statistical ecosystem encourages 'point-and-shoot' analysis through user-friendly software, which masks underlying assumptions and promotes a false sense of objectivity.
• The proposal to mandate full documentation of all methodological choices—including code and data—faces practical barriers due to low adoption among practitioners and journals.
• A critical, falsificationist approach to modeling—where models are treated as hypotheses to be tested and updated—offers a more defensible epistemological foundation than traditional dichotomies.