[Paper Review] Inferactive data analysis

Nan Bi, Jelena Markovic+2 more|arXiv (Cornell University)|Jul 20, 2017

Gaussian Processes and Bayesian Inference6 citations

TL;DR

This paper introduces 'inferactive data analysis,' a framework that enables valid statistical inference after interactive, data-driven model selection. By combining selective inference with a DAG-DAG model of queries and data, it provides frequentist guarantees—such as valid p-values and confidence intervals—through randomized selective inference, even when models are chosen post-hoc based on data exploration.

ABSTRACT

We describe inferactive data analysis, so-named to denote an interactive approach to data analysis with an emphasis on inference after data analysis. Our approach is a compromise between Tukey's exploratory (roughly speaking "model free") and confirmatory data analysis (roughly speaking classical and "model based"), also allowing for Bayesian data analysis. We view this approach as close in spirit to current practice of applied statisticians and data scientists while allowing frequentist guarantees for results to be reported in the scientific literature, or Bayesian results where the data scientist may choose the statistical model (and hence the prior) after some initial exploratory analysis. While this approach to data analysis does not cover every scenario, and every possible algorithm data scientists may use, we see this as a useful step in concrete providing tools (with frequentist statistical guarantees) for current data scientists. The basis of inference we use is selective inference [Lee et al., 2016, Fithian et al., 2014], in particular its randomized form [Tian and Taylor, 2015a]. The randomized framework, besides providing additional power and shorter confidence intervals, also provides explicit forms for relevant reference distributions (up to normalization) through the {\em selective sampler} of Tian et al. [2016]. The reference distributions are constructed from a particular conditional distribution formed from what we call a DAG-DAG -- a Data Analysis Generative DAG. As sampling conditional distributions in DAGs is generally complex, the selective sampler is crucial to any practical implementation of inferactive data analysis. Our principal goal is in reviewing the recent developments in selective inference as well as describing the general philosophy of selective inference.

Motivation & Objective

• To address the 'quiet scandal of statistics'—where classical inference fails when hypotheses are formed after data exploration—by providing a principled framework for inference after interactive data analysis.
• To reconcile exploratory data analysis with formal statistical inference by allowing data scientists to choose models and hypotheses based on observed data while still maintaining statistical validity.
• To develop a practical, computationally feasible approach to inference that supports common data science workflows, including LASSO, marginal screening, and forward stepwise selection.
• To extend selective inference to multi-step, interactive data analysis through a DAG-DAG representation of query dependencies and selection events.
• To demonstrate the framework's utility with real data, including a real HIV dataset, and provide open-source implementations for reproducibility.

Proposed method

• The framework uses randomized selective inference (Tian & Taylor, 2015a) to construct valid reference distributions for inference after data-dependent model selection.
• It introduces a DAG-DAG (Data Analysis Generative Directed Acyclic Graph) to model the sequence of queries and their dependencies, enabling conditional inference based on the path of selections.
• The selective sampler from Tian et al. (2016) is used to generate conditional distributions under selection events, making inference computationally feasible despite complex conditioning.
• For each query, the selection event is defined via optimization conditions (e.g., active set and signs in LASSO), and the joint conditional density is derived using reconstruction maps and Jacobian terms.
• The method supports multi-step inference by composing selection events across queries, with the final density proportional to the product of data density, prior densities on auxiliary variables, and Jacobian terms.
• The framework handles high-dimensional regression by using asymptotic normality of pre-selection statistics and applying the selective central limit theorem for inference.

Experimental results

Research questions

• RQ1 How can valid statistical inference be performed when the statistical model and hypotheses are selected after observing the data?
• RQ2 What is a practical and generalizable framework for inference after interactive, data-driven model selection in real-world data science workflows?
• RQ3 How can selective inference be extended to multi-step, sequential queries involving complex algorithms like LASSO and marginal screening?
• RQ4 What role does randomization play in improving power and computational feasibility in post-selection inference?
• RQ5 Can the DAG-DAG representation effectively model the dependency structure of queries and selection events to enable scalable and interpretable inference?

Key findings

• The inferactive data analysis framework successfully provides valid p-values and confidence intervals after two-stage queries on a real HIV dataset, including marginal screening followed by LASSO.
• Randomized selective inference increases statistical power and shortens confidence intervals compared to non-randomized counterparts, as demonstrated in the LASSO-based analysis.
• The selective sampler enables efficient sampling from complex conditional distributions, making the framework computationally viable for practical use.
• The DAG-DAG model effectively captures the dependency structure of sequential queries, allowing for modular and extensible inference across multiple data analysis steps.
• The framework supports a wide range of algorithms—including LASSO, group LASSO, forward stepwise, and their combinations—by expressing selection events in terms of optimization conditions.
• The implementation is publicly available on GitHub, enabling reproducibility and adoption by the data science community.