[Paper Review] Expanding tidy data principles to facilitate missing data exploration, visualization and assessment of imputations

Nicholas Tierney, Dianne H Cook|arXiv (Cornell University)|Sep 6, 2018

Statistical Methods and Bayesian InferenceMathematics22 citations

TL;DR

This paper introduces the naniar R package, extending tidy data principles to streamline missing data exploration, visualization, and imputation. By defining a new 'nabular' data structure and integrating with tidyverse tools, it enables consistent, workflow-friendly handling of missing values through dedicated functions for visualization (e.g., geom_miss_point), numerical summaries, and imputation, improving transparency and reproducibility in data analysis pipelines.

ABSTRACT

Despite the large body of research on missing value distributions and imputation, there is comparatively little literature with a focus on how to make it easy to handle, explore, and impute missing values in data. This paper addresses this gap. The new methodology builds upon tidy data principles, with the goal of integrating missing value handling as a key part of data analysis workflows. We define a new data structure, and a suite of new operations. Together, these provide a connected framework for handling, exploring, and imputing missing values. These methods are available in the R package `naniar`.

Motivation & Objective

• Address the lack of integrated, principled tools for handling missing data within the tidy data framework.
• Bridge the gap between missing data exploration and standard data analysis workflows in R.
• Develop a cohesive, reusable framework that supports visualization, summary, and imputation of missing values using tidy tools.
• Enable analysts to fluently explore missingness patterns and assess imputation quality within a single, consistent pipeline.

Proposed method

• Introduce a new 'nabular' data structure that extends tidy data principles to explicitly represent missing values as first-class entities.
• Implement a suite of functions in the naniar R package that integrate with dplyr, tidyr, and ggplot2 for consistent data manipulation and visualization.
• Develop specialized visualization tools such as geom_miss_point to display missing values directly in plots, preserving all observations.
• Provide numerical summary functions (e.g., missingness patterns, frequency, distribution) to assess missing data structure.
• Support multiple imputation methods (e.g., KNN, linear models) with consistent output formats compatible with downstream modeling.
• Enable assessment of imputed datasets through coefficient comparison and residual analysis across imputed versions.

Experimental results

Research questions

• RQ1 How can missing data be visualized in a way that preserves all observations and reveals patterns of missingness without dropping values?
• RQ2 Can missing data handling be integrated into the tidy data workflow to improve reproducibility and reduce error in data analysis?
• RQ3 How can imputed datasets be compared and assessed for consistency and reliability using visualization and numerical summaries?
• RQ4 What role do specialized data structures play in enabling consistent, modular, and composable missing data operations?

Key findings

• The naniar package successfully extends tidy data principles to missing data, enabling seamless integration of missingness exploration into data analysis pipelines.
• Visualizations such as geom_miss_point allow missing values to be explicitly shown in plots, avoiding the loss of information common in standard plotting tools.
• Imputed datasets generated using KNN or linear models showed distinct differences in distributional characteristics—e.g., linear model imputation reduced extreme values in 'bedrooms' and 'cars' variables.
• Coefficient estimates for 'number of rooms' in a log-price model were systematically lower in complete-case analysis compared to imputed datasets, indicating potential bias in case-deletion approaches.
• Partial residual plots revealed that imputed datasets produced more clustered predictions around zero compared to complete cases, suggesting more stable model behavior.
• The framework supports interactive and animated exploration (e.g., via plotly and gganimate), enabling dynamic assessment of missingness and imputation effects.