[Paper Review] Handling Attrition in Longitudinal Studies: The Case for Refreshment Samples

Yiting Deng, D. Sunshine Hillygus+3 more|arXiv (Cornell University)|Jun 12, 2013

Electoral Systems and Political Participation93 references111 citations

TL;DR

This paper advocates for the use of refreshment samples—new, randomly selected respondents surveyed alongside later waves of a panel—to diagnose and correct for bias due to non-ignorable attrition in longitudinal studies. It proposes Bayesian and multiple imputation methods to combine panel and refreshment data, demonstrating through simulation and analysis of the 2007–2008 AP-Yahoo! News Election Poll that refreshment samples significantly improve bias correction and variance estimation, especially when attrition is non-ignorable.

ABSTRACT

Panel studies typically suffer from attrition, which reduces sample size and can result in biased inferences. It is impossible to know whether or not the attrition causes bias from the observed panel data alone. Refreshment samples - new, randomly sampled respondents given the questionnaire at the same time as a subsequent wave of the panel - offer information that can be used to diagnose and adjust for bias due to attrition. We review and bolster the case for the use of refreshment samples in panel studies. We include examples of both a fully Bayesian approach for analyzing the concatenated panel and refreshment data, and a multiple imputation approach for analyzing only the original panel. For the latter, we document a positive bias in the usual multiple imputation variance estimator. We present models appropriate for three waves and two refreshment samples, including nonterminal attrition. We illustrate the three-wave analysis using the 2007-2008 Associated Press-Yahoo! News Election Poll.

Motivation & Objective

• To address the critical challenge of non-ignorable attrition in longitudinal panel studies, which can lead to biased inferences and reduced statistical power.
• To demonstrate that refreshment samples—new respondents surveyed at later waves—provide essential external information to diagnose and correct for attrition bias.
• To develop and validate statistical models, including Bayesian and multiple imputation approaches, for integrating panel and refreshment data in three-wave settings with nonterminal attrition.
• To evaluate the performance of standard multiple imputation variance estimators, revealing a positive bias in such estimators when applied to attrited panel data.
• To encourage data collectors and researchers to adopt refreshment samples as a standard practice to enhance the validity and robustness of longitudinal survey analysis.

Proposed method

• Proposes a fully Bayesian approach to jointly model the original panel and refreshment samples, allowing for flexible modeling of attrition mechanisms and outcome distributions.
• Introduces a multiple imputation framework that uses refreshment sample data to inform imputations for attrited panel members, improving bias correction in complete-case analyses.
• Develops models for three-wave panel studies with two refreshment samples, explicitly accounting for nonterminal attrition (i.e., respondents who drop out and later return).
• Employs conditional probability models and regression-based imputation to ensure compatibility between observed panel data, refreshment samples, and imputed values.
• Performs diagnostic checks by comparing imputed data distributions to observed data in both the panel and refreshment samples to assess model fit.
• Conducts simulations and applies the methods to real data from the 2007–2008 AP-Yahoo! News Election Poll to validate the approach.

Experimental results

Research questions

• RQ1 Can refreshment samples provide sufficient external information to detect and correct for non-ignorable attrition bias in longitudinal panel studies?
• RQ2 How do Bayesian and multiple imputation methods compare in their ability to correct for attrition bias when combined with refreshment sample data?
• RQ3 What is the impact of nonterminal attrition—where respondents return after dropping out—on the performance of attrition correction models?
• RQ4 Does the standard multiple imputation variance estimator remain valid when applied to panel data with non-ignorable attrition, or is it systematically biased?
• RQ5 To what extent can refreshment samples reduce bias in longitudinal inferences when the initial panel is subject to non-ignorable nonresponse?

Key findings

• The standard multiple imputation variance estimator exhibits a positive bias when applied to panel data with non-ignorable attrition, undermining the precision of statistical inference.
• The Bayesian and multiple imputation methods that incorporate refreshment sample data produce imputations that are highly compatible with both the observed panel and refreshment sample data, indicating good model fit.
• In the analysis of the 2007–2008 AP-Yahoo! News Election Poll, the use of refreshment samples led to more accurate estimates of candidate support trends over time compared to complete-case analysis.
• The models successfully accounted for nonterminal attrition, showing that respondents who re-enter the panel can be meaningfully incorporated into the analysis using refreshment sample information.
• Sensitivity analyses revealed that the validity of the results depends critically on assumptions about the missing data mechanism, particularly regarding interaction effects in selection models.
• The study concludes that refreshment samples significantly reduce reliance on untestable assumptions about attrition mechanisms, thereby improving the robustness of longitudinal inferences.