[Paper Review] Difficult Lessons on Social Prediction from Wisconsin Public Schools

Juan C. Perdomo, Tolani Britton+2 more|arXiv (Cornell University)|Apr 13, 2023

Educational Assessment and Improvement23 citations

TL;DR

This paper evaluates the long-term impact of Wisconsin’s Dropout Early Warning System (DEWS) on graduation and argues that environmental features can match or exceed the targeting power of individual risk scores for interventions. It also documents modest, uncertain effects and highlights the primacy of structural factors in dropout outcomes.

ABSTRACT

Early warning systems (EWS) are predictive tools at the center of recent efforts to improve graduation rates in public schools across the United States. These systems assist in targeting interventions to individual students by predicting which students are at risk of dropping out. Despite significant investments in their widespread adoption, there remain large gaps in our understanding of the efficacy of EWS, and the role of statistical risk scores in education. In this work, we draw on nearly a decade's worth of data from a system used throughout Wisconsin to provide the first large-scale evaluation of the long-term impact of EWS on graduation outcomes. We present empirical evidence that the prediction system accurately sorts students by their dropout risk. We also find that it may have caused a single-digit percentage increase in graduation rates, though our empirical analyses cannot reliably rule out that there has been no positive treatment effect. Going beyond a retrospective evaluation of DEWS, we draw attention to a central question at the heart of the use of EWS: Are individual risk scores necessary for effectively targeting interventions? We propose a simple mechanism that only uses information about students' environments -- such as their schools, and districts -- and argue that this mechanism can target interventions just as efficiently as the individual risk score-based mechanism. Our argument holds even if individual predictions are highly accurate and effective interventions exist. In addition to motivating this simple targeting mechanism, our work provides a novel empirical backbone for the robust qualitative understanding among education researchers that dropout is structurally determined. Combined, our insights call into question the marginal value of individual predictions in settings where outcomes are driven by high levels of inequality.

Motivation & Objective

• Assess the long-term impact of DEWS on high school graduation in Wisconsin.
• Examine the accuracy and fairness of DEWS risk predictions across student groups.
• Investigate whether targeting interventions by environment-based features can match DEWS’s outcomes.
• Quantify the causal effect of assigning high-risk labels on graduation using a regression discontinuity design.
• Propose a simpler, environment-based targeting mechanism for interventions.

Proposed method

• Treat DEWS as a ranking/sorting tool for intervention need based on a continuous risk score p(x).
• Evaluate calibration and discrimination of DEWS 8th-grade predictions across multiple cohorts (n≈215,000 for 8th grade 2013–2021).
• Use ROC/AUC and calibration curves to assess relative risk and absolute risk accuracy (AUC ≈ 0.8; calibration underestimates absolute risk).
• Apply regression discontinuity design around the DEWS threshold t* = 0.785 to estimate the causal impact of higher-risk labeling on on-time graduation.
• Define adjusted scores (ℓ(x), u(x)) around p(x) with near-constant error e(x) ≈ 0.03 to implement the RD analysis.
• Compare environmental-feature-only targeting to DEWS-based targeting to determine if simpler, district-level targeting suffices for intervention efficacy.

Experimental results

Research questions

• RQ1 Does DEWS accurately identify dropout risk and rank students by their true risk across the population and marginalized groups?
• RQ2 Do higher DEWS risk labels causally improve on-time graduation rates according to a regression discontinuity design?
• RQ3 Are environmental (district/school-level) features sufficient for effective intervention targeting versus individual risk scores?
• RQ4 How do predictive accuracies and calibration differ across demographic groups (e.g., race, disability, free lunch status)?
• RQ5 What are the practical and policy implications of adopting environment-based targeting over individual-risk-based targeting?

Key findings

• DEWS risk categories meaningfully separate students by actual on-time graduation rates (≈97% for low-risk vs <70% for high-risk; 83% for moderate-risk).
• DEWS scores show good relative discrimination (AUC ≈ 0.8) but miscalibrated in absolute risk, yet rank-preserving.
• Regression discontinuity suggests a point estimate of about a 5% uplift in on-time graduation when moving a student to a higher-risk label, with a 95% CI of -2% to 12% (not statistically conclusive).
• Two-thirds of Wisconsin districts regularly use DEWS, with higher usage in larger districts and among districts with higher marginalized populations.
• An environment-based targeting mechanism using school/district features can be as effective as DEWS for intervention targeting, and it is simpler, cheaper, and easier to justify.