[论文解读] Difficult Lessons on Social Prediction from Wisconsin Public Schools

Juan C. Perdomo, Tolani Britton+2 more|arXiv (Cornell University)|Apr 13, 2023

Educational Assessment and Improvement被引用 23

一句话总结

这篇论文评估威斯康星州的 Dropout Early Warning System (DEWS) 对毕业的长期影响，并论证环境因素可以匹配甚至超过针对干预的个体风险分数的定向能力。它还记录了温和、不确定的效应，并强调结构性因素在辍学结果中的主导地位。

ABSTRACT

Early warning systems (EWS) are predictive tools at the center of recent efforts to improve graduation rates in public schools across the United States. These systems assist in targeting interventions to individual students by predicting which students are at risk of dropping out. Despite significant investments in their widespread adoption, there remain large gaps in our understanding of the efficacy of EWS, and the role of statistical risk scores in education. In this work, we draw on nearly a decade's worth of data from a system used throughout Wisconsin to provide the first large-scale evaluation of the long-term impact of EWS on graduation outcomes. We present empirical evidence that the prediction system accurately sorts students by their dropout risk. We also find that it may have caused a single-digit percentage increase in graduation rates, though our empirical analyses cannot reliably rule out that there has been no positive treatment effect. Going beyond a retrospective evaluation of DEWS, we draw attention to a central question at the heart of the use of EWS: Are individual risk scores necessary for effectively targeting interventions? We propose a simple mechanism that only uses information about students' environments -- such as their schools, and districts -- and argue that this mechanism can target interventions just as efficiently as the individual risk score-based mechanism. Our argument holds even if individual predictions are highly accurate and effective interventions exist. In addition to motivating this simple targeting mechanism, our work provides a novel empirical backbone for the robust qualitative understanding among education researchers that dropout is structurally determined. Combined, our insights call into question the marginal value of individual predictions in settings where outcomes are driven by high levels of inequality.

研究动机与目标

• 评估 DEWS 对威斯康星州高中毕业的长期影响。
• 检查 DEWS 风险预测在不同学生群体中的准确性和公平性。
• 研究是否通过基于环境的特征来定向干预，能否达到与 DEWS 相同的效果。
• 使用回归不连续性设计量化将高风险标签分配给学生对毕业的因果效应。
• 提出一种更简单、基于环境的干预定向机制。

提出的方法

• 将 DEWS 视为基于连续风险分数 p(x) 的干预需求排序/排序工具。
• 评估 DEWS 对 8th 年级预测在多个队列中的校准度和辨别力（8th grade 2013–2021 约 n≈215,000）。
• 使用 ROC/AUC 和校准曲线来评估相对风险和绝对风险的准确性（AUC ≈ 0.8；校准往往低估绝对风险）。
• 在 DEWS 阈值 t* = 0.785 附近应用回归不连续性设计以估计将标签提升到高风险对准时毕业的因果影响。
• 在 p(x) 周围定义调整分数 (ℓ(x), u(x))，误差接近常数 e(x) ≈ 0.03，以实现 RD 分析。
• 将仅环境特征的定向与基于 DEWS 的定向进行比较，以确定更简单的、以学区为单位的定向是否足以实现干预效果。

实验结果

研究问题

• RQ1 DEWS 是否能够准确识别辍学风险并按人群与总体相比对学生进行真实风险排序？
• RQ2 根据回归不连续性设计，将更高风险标签的分配是否会因果性地提高按时毕业率？
• RQ3 环境特征（学区/学校层面）是否足以实现有效的干预定向，而不是依赖个体风险分数？
• RQ4 预测准确性和校准在不同人口群体（如种族、残疾、免费午餐状态）之间有何差异？
• RQ5 采用环境定向替代个体风险定向的实际与政策意义是什么？

主要发现

• DEWS 风险类别在实际按时毕业率上显著区分学生（低风险约 97%、高风险<70%；中等风险 83%）。
• DEWS 分数显示出良好的相对辨别能力（AUC ≈ 0.8），但在绝对风险方面的校准不正确，但保持排序不变。
• 回归不连续性分析表明，将学生提升到更高风险标签时，按时毕业的点估计提升约 5%，95% 置信区间为 -2% 至 12%（统计上未能得出结论）。
• 威斯康星州有三分之二的学区定期使用 DEWS，在较大学区以及边缘化人口比例较高的学区中使用率更高。
• 使用学校/学区特征的基于环境的定向机制在干预定向方面可以与 DEWS 同样有效，并且更简单、成本更低、以及更易于辩护。