[论文解读] Machine Learning Who to Nudge: Causal vs Predictive Targeting in a Field Experiment on Student Financial Aid Renewal

Susan Athey, Niall Keleher+1 more|arXiv (Cornell University)|Oct 12, 2023

Advanced Causal Inference Techniques被引用 8

一句话总结

该论文比较因果定向（基于 CATE）与预测定向在 FAFSA 续报中的 nudges 效果，显示结合预测与因果效应的混合模型表现最好，并发现中间基线结果带来最大的政策收益。

ABSTRACT

In many settings, interventions may be more effective for some individuals than others, so that targeting interventions may be beneficial. We analyze the value of targeting in the context of a large-scale field experiment with over 53,000 college students, where the goal was to use "nudges" to encourage students to renew their financial-aid applications before a non-binding deadline. We begin with baseline approaches to targeting. First, we target based on a causal forest that estimates heterogeneous treatment effects and then assigns students to treatment according to those estimated to have the highest treatment effects. Next, we evaluate two alternative targeting policies, one targeting students with low predicted probability of renewing financial aid in the absence of the treatment, the other targeting those with high probability. The predicted baseline outcome is not the ideal criterion for targeting, nor is it a priori clear whether to prioritize low, high, or intermediate predicted probability. Nonetheless, targeting on low baseline outcomes is common in practice, for example because the relationship between individual characteristics and treatment effects is often difficult or impossible to estimate with historical data. We propose hybrid approaches that incorporate the strengths of both predictive approaches (accurate estimation) and causal approaches (correct criterion); we show that targeting intermediate baseline outcomes is most effective in our specific application, while targeting based on low baseline outcomes is detrimental. In one year of the experiment, nudging all students improved early filing by an average of 6.4 percentage points over a baseline average of 37% filing, and we estimate that targeting half of the students using our preferred policy attains around 75% of this benefit.

研究动机与目标

• 衡量 FAFSA 申报 nudge 效应的异质性。
• 基于估计的处理效应与基线预测分数评估定向策略。
• 评估将预测与因果推断结合的混合方法以用于政策定向。
• 在预算约束下就何时以及如何定向以最大化政策影响提供指南。

提出的方法

• 分析涉及 53,000 多名学生的两项大型随机对照试验（2017 年和 2018 年）的数据。
• 使用因果森林（广义随机森林）估计条件平均处理效应（CATE）。
• 使用早期（学期前）和晚期协变量；应用反向概率加权和增强反向倾向加权（AIPW）。
• 通过交叉拟合和 Chernozhukov 等人（2019）诊断评估 CATE 的校准性；对分组估计使用排序组平均处理效应（GATES）以获得无偏的分组估计。
• 通过将 tau(x) 大于阈值的个体分配处理来评估定向策略，并与基于基线申报概率的预测模型进行比较。
• 开发并比较将预测的基线结果与非参数 CATE 估计相结合的混合模型。

实验结果

研究问题

• RQ1 nudges 是否在学生之间存在异质效应，哪些协变量驱动这种异质性？
• RQ2 基于估计的处理效应的因果定向策略相对于基于基线申报概率的预测定向策略表现如何？
• RQ3 结合预测与因果估计的混合方法是否提高了定向性能？
• RQ4 哪些入学或状态因素会影响提醒和定向决策的有效性？

主要发现

• nudges 在 2017 年将早期 FAFSA 申报提高了 6.4 个百分点，在 2018 年提高了 12.1 个百分点（基线申报率分别为 37% 和 38%）。
• 入学状态对处理效应具有高度预测性，已入学学生之间存在额外的异质性。
• 将基线预测结果与非参数 CATE 估计相结合的混合模型优于纯非参数模型和纯预测定向。
• 基于中间预测的基线结果定向比针对最低或最高基线结果组定向带来更大收益；对最低基线结果的定向表现不佳，而对最高基线结果的定向可能有效但并非最优。
• 纯预测定向可能不如因果定向，强调在政策定向中整合因果推断与机器学习的价值。