[Paper Review] Digital interventions and habit formation in educational technology

Keshav Agrawal, Susan Athey+2 more|arXiv (Cornell University)|Oct 16, 2023

Child Development and Digital Technology4 citations

TL;DR

This study evaluates a contest-based digital intervention in an Indian English reading app, finding that a 12-week leaderboard competition with book prizes increased story consumption by 75% post-intervention and improved retention by 1.3 percentage points, suggesting such gamified incentives can successfully form lasting reading habits without crowding out intrinsic motivation.

ABSTRACT

As online educational technology products have become increasingly prevalent, rich evidence indicates that learners often find it challenging to establish regular learning habits and complete their programs. Concurrently, online products geared towards entertainment and social interactions are sometimes so effective in increasing user engagement and creating frequent usage habits that they inadvertently lead to digital addiction, especially among youth. In this project, we carry out a contest-based intervention, common in the entertainment context, on an educational app for Indian children learning English. Approximately ten thousand randomly selected learners entered a 100-day reading contest. They would win a set of physical books if they ranked sufficiently high on a leaderboard based on the amount of educational content consumed. Twelve weeks after the end of the contest, when the treatment group had no additional incentives to use the app, they continued their engagement with it at a rate 75\% higher than the control group, indicating a successful formation of a reading habit. In addition, we observed a 6\% increase in retention within the treatment group. These results underscore the potential of digital interventions in fostering positive engagement habits with educational technology products, ultimately enhancing users' long-term learning outcomes.

Motivation & Objective

• To assess whether a contest-based digital intervention can instill long-term reading habits in children using an educational app.
• To examine whether external incentives like prizes crowd out intrinsic motivation or instead reinforce it.
• To evaluate the sustained impact of a gamified intervention on app usage and retention after incentives are removed.
• To provide empirical evidence on scalable, low-cost interventions in digital education for low-income contexts.
• To explore the effectiveness of gamification in promoting consistent engagement in EdTech platforms.

Proposed method

• A randomized controlled trial was conducted with approximately 10,000 children across seven batches of school-based enrollments.
• Treated students were enrolled in a '100 days reading contest' with a leaderboard and top 1,000 winners receiving books.
• Outcome variables included number of stories read and app retention, measured over 12 weeks post-contest with no further incentives.
• Poisson regression was used to model the number of difficult stories consumed, with treatment as a key predictor.
• Cox proportional hazards models were applied to estimate retention probabilities, accounting for time-to-dropout.
• Data was anonymized and processed to exclude extreme outliers (top 0.1% in story completion) to reduce instrumentation errors.

Experimental results

Research questions

• RQ1 Does a contest-based digital intervention increase short-term engagement in an educational reading app?
• RQ2 Can such an intervention lead to sustained increases in app usage after incentives are removed?
• RQ3 Does the use of external rewards in a gamified setting undermine intrinsic motivation to read?
• RQ4 To what extent does the intervention improve long-term retention in the absence of ongoing incentives?
• RQ5 Can low-cost, scalable digital interventions effectively form reading habits in children from low-income educational settings?

Key findings

• During the 12-week post-contest period with no further incentives, treated students read 0.30 more stories on average than the control group (p < 0.05).
• The treatment group read 75% more stories than the control group during the post-contest phase, with the control group averaging 0.40 stories.
• Treated students were 1.3 percentage points more likely to remain active in the app over 12 weeks, relative to the control group’s 19% retention rate.
• The intervention was effective during the contest, with treated students reading 0.96 more stories than controls (p < 0.05), a 45% increase over the control mean of 2.12.
• There was no statistically significant difference in pre-randomization app usage between treatment and control groups (p = 0.316), supporting randomization balance.
• The results suggest that the contest did not crowd out intrinsic motivation, as engagement remained higher after incentives ended.