[Paper Review] Authoring Programming Exercises for Automated Assessment Assisted by Generative AI

Erik Brynjolfsson, Danielle Li+1 more|arXiv (Cornell University)|Apr 23, 2023

Digital Economy and Work Transformation13 citations

TL;DR

This study evaluates a generative AI chat assistant deployed at scale in a customer support environment, showing it boosts worker productivity by 15% on average, with larger gains for less-experienced and lower-skilled agents. The AI enhances speed, quality, and learning, particularly for novices, while improving customer sentiment and reducing attrition.

ABSTRACT

Generative AI presents both challenges and opportunities for educators. This paper explores its potential for automating the creation of programming exercises designed for automated assessment. Traditionally, creating these exercises is a time-intensive and error-prone task that involves developing exercise statements, solutions, and test cases. This ongoing research analyzes the capabilities of the OpenAI GPT API to automatically create these components. An experiment using the OpenAI GPT API to automatically create 120 programming exercises produced interesting results, such as the difficulties encountered in generating valid JSON formats and creating matching test cases for solution code. Learning from this experiment, an enhanced feature was developed to assist teachers in creating programming exercises and was integrated into Agni, a virtual learning environment (VLE). Despite the challenges in generating entirely correct programming exercises, this approach shows potential for reducing the time required to create exercises, thus significantly aiding teachers. The evaluation of this approach, comparing the efficiency and usefulness of using the OpenAI GPT API or authoring the exercises oneself, is in progress.

Motivation & Objective

• To assess the real-world impact of generative AI on worker productivity in a large-scale customer service setting.
• To examine how AI assistance affects workers of varying experience and skill levels.
• To investigate whether AI adoption facilitates worker learning and improves job experience metrics such as customer sentiment and attrition.
• To explore the mechanisms behind productivity gains, including adherence to AI recommendations and durable learning effects.
• To evaluate the long-term implications for organizational design, worker compensation, and labor market equilibrium.

Proposed method

• Conducted a staggered deployment of a GPT-based conversational AI assistant across 5,172 customer support agents at a Fortune 500 firm.
• Used panel data with fixed effects to estimate causal impacts on productivity, quality, and attrition, controlling for time, location, and agent-specific effects.
• Measured productivity as issues resolved per hour, with decomposition into handling time, multitasking capacity, and resolution rate.
• Analyzed outage periods—when AI was unavailable—to isolate durable learning effects from reliance on AI.
• Tracked adherence to AI recommendations and correlated it with productivity gains, especially over time.
• Used attrition data to assess long-term worker experience, comparing treated and untreated agents with equal tenure.

Experimental results

Research questions

• RQ1 What is the impact of generative AI on worker productivity in a real-world customer support setting?
• RQ2 How do productivity gains from AI vary across workers of different experience and skill levels?
• RQ3 To what extent do AI recommendations lead to durable learning, independent of ongoing AI assistance?
• RQ4 How does AI adoption affect customer sentiment and worker attrition?
• RQ5 What are the implications of AI use for worker compensation, organizational design, and labor market equilibrium?

Key findings

• AI assistance increased worker productivity by 15% on average, measured as issues resolved per hour.
• Less-experienced and lower-skilled agents saw a 30% increase in issues resolved per hour, closing the performance gap with more experienced peers.
• More-experienced and higher-skilled agents showed minimal gains in speed and a small decline in conversation quality.
• Productivity gains persisted during AI outages, especially among agents who had previously followed AI recommendations closely.
• AI adoption reduced attrition by 10 percentage points among agents with less than six months of tenure—equivalent to a 40% reduction relative to baseline.
• AI assistance improved customer sentiment, with customers being less likely to request to speak to a manager and more likely to be polite.