[Paper Review] How much does AI impact development speed? An enterprise-based randomized controlled trial

Elise Paradis, Kate Grey+7 more|arXiv (Cornell University)|Oct 16, 2024

Artificial Intelligence in Healthcare and EducationMedicine3 citations

TL;DR

This enterprise-based randomized controlled trial with 96 Google software engineers evaluates the impact of three AI coding features—AI Code Completion, Smart Paste, and Natural Language to Code—on development speed. The study finds that AI reduced time on task by an estimated 21%, with a notable interaction effect showing more experienced developers benefiting more from AI assistance, though confidence intervals are wide.

ABSTRACT

How much does AI assistance impact developer productivity? To date, the software engineering literature has provided a range of answers, targeting a diversity of outcomes: from perceived productivity to speed on task and developer throughput. Our randomized controlled trial with 96 full-time Google software engineers contributes to this literature by sharing an estimate of the impact of three AI features on the time developers spent on a complex, enterprise-grade task. We found that AI significantly shortened the time developers spent on task. Our best estimate of the size of this effect, controlling for factors known to influence developer time on task, stands at about 21\%, although our confidence interval is large. We also found an interesting effect whereby developers who spend more hours on code-related activities per day were faster with AI. Product and future research considerations are discussed. In particular, we invite further research that explores the impact of AI at the ecosystem level and across multiple suites of AI-enhanced tools, since we cannot assume that the effect size obtained in our lab study will necessarily apply more broadly, or that the effect of AI found using internal Google tooling in the summer of 2024 will translate across tools and over time.

Motivation & Objective

• To estimate the impact of AI-enhanced developer tools on time spent completing a complex, enterprise-grade software development task.
• To investigate how developer-level and task-level characteristics moderate the effect of AI on development speed.
• To explore whether AI use patterns or user expertise influence productivity gains, especially in high-fidelity, real-world development environments.
• To inform product design and future research on AI tools by identifying differential impacts across developer experience levels and workflow contexts.
• To address gaps in existing literature by using a controlled, large-scale study in a real enterprise setting with realistic tasks and tools.

Proposed method

• Conducted a randomized controlled trial (RCT) with 96 full-time Google software engineers, randomly assigned to AI or control groups.
• Used proprietary internal tools: AI Code Completion, Smart Paste, and Natural Language to Code, integrated into the developers’ standard workflow.
• Measured time on task for a complex, enterprise-grade development task involving code writing, build file updates, and testing in a real infrastructure.
• Applied t-tests and linear regression to assess the main effect of AI on time on task, controlling for developer and task-level covariates.
• Used multivariate regression based on a theoretical framework to test robustness and interaction effects.
• Analyzed usage logs and self-reported data to explore relationships between AI usage frequency and performance, despite low statistical power.

Figure 1: AI Code Completion in Cider V. When a user starts typing code, the feature auto-completes the code block in light-grey font based on the context provided. After typing the first line of a new function to be evaluated at compile time, the user starts to type the return logic and the AI Code

Experimental results

Research questions

• RQ1 RQ1: What impact does AI have on time spent completing an enterprise-grade development task?
• RQ2 RQ2: How do developer and task characteristics influence the estimated impact of AI assistance on time spent on task?
• RQ3 RQ3: How do developer and task characteristics interact with AI use to accelerate or slow down certain developers?
• RQ4 RQ4: Does increased frequency of AI tool use correlate with greater speed gains during the task?
• RQ5 RQ5: Why were task domain and language expertise not significant predictors of task speed despite theoretical expectations?

Key findings

• The best-estimated impact of AI on development speed was a 21% reduction in time spent on task, though the 95% confidence interval was wide.
• There was a statistically significant interaction effect showing that developers who spent more hours on code-related activities per day benefited more from AI assistance.
• No statistically significant relationship was found between frequency of AI tool use and task speed, suggesting possible ceiling effects or usability floors in the tools.
• The study found no significant effect of task domain expertise or programming language familiarity on time on task, possibly due to sample homogeneity.
• AI tools showed potential for equity concerns: senior developers may benefit more, raising questions about deskilling and access to advanced features.
• The results suggest a need for 'expert modes' and better knowledge transfer across AI tools to support high-velocity developers without increasing cognitive load.

Figure 2: Smart Paste feature in Cider V. When a user pastes code, Smart Paste provides an automatic adjustment to the code, then shows the inline diff highlights the removal of tryfromenv (strikethrough) and the insertion of flagfile (italic and lower opacity). The user can accept the adjustment us