[Paper Review] An Empirical Study on Usage and Perceptions of LLMs in a Software Engineering Project

Sanka Rasnayaka, Guanlin Wang+2 more|arXiv (Cornell University)|Jan 29, 2024

Technology Adoption and User Behaviour5 citations

TL;DR

The paper analyzes how 214 undergraduate CS students used and perceived large language models (LLMs) for a team-based software engineering project, examining code generation, prompts, human intervention, and student attitudes. It provides a framework for integrating LLMs in education and highlights implications for human-AI collaboration skills.

ABSTRACT

Large Language Models (LLMs) represent a leap in artificial intelligence, excelling in tasks using human language(s). Although the main focus of general-purpose LLMs is not code generation, they have shown promising results in the domain. However, the usefulness of LLMs in an academic software engineering project has not been fully explored yet. In this study, we explore the usefulness of LLMs for 214 students working in teams consisting of up to six members. Notably, in the academic course through which this study is conducted, students were encouraged to integrate LLMs into their development tool-chain, in contrast to most other academic courses that explicitly prohibit the use of LLMs. In this paper, we analyze the AI-generated code, prompts used for code generation, and the human intervention levels to integrate the code into the code base. We also conduct a perception study to gain insights into the perceived usefulness, influencing factors, and future outlook of LLM from a computer science student's perspective. Our findings suggest that LLMs can play a crucial role in the early stages of software development, especially in generating foundational code structures, and helping with syntax and error debugging. These insights provide us with a framework on how to effectively utilize LLMs as a tool to enhance the productivity of software engineering students, and highlight the necessity of shifting the educational focus toward preparing students for successful human-AI collaboration.

Motivation & Objective

• Investigate the usefulness of LLMs in an academic, team-based software engineering project.
• Analyze AI-generated code, prompts, and the level of human intervention required to integrate AI output.
• Assess student perceptions, usefulness, and future outlook of LLM-assisted coding.
• Examine how LLM usage correlates with coding ability, workload, and prior AI experience.
• Provide actionable insights for education on preparing students for human-AI collaboration in SE.

Proposed method

• Collect AI-generated code artifacts, prompts, and human intervention annotations from student repositories at three milestones.
• Categorize prompts into predefined categories and quantify prompt usage.
• Survey students post-project using the Unified Theory of Acceptance and Use of Technology (UTAUT) model to assess constructs like Performance Expectancy, Effort Expectancy, Social Influence, and Facilitating Conditions.
• Analyze correlations between BI/U (behavioral intention/usage) and personal factors, using reliability metrics (Cronbach’s Alpha) and hypothesis testing.
• Evaluate correctness and quality of student projects to determine impact of AI-generated code with human vetting.

Figure 1. High-level software architecture of the Static Program Analyzer (SPA), the software to be developed by the students

Experimental results

Research questions

• RQ1 How do teams use AI-generated code in an academic SE project, and what kinds of tasks are most common?
• RQ2 What levels of human intervention are required to integrate AI-generated code into student projects?
• RQ3 How do students perceive the usefulness and future impact of AI code generation in CS and SE careers?
• RQ4 What factors (coding skill, workload, prior AI experience, paid access) influence acceptance and usage of AI code generators?
• RQ5 Does AI-assisted coding affect code correctness and performance when properly vetted by humans?

Key findings

• AI-generated code is most used for initial structures, basic design patterns, data structures, and simple C++ tasks, with DFS-like solutions being common in prompts.
• Copilot outputs frequently required human intervention, while paid ChatGPT-4 often did not require major changes; free ChatGPT showed more intervention.
• Average lines per AI-generated snippet increased over milestones, suggesting users learned to prompt for larger chunks over time.
• The Pareto pattern emerged: ~80% of AI-generated code came from ~20% of teams, yet overall code quality did not significantly differ between teams with varying AI usage.
• UTAUT analysis showed high behavioral intention and usage, with strong correlations among FC, PE, EE, and BI/U; coding skill and paid access moderated usage.
• Correctness and quality did not differ significantly between teams heavily using AI and those with less/no AI usage when human intervention was applied.

Figure 2. The timeline of the software engineering project, highlighting important milestones along with specific tasks in each phase. The development activities for the project span from Week 1-13, with three key milestones that need to be achieved by the students.