[Paper Review] Study of software developers' experience using the Github Copilot Tool in the software development process

Mateusz Jaworski, Dariusz Piotrkowski|arXiv (Cornell University)|Jan 12, 2023

Software Engineering Research4 citations

TL;DR

This study investigates software developers' experiences with GitHub Copilot, an AI-driven code generation tool, through a survey of 42 developers. Despite a positive perception of the tool’s potential to speed up development, only 14.3% had used it, largely due to concerns over data security, copyright, and code quality, with significant skepticism persisting even after official assurances.

ABSTRACT

In software development there is a constant pressure to produce code faster and faster without compromising on quality. New tools supporting developers are created in response to this demand. Currently a new generation of such solutions is about to be launched - Artificial Intelligence driven tools. On 29 June 2021 Github Copilot was announced. It uses trained model to generate code based on human understandable language. The focus of this research was to investigate software developers' approach to this tool. For this purpose a survey containing 18 questions was prepared and shared with programmers. A total of 42 answers were gathered. The results of the research indicate that developers' opinions are divided. Most of them met Github Copilot before attending the survey. The attitude to the tool was mostly positive but not many participants were willing to use it. Concerns are caused by security issues associated with using of Github Copilot.

Motivation & Objective

• To understand software developers’ experiences and attitudes toward GitHub Copilot in their daily development workflows.
• To assess developers’ perceptions of GitHub Copilot’s impact on software engineering productivity and future employment prospects.
• To investigate concerns related to security, data leakage, and code quality associated with AI-generated code.
• To evaluate the tool’s perceived usefulness across different developer seniority levels and specializations (e.g., frontend, backend, fullstack).

Proposed method

• A 18-question online survey was designed by experienced software developers and reviewed for clarity and relevance.
• The survey collected data on developers’ prior exposure to GitHub Copilot, usage patterns, perceived benefits, and concerns.
• Data were analyzed quantitatively using frequency distributions and cross-tabulations across developer profiles (seniority, specialization).
• Open-ended responses were thematically analyzed to capture nuanced opinions on tool usefulness, limitations, and ethical concerns.
• The study focused on self-reported experiences and attitudes, with an emphasis on security and trust issues.
• Results were contextualized against existing literature on AI in programming and prior tools like genetic programming and code completion systems.

Figure 1: Hearing about GitHub Copilot

Experimental results

Research questions

• RQ1 How familiar are software developers with GitHub Copilot, and what is their level of actual usage?
• RQ2 What are developers’ primary motivations or barriers to adopting GitHub Copilot in their development process?
• RQ3 How do developers perceive the impact of GitHub Copilot on software development productivity and future employment, particularly across different experience levels?
• RQ4 What are the main security and ethical concerns developers have regarding data leakage and code quality from AI-generated suggestions?
• RQ5 How do perceptions of GitHub Copilot vary across different programming specializations (e.g., frontend, backend, fullstack)?

Key findings

• 85.7% of surveyed developers had not used GitHub Copilot, despite 66.7% having heard of it before the survey.
• Only 14.3% of respondents had used the tool, primarily for non-commercial purposes such as testing, learning, or open-source contributions.
• Although 85.7% expressed interest in testing the tool after learning about it, only 38.1% were eager to use it, indicating a gap between perceived value and actual adoption.
• A significant portion (31%) expressed concern about data leaks from projects, while 16.7% were uncertain, and only 13% were not afraid, highlighting ongoing trust issues.
• Most developers (50%) believed GitHub Copilot would have no impact on their employment, but juniors were more likely to see a negative impact, suggesting concerns about job displacement.
• Despite official assurances, 40% of developers remained skeptical about data security, with security and copyright issues being the primary barriers to adoption.

Figure 2: Usage of GitHub Copilot among the respondents