[Paper Review] Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants

Gustavo Sandoval, Hammond Pearce+4 more|arXiv (Cornell University)|Aug 20, 2022

Software Engineering Research39 citations

TL;DR

The paper conducts a security-focused user study (N=58) to assess whether Codex-based AI code suggestions increase security bugs when students implement a C singly-linked shopping list; findings show security impact is small, with bugs in AI-assisted code not exceeding 10% more than controls.

ABSTRACT

Large Language Models (LLMs) such as OpenAI Codex are increasingly being used as AI-based coding assistants. Understanding the impact of these tools on developers' code is paramount, especially as recent work showed that LLMs may suggest cybersecurity vulnerabilities. We conduct a security-driven user study (N=58) to assess code written by student programmers when assisted by LLMs. Given the potential severity of low-level bugs as well as their relative frequency in real-world projects, we tasked participants with implementing a singly-linked 'shopping list' structure in C. Our results indicate that the security impact in this setting (low-level C with pointer and array manipulations) is small: AI-assisted users produce critical security bugs at a rate no greater than 10% more than the control, indicating the use of LLMs does not introduce new security risks.

Motivation & Objective

• Motivate the study by evaluating whether AI code assistants affect security in real-world-like coding tasks.
• Assess the incidence of security bugs in AI-assisted code compared to non-assisted code.
• Analyze how and where security bugs originate in AI-assisted coding tasks (human vs. AI contributions).
• Provide open data for replication and broader analysis.

Proposed method

• Randomized controlled design with two groups: control (no Codex access) and assisted (Codex access).
• Cloud-based IDE logs user interactions with Codex to analyze acceptance of AI suggestions.
• Participants complete 12 C functions for a singly-linked shopping list to stress memory-related bugs.
• Security and functionality are evaluated using CWE taxonomy and both static/run-time analyses and manual review.
• Autopilot condition generates code entirely by Codex for comparison (30 solutions across three Codex models).
• Statistical framework includes comparative tests for efficacy and non-inferiority tests with a 10% vulnerability margin.

Experimental results

Research questions

• RQ1 RQ1: Do AI code assistants help novice users write better functional code?
• RQ2 RQ2: Do AI-assisted solutions have an acceptable incidence of security bugs compared to non-assisted code?
• RQ3 RQ3: Where do bugs originate in an LLM-assisted system (human-written vs. AI-suggested code)?

Key findings

• AI-assisted users produced functional code with productivity benefits consistent with prior work.
• Security bugs in the AI-assisted group were not more than 10% higher in incidence per line of code than the control group.
• 63% of bugs originated in code written by humans, while 36% were present in AI-suggested code.
• The study used CWE taxonomy to categorize bugs and combined manual analysis with static and runtime checks.
• Data and materials from the study are provided as open source.