[Paper Review] How Does Naming Affect LLMs on Code Analysis Tasks?

Zhilong Wang, Lan Zhang+4 more|arXiv (Cornell University)|Jul 24, 2023

Software Engineering Research4 citations

TL;DR

This paper investigates how naming conventions affect large language models (LLMs) like ChatGPT and CodeBERT in security-oriented code analysis tasks. It reveals that both models heavily rely on meaningful variable, function, and method names, with performance dropping drastically—down to 17–24% accuracy—when names are anonymized, indicating a critical limitation in learning semantic logic from code without literal identifiers.

ABSTRACT

The Large Language Models (LLMs), such as GPT and BERT, were proposed for natural language processing (NLP) and have shown promising results as general-purpose language models. An increasing number of industry professionals and researchers are adopting LLMs for program analysis tasks. However, one significant difference between programming languages and natural languages is that a programmer has the flexibility to assign any names to variables, methods, and functions in the program, whereas a natural language writer does not. Intuitively, the quality of naming in a program affects the performance of LLMs in program analysis tasks. This paper investigates how naming affects LLMs on code analysis tasks. Specifically, we create a set of datasets with code containing nonsense or misleading names for variables, methods, and functions, respectively. We then use well-trained models (CodeBERT) to perform code analysis tasks on these datasets. The experimental results show that naming has a significant impact on the performance of code analysis tasks based on LLMs, indicating that code representation learning based on LLMs heavily relies on well-defined names in code. Additionally, we conduct a case study on some special code analysis tasks using GPT, providing further insights.

Motivation & Objective

• To evaluate the effectiveness of LLMs, specifically ChatGPT and CodeBERT, in security-oriented code analysis tasks.
• To investigate how the presence or absence of meaningful variable, function, and method names affects LLM performance.
• To assess whether LLMs can learn semantic logic from code when literal identifiers are removed or replaced with meaningless strings.
• To identify the strengths and limitations of LLMs in understanding code semantics beyond surface-level naming.
• To provide actionable insights for researchers and practitioners on the reliable use of LLMs in software security applications.

Proposed method

• Conducted a qualitative case study on ChatGPT’s responses to security-oriented code analysis tasks, introducing deliberate challenges to assess reasoning and accuracy.
• Performed a quantitative analysis on CodeBERT by systematically anonymizing code elements: variable names, method definitions, and method invocations.
• Generated eight variants of the original dataset by applying two anonymization strategies: 'randomly-generated' (nonsensical strings) and 'meaningfully-generated' (semantically misleading but syntactically valid names).
• Retrained CodeBERT and GraphCodeBERT on anonymized datasets and evaluated performance on two downstream tasks: natural language code search and code clone detection.
• Compared model accuracy across original and anonymized datasets to measure the impact of literal naming features on model performance.
• Analyzed the relative degradation in performance when different types of developer-defined names were removed or replaced.

Experimental results

Research questions

• RQ1 How does the presence of meaningful variable, function, and method names affect the performance of LLMs like ChatGPT and CodeBERT in security-oriented code analysis?
• RQ2 To what extent can LLMs generalize to code with anonymized identifiers, particularly when names are replaced with meaningless or misleading strings?
• RQ3 Does the use of 'meaningfully-generated' names (e.g., replacing 'bubble_sort' with 'aes_encryption') mislead LLMs more than 'randomly-generated' names?
• RQ4 How do different types of code anonymization (variable, method definition, method invocation) individually impact model accuracy in code search and clone detection tasks?
• RQ5 Can current LLMs effectively learn semantic logic from code when literal identifiers are absent or obscured?

Key findings

• CodeBERT’s performance on code search dropped to 17.42% accuracy when all developer-defined names were removed, using randomly-generated strings, compared to 70.36% on the original Java dataset.
• On Python code search, performance fell to 24.09% with full anonymization using randomly-generated names, down from 68.17% on the original dataset.
• For clone detection, CodeBERT’s accuracy dropped to 86.77% on Java and 84.76% on Python when all names were anonymized, compared to 94.87% and 94.27% on the original datasets.
• The 'meaningfully-generated' anonymization strategy consistently led to worse performance than 'randomly-generated' strings, suggesting that misleading semantic cues can mislead models.
• The study confirms that current LLMs like CodeBERT and GraphCodeBERT rely heavily on literal features (e.g., names) rather than semantic or logical features in code, making them vulnerable to obfuscation attacks.
• ChatGPT demonstrated strong capability in understanding high-level code semantics when names were meaningful, but its performance degraded significantly with ambiguous or missing naming context.