[Paper Review] A Survey of Neural Code Intelligence: Paradigms, Advances and Beyond

Qiushi Sun, Zhirui Chen+16 more|arXiv (Cornell University)|Mar 21, 2024

Machine Learning in Bioinformatics5 citations

TL;DR

The survey systematically reviews neural code intelligence evolution across three phases—neural language models for code, CodePTMs, and CodeLLMs—covering tasks, benchmarks, cross-domain synergies, and future directions.

ABSTRACT

Neural Code Intelligence -- leveraging deep learning to understand, generate, and optimize code -- holds immense potential for transformative impacts on the whole society. Bridging the gap between Natural Language and Programming Language, this domain has drawn significant attention from researchers in both research communities over the past few years. This survey presents a systematic and chronological review of the advancements in code intelligence, encompassing over 50 representative models and their variants, more than 20 categories of tasks, and an extensive coverage of over 680 related works. We follow the historical progression to trace the paradigm shifts across different research phases (e.g., from modeling code with recurrent neural networks to the era of Large Language Models). Concurrently, we highlight the major technical transitions in models, tasks, and evaluations spanning through different stages. For applications, we also observe a co-evolving shift. It spans from initial endeavors to tackling specific scenarios, through exploring a diverse array of tasks during its rapid expansion, to currently focusing on tackling increasingly complex and varied real-world challenges. Building on our examination of the developmental trajectories, we further investigate the emerging synergies between code intelligence and broader machine intelligence, uncovering new cross-domain opportunities and illustrating the substantial influence of code intelligence across various domains. Finally, we delve into both the opportunities and challenges associated with this field, alongside elucidating our insights on the most promising research directions. An ongoing, dynamically updated project and resources associated with this survey have been released at https://github.com/QiushiSun/Awesome-Code-Intelligence.

Motivation & Objective

• Trace the historical progression of neural code intelligence and its paradigm shifts across phases.
• Classify code-related tasks and benchmarks into coherent categories for structured analysis.
• Summarize core model architectures, training objectives, and the role of code structure (ASTs, data flow, control flow).
• Discuss applications, cross-domain synergies, and practical challenges to guide future research directions.

Proposed method

• Conduct a systematic, chronological review of 50+ representative models and 680+ related works.
• Organize models into three eras: neural language models for code, code pre-trained models (CodePTMs), and CodeLLMs.
• Analyze architectural choices, training data, and objectives with emphasis on structural code information.
• Catalog and summarize a wide range of code-related tasks and benchmarks across 20+ categories.
• Discuss cross-domain integrations and implications for real-world applications and evaluation.
• Identify open issues and promising directions for future work.

Figure 1 : Cumulative number of publications/preprints related to neural code intelligence (from arXiv). Over the past few years, the number of articles has been steadily increasing.

Experimental results

Research questions

• RQ1 What are the major paradigm shifts in neural code intelligence from early neural LMs to CodePTMs and CodeLLMs?
• RQ2 How do CodePTMs compare to large language models in terms of performance, data needs, and task coverage?
• RQ3 What are the key benchmarks and tasks driving progress in code intelligence, and how do they evolve across eras?
• RQ4 What cross-domain opportunities and challenges arise when integrating code intelligence with broader machine intelligence and real-world applications?

Key findings

• Three escalating phases have driven progress: neural language models for code, code pre-trained models, and CodeLLMs.
• CodePTMs like CodeBERT and CodeT5 established the pre-train/fine-tune paradigm for code.
• CodeLLMs show shifts toward prompting and in-context learning, expanding beyond code-only tasks to real-world scenarios.
• A wide array of tasks (over 20 categories) and benchmarks (thousands of datasets) underpin the field’s progress.
• There are significant cross-domain synergies and real-world challenges that shape future directions in code intelligence.

Figure 2 : A chronological overview of representative works in neural code intelligence over recent years. Works are differentiated by background colors to represent distinct evolutionary phases: ${\color[rgb]{1,0.88671875,0.58203125}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.88671875,0.58203125}\