[论文解读] A Survey of Neural Code Intelligence: Paradigms, Advances and Beyond

Qiushi Sun, Zhirui Chen+16 more|arXiv (Cornell University)|Mar 21, 2024

Machine Learning in Bioinformatics被引用 5

一句话总结

本综述系统性地回顾了神经代码智能在三个阶段的发展演变——用于代码的神经语言模型、CodePTMs 和 CodeLLMs——覆盖任务、基准、跨领域协同作用及未来方向。

ABSTRACT

Neural Code Intelligence -- leveraging deep learning to understand, generate, and optimize code -- holds immense potential for transformative impacts on the whole society. Bridging the gap between Natural Language and Programming Language, this domain has drawn significant attention from researchers in both research communities over the past few years. This survey presents a systematic and chronological review of the advancements in code intelligence, encompassing over 50 representative models and their variants, more than 20 categories of tasks, and an extensive coverage of over 680 related works. We follow the historical progression to trace the paradigm shifts across different research phases (e.g., from modeling code with recurrent neural networks to the era of Large Language Models). Concurrently, we highlight the major technical transitions in models, tasks, and evaluations spanning through different stages. For applications, we also observe a co-evolving shift. It spans from initial endeavors to tackling specific scenarios, through exploring a diverse array of tasks during its rapid expansion, to currently focusing on tackling increasingly complex and varied real-world challenges. Building on our examination of the developmental trajectories, we further investigate the emerging synergies between code intelligence and broader machine intelligence, uncovering new cross-domain opportunities and illustrating the substantial influence of code intelligence across various domains. Finally, we delve into both the opportunities and challenges associated with this field, alongside elucidating our insights on the most promising research directions. An ongoing, dynamically updated project and resources associated with this survey have been released at https://github.com/QiushiSun/Awesome-Code-Intelligence.

研究动机与目标

• 回顾神经代码智能的历史演进及各阶段的范式转变。
• 将与代码相关的任务和基准分类为连贯的类别，以便进行结构化分析。
• 总结核心模型架构、训练目标，以及代码结构（抽象语法树 AST、数据流、控制流）在其中的作用。
• 讨论应用、跨领域协同作用以及实际挑战，以指导未来研究方向。

提出的方法

• 对50多种代表性模型和680+相关工作进行系统、按时间顺序的综述。
• 将模型分为三个时期：用于代码的神经语言模型、代码预训练模型（CodePTMs）和 CodeLLMs。
• 分析架构选择、训练数据和目标，重点关注结构化代码信息。
• 编目并总结跨20+类别的广泛代码相关任务和基准。
• 讨论跨域整合及对现实应用与评估的影响。
• 指出未解问题和未来工作的有前景方向。

Figure 1 : Cumulative number of publications/preprints related to neural code intelligence (from arXiv). Over the past few years, the number of articles has been steadily increasing.

实验结果

研究问题

• RQ1 神经代码智能从早期的神经语言模型到 CodePTMs 和 CodeLLMs 的主要范式转变是什么？
• RQ2 CodePTMs 在性能、数据需求和任务覆盖方面与大型语言模型相比如何？
• RQ3 推动代码智能进展的关键基准和任务是什么？它们如何在各阶段演变？
• RQ4 将代码智能与更广泛的机器智能和现实世界应用整合时出现哪些跨域机会与挑战？

主要发现

• 三个递进阶段推动了进展：用于代码的神经语言模型、代码预训练模型，以及 CodeLLMs。
• CodePTMs 如 CodeBERT 和 CodeT5 为代码建立了预训练/微调范式。
• CodeLLMs 展现出对提示和上下文学习的转变，超越仅限代码的任务，扩展到现实世界场景。
• 大量任务（超过 20 个类别）和基准（数千个数据集）支撑该领域的进展。
• 存在显著的跨域协同作用和现实世界挑战，这些将塑造代码智能的未来方向。

Figure 2 : A chronological overview of representative works in neural code intelligence over recent years. Works are differentiated by background colors to represent distinct evolutionary phases: ${\color[rgb]{1,0.88671875,0.58203125}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.88671875,0.58203125}\