[Paper Review] A Survey on Natural Language Processing for Programming

Qingfu Zhu, Xianzhen Luo+3 more|arXiv (Cornell University)|Dec 12, 2022

Software Engineering Research4 citations

TL;DR

This survey systematically reviews natural language processing for programming (NLP4P) by identifying two core properties of programming languages: structure-based and functionality-oriented. It analyzes how these properties shape tasks, datasets, evaluation methods, techniques, and state-of-the-art models, and proposes future directions in iterative, multilingual, and multi-modal NLP4P to address current limitations in program understanding and generation.

ABSTRACT

Natural language processing for programming aims to use NLP techniques to assist programming. It is increasingly prevalent for its effectiveness in improving productivity. Distinct from natural language, a programming language is highly structured and functional. Constructing a structure-based representation and a functionality-oriented algorithm is at the heart of program understanding and generation. In this paper, we conduct a systematic review covering tasks, datasets, evaluation methods, techniques, and models from the perspective of the structure-based and functionality-oriented property, aiming to understand the role of the two properties in each component. Based on the analysis, we illustrate unexplored areas and suggest potential directions for future work.

Motivation & Objective

• To identify and analyze the dual intrinsic properties of programming languages—structure-based and functionality-oriented—that underlie program understanding and generation.
• To systematically review existing NLP4P work through the lens of these two properties across tasks, datasets, evaluation methods, techniques, and models.
• To highlight unexplored research areas and propose future directions, including iterative, multilingual, and multi-modal NLP4P.
• To clarify the distinct roles of structure-based modeling in model architecture and functionality-oriented design in evaluation and generation.
• To provide a structured framework for future research by aligning NLP4P components with the fundamental characteristics of programming languages.

Proposed method

• Categorize NLP4P tasks into structure-based (e.g., summarization, retrieval, classification) and functionality-oriented (e.g., program synthesis, code completion) based on input/output semantics and the role of programming language structure.
• Analyze datasets and evaluation methods through the dual lens of structure and functionality, emphasizing how structural representation and functional correctness shape data and metrics.
• Survey techniques and models, focusing on how backbone architectures (e.g., tree-based, graph-based) support structure modeling, and how decoding strategies and test feedback support functional correctness.
• Examine state-of-the-art models to identify how they leverage structure-based representations (e.g., ASTs, code embeddings) and functionality-oriented training (e.g., test feedback, execution accuracy).
• Propose future research directions by extending current paradigms: iterative NLP4P for historical context, multilingual NLP4P for low-resource languages, and multi-modal NLP4P using vision-language pre-training.
• Use a systematic review framework to map existing work to the two properties, identifying gaps in modeling, data, and evaluation.

Experimental results

Research questions

• RQ1 How do the structure-based and functionality-oriented properties of programming languages shape the design of NLP4P tasks?
• RQ2 To what extent do current datasets and evaluation methods reflect the structural and functional characteristics of programming languages?
• RQ3 How do state-of-the-art models leverage structure-based representations and functionality-oriented training for program understanding and generation?
• RQ4 What are the key limitations in current NLP4P approaches related to historical context, low-resource languages, and multi-modal inputs?
• RQ5 What future research directions can emerge from integrating iterative development, multilingual learning, and multi-modal understanding into NLP4P?

Key findings

• The structure-based property is central to model architecture and representation learning, with tree- and graph-based models showing superior performance in capturing program syntax and semantics.
• The functionality-oriented property is critical for evaluation, as models must generate functionally correct programs, and test feedback significantly improves generalization and correctness.
• Current models often ignore historical context, leading to redundant rework; iterative NLP4P that leverages prior versions and error feedback is a promising but underexplored direction.
• Multilingual NLP4P is hindered by data imbalance, with low-resource languages like Ruby underperforming; multilingual pre-training offers a scalable solution to reduce cascading errors from translation.
• Multi-modal NLP4P is nascent, with no existing datasets; leveraging pre-trained vision-language models (e.g., CLIP, Flamingo) on existing multi-modal data could enable better understanding of visual or diagrammatic program specifications.
• Despite strong performance in English and major languages like Python and Java, low-resource programming languages and natural language pairs remain under-served, indicating a significant gap in accessibility and fairness.