[Paper Review] Deep Learning for Source Code Modeling and Generation: Models, Applications and Challenges

Triet Huynh Minh Le, Hao Chen+1 more|arXiv (Cornell University)|Feb 13, 2020

Software Engineering Research280 references36 citations

TL;DR

A comprehensive survey of deep learning approaches for source code modeling and generation, framing program learning tasks within encoder–decoder architectures and outlining datasets, practices, challenges, and future directions.

ABSTRACT

Deep Learning (DL) techniques for Natural Language Processing have been evolving remarkably fast. Recently, the DL advances in language modeling, machine translation and paragraph understanding are so prominent that the potential of DL in Software Engineering cannot be overlooked, especially in the field of program learning. To facilitate further research and applications of DL in this field, we provide a comprehensive review to categorize and investigate existing DL methods for source code modeling and generation. To address the limitations of the traditional source code models, we formulate common program learning tasks under an encoder-decoder framework. After that, we introduce recent DL mechanisms suitable to solve such problems. Then, we present the state-of-the-art practices and discuss their challenges with some recommendations for practitioners and researchers as well.

Motivation & Objective

• Motivate the use of deep learning for Big Code by highlighting limitations of traditional code models.
• Systematically map source code modeling tasks to encoder–decoder frameworks.
• Review state-of-the-art DL models and mechanisms applicable to code modeling and generation.
• Identify datasets, challenges, and recommendations for practitioners and researchers.

Proposed method

• Formulates common program learning tasks under an encoder–decoder framework to address flexibility and scalability.
• Reviews traditional approaches (DSLs, probabilistic grammars, n-grams, simple neural models) and their limitations.
• Describes deep sequence modeling models (RNNs, CNNs, Transformer) and techniques (attention, external memory, beam search) for code tasks.
• Discusses input embeddings and handling large code vocabularies including sub-word and character-level representations.
• Outlines stable training techniques (SGD variants, dropout variants, normalization, regularization) specific to sequence models.
• Provides a systematic review of DL-based practices for Big Code tasks and discusses datasets.

Experimental results

Research questions

• RQ1 What are the major DL-based methods applicable to source code modeling and generation?
• RQ2 How can encoder–decoder frameworks be applied to diverse Big Code tasks?
• RQ3 What datasets support DL-based source code modeling and generation?
• RQ4 What are the current challenges and future directions for deep source code models?
• RQ5 How do DL approaches compare with traditional models in terms of capturing syntax, semantics, and long-term dependencies?

Key findings

• DL approaches outperform traditional models by automatically learning features and capturing long-term dependencies in code.
• Encoder–decoder frameworks with attention/memory mechanisms provide effective context handling for code sequences.
• Transformer-based and other non-recurrent architectures offer strong performance and faster training for code-related tasks.
• Character- and sub-word-level embeddings help manage large code vocabularies and enable better code completion.
• A variety of datasets and practical recommendations exist, along with identified challenges like scalability, generalization, and dataset quality.