[Paper Review] Exploring Software Naturalness through Neural Language Models

Luca Buratti, Saurabh Pujar+9 more|arXiv (Cornell University)|Jun 22, 2020

Software Engineering ResearchComputer Science37 references54 citations

TL;DR

The paper investigates software naturalness by pre-training transformer LMs on raw C code (C-BERT) with different tokenizers and pre-training strategies, then evaluating AST node tagging and vulnerability identification without relying on compiler-derived features.

ABSTRACT

The Software Naturalness hypothesis argues that programming languages can be understood through the same techniques used in natural language processing. We explore this hypothesis through the use of a pre-trained transformer-based language model to perform code analysis tasks. Present approaches to code analysis depend heavily on features derived from the Abstract Syntax Tree (AST) while our transformer-based language models work on raw source code. This work is the first to investigate whether such language models can discover AST features automatically. To achieve this, we introduce a sequence labeling task that directly probes the language models understanding of AST. Our results show that transformer based language models achieve high accuracy in the AST tagging task. Furthermore, we evaluate our model on a software vulnerability identification task. Importantly, we show that our approach obtains vulnerability identification results comparable to graph based approaches that rely heavily on compilers for feature extraction.

Motivation & Objective

• Test whether transformer-based language models can learn AST-like structure from raw code without AST features.
• Assess how tokenization strategies affect learning and downstream task performance on C code.
• Evaluate vulnerability identification performance against graph-based, compiler-dependent approaches.

Proposed method

• Pre-train a BERT-like transformer (C-BERT) from scratch on 100 open-source C repositories.
• Explore three tokenization strategies: Char, KeyChar (Char + C keywords), and SentencePiece (SPE).
• Employ three pre-training objectives: MLM, Whole Word Masking (WWM) for stronger masking of strings, and task-specific fine-tuning objectives.
• Introduce an AST node tagging task to probe the model’s understanding of token_kind and cursor_kind via Clang-derived labels.
• Fine-tune models on Vulnerability Identification (VI) and compare to graph-based baselines such as GGNN.
• Evaluate on FFmpeg and QEMU datasets with fixed-width 250-token windows and sliding-window aggregation for long inputs.

Experimental results

Research questions

• RQ1 Can a transformer LM trained directly on raw C source code discover AST-like features without explicit structural information?
• RQ2 How do tokenization choices (Char, KeyChar, SPE) and pre-training objectives affect the learning of syntactic/semantic aspects of code?
• RQ3 How does a language-model-based approach perform on vulnerability identification compared to graph-based, compiler-dependent methods?

Key findings

• Transformers trained on raw C code with different tokenizers outperform a BiLSTM baseline on AST cursor_kind tagging across datasets.
• C-BERT with SPE generally achieves high accuracy (97.72–97.73) and F1 (98.29–98.31) on FFmpeg, and strong results on QEMU (81.11–87.79).
• Character-based tokenization with Whole Word Masking yields strong results, and WWM generally improves VI performance while reducing OOV concerns.
• For AST cursor_kind tagging, C-BERT with Char or SPE tokenization achieves the best scores, with FFmpeg consistently easier than QEMU.
• In VI, C-BERT models with MLM-based pre-training outperform Naive, BiLSTM, CNN baselines and GGNN baselines on full and reduced datasets, with WWM helping Char/KeyChar variants.