[Paper Review] Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks

Yaqin Zhou, Shangqing Liu+3 more|arXiv (Cornell University)|Sep 8, 2019

Software Engineering Research22 references361 citations

TL;DR

Devign proposes a graph neural network that learns composite program semantics from multiple code representations to identify vulnerable functions, outperforming baselines with a specialized Conv module for graph-level classification.

ABSTRACT

Vulnerability identification is crucial to protect the software systems from attacks for cyber security. It is especially important to localize the vulnerable functions among the source code to facilitate the fix. However, it is a challenging and tedious process, and also requires specialized security expertise. Inspired by the work on manually-defined patterns of vulnerabilities from various code representation graphs and the recent advance on graph neural networks, we propose Devign, a general graph neural network based model for graph-level classification through learning on a rich set of code semantic representations. It includes a novel Conv module to efficiently extract useful features in the learned rich node representations for graph-level classification. The model is trained over manually labeled datasets built on 4 diversified large-scale open-source C projects that incorporate high complexity and variety of real source code instead of synthesis code used in previous works. The results of the extensive evaluation on the datasets demonstrate that Devign outperforms the state of the arts significantly with an average of 10.51% higher accuracy and 8.68\\% F1 score, increases averagely 4.66% accuracy and 6.37% F1 by the Conv module.

Motivation & Objective

• Motivate vulnerability identification at the function level to localize security flaws in large codebases.
• Encode rich program semantics by constructing a joint graph from multiple representations (AST, CFG, DFG, and token sequence).
• Train a graph neural network with a novel Conv module for effective graph-level classification of vulnerable functions.
• Evaluate on manually labeled datasets from four large open-source C projects and compare to static analyzers and baselines.

Proposed method

• Construct composite code graphs by integrating AST, CFG, DFG (represented as LastRead/LastWrite/ComputedFrom), and natural code sequence edges into a heterogeneous graph with node features (code tokens via word2vec and node types).
• Use a gated graph recurrent network to propagate and update node embeddings across edge types with GRU-based fusion (Equation 3 and 4).
• Apply a Conv module consisting of 1-D convolutional and dense layers to extract graph-level features from node representations (Equations 6-9).
• Train end-to-end with cross-entropy loss plus regularization to predict vulnerable vs. non-vulnerable functions (Equation 1).
• Prepare manually labeled datasets from Linux Kernel, QEMU, Wireshark, and FFmpeg by sourcing vulnerable/non-vulnerable functions from security-related commits and manual verification.

Experimental results

Research questions

• RQ1 Can Devign outperform state-of-the-art learning-based vulnerability identification methods on real-world code?
• RQ2 Does the Conv module improve graph-level classification over simple summation of node embeddings?
• RQ3 Is learning from composite graphs (combining multiple code representations) superior to single-edge graphs for vulnerability detection?
• RQ4 How does Devign perform under imbalanced data and on recent CVEs?
• RQ5 Can Devign generalize to zero-day vulnerabilities identified via CVEs in newly labeled datasets?

Key findings

• Devign with composite code representations achieves higher accuracy and F1 scores than baseline methods across multiple datasets (Linux Kernel, QEMU, Wireshark, FFmpeg).
• The Conv module provides additional accuracy and F1 gains over a simple Gated Graph Recurrent Network with sum-based graph classification.
• Single-edge graphs sometimes perform competitively, but composite graphs generally offer superior or comparable performance, with average improvements in accuracy and F1 when using composite representations.
• In imbalanced settings, Devign and its composites outperform static analyzers, with substantially better F1 scores.
• Applied to 40 CVEs, Devign achieved an average accuracy of 74.11% on the extracted vulnerable functions, indicating practical applicability.