[Paper Review] Automatic feature learning for vulnerability prediction

Hoa Khanh Dam, Truyen Tran+4 more|arXiv (Cornell University)|Aug 8, 2017

Software Engineering ResearchComputer Science27 references90 citations

TL;DR

The paper proposes a deep learning approach using Long Short-Term Memory (LSTM) to automatically learn semantic and syntactic features from code for predicting software vulnerabilities, achieving competitive or superior results to state-of-the-art models.

ABSTRACT

Code flaws or vulnerabilities are prevalent in software systems and can potentially cause a variety of problems including deadlock, information loss, or system failure. A variety of approaches have been developed to try and detect the most likely locations of such code vulnerabilities in large code bases. Most of them rely on manually designing features (e.g. complexity metrics or frequencies of code tokens) that represent the characteristics of the code. However, all suffer from challenges in sufficiently capturing both semantic and syntactic representation of source code, an important capability for building accurate prediction models. In this paper, we describe a new approach, built upon the powerful deep learning Long Short Term Memory model, to automatically learn both semantic and syntactic features in code. Our evaluation on 18 Android applications demonstrates that the prediction power obtained from our learned features is equal or even superior to what is achieved by state of the art vulnerability prediction models: 3%--58% improvement for within-project prediction and 85% for cross-project prediction.

Motivation & Objective

• Motivate vulnerability prediction as a high-value early security task.
• Overcome limitations of manually engineered features by learning representations directly from code.
• Capture both semantic and syntactic information to improve within- and cross-project predictions.

Proposed method

• Represent each Java file as a header plus methods; parse into token sequences per method.
• Train an LSTM to learn token embeddings and dynamic token states to form method vectors.
• Pool method vectors to obtain syntactic features for a file; build a semantic codebook by clustering token states across projects.
• Generate semantic features via codebook clustering and semantic pooling to enable cross-project generalization.
• Train a vulnerability classifier (Random Forest) on learned features and compare against baselines.

Experimental results

Research questions

• RQ1 RQ1: Can automatically learned features via LSTM yield effective within-project vulnerability prediction?
• RQ2 RQ2: How does the approach perform in cross-version vulnerability prediction within the same application?
• RQ3 RQ3: Is the approach effective for cross-project vulnerability prediction across different applications?
• RQ4 RQ4: How does the proposed method compare to software metrics, Bag-of-Words, and Deep Belief Network baselines?

Key findings

• Within-project results show competitive to superior performance compared to benchmarks across multiple apps.
• The approach achieves high precision and recall using joint syntactic and semantic features.
• Semantic features via codebook clustering improve cross-project generalization.
• The paper reports substantial improvements over some baselines in cross-project settings (e.g., up to 85% for cross-project prediction).
• Experiments used 18 Android apps with 10-fold cross-validation for within-project evaluation.