[Paper Review] Automatically generating features for learning program analysis heuristics

Kwonsoo Chae, Hakjoo Oh+2 more|arXiv (Cornell University)|Dec 30, 2016

Software Testing and Debugging Techniques24 references3 citations

TL;DR

This paper presents a framework that automatically generates features for data-driven static program analyses by reducing and abstracting representative program-query pairs using a program reducer. The method produces compact, generalizable code snippets as boolean features, enabling machine learning to derive effective analysis heuristics; experiments show performance comparable to manually crafted features across three C program analyses.

ABSTRACT

We present a technique for automatically generating features for data-driven program analyses. Recently data-driven approaches for building a program analysis have been proposed, which mine existing codebases and automatically learn heuristics for finding a cost-effective abstraction for a given analysis task. Such approaches reduce the burden of the analysis designers, but they do not remove it completely; they still leave the highly nontrivial task of designing so called features to the hands of the designers. Our technique automates this feature design process. The idea is to use programs as features after reducing and abstracting them. Our technique goes through selected program-query pairs in codebases, and it reduces and abstracts the program in each pair to a few lines of code, while ensuring that the analysis behaves similarly for the original and the new programs with respect to the query. Each reduced program serves as a boolean feature for program-query pairs. This feature evaluates to true for a given program-query pair when (as a program) it is included in the program part of the pair. We have implemented our approach for three real-world program analyses. Our experimental evaluation shows that these analyses with automatically-generated features perform comparably to those with manually crafted features.

Motivation & Objective

• To address the major bottleneck in data-driven program analysis: the labor-intensive, expert-dependent design of input features.
• To automate the generation of meaningful, generalizable features that capture patterns where precision improvements significantly impact analysis outcomes.
• To enable effective learning of analysis heuristics without requiring manual feature engineering, reducing dependency on domain expertise.
• To generalize across different static analysis types (e.g., interval, pointer, Octagon) by using abstracted program snippets as features.
• To demonstrate that automatically generated features can yield heuristics with performance comparable to those derived from manually crafted features.

Proposed method

• Uses a program reducer (e.g., C-Reduce) on program-query pairs from codebases to generate minimal, representative code snippets that trigger high-precision analysis behavior.
• Treats each reduced program as a boolean feature that evaluates to true when the program is present in a given program-query pair.
• Abstracts the reduced programs into data-flow graphs to generalize patterns and create reusable, language-agnostic features.
• Applies the abstracted features to train machine learning models that learn optimal parameter settings for parametric static analyses.
• Integrates the feature generation and heuristic learning into a unified framework applicable to various C program analyses with infinite abstract domains.
• Employs a black-box approach that does not require access to the internal logic of the static analysis, enhancing generalizability.

Experimental results

Research questions

• RQ1 Can program reduction be used to automatically generate meaningful, minimal code snippets that serve as effective features for learning program analysis heuristics?
• RQ2 Can abstracted data-flow graphs derived from reduced programs generalize across diverse program patterns and serve as robust, reusable features?
• RQ3 To what extent can heuristics learned using automatically generated features match the performance of those learned with manually crafted features?
• RQ4 Does the proposed method scale across different types of static analyses, including those with infinite abstract domains like interval and Octagon analyses?
• RQ5 Can the framework be applied effectively to real-world codebases without requiring deep analysis-specific knowledge?

Key findings

• The proposed method successfully generated features relevant to three distinct static analyses: partially flow-sensitive interval and pointer analyses, and partial Octagon analysis.
• The heuristics learned using automatically generated features achieved performance comparable to those learned with manually crafted features, as validated on 60 programs from Linux and GNU packages.
• The framework demonstrated generalizability across different analysis types, including those with infinite abstract domains, by relying on abstracted program snippets as features.
• The use of program reducers to generate minimal, behaviorally significant code snippets proved effective in capturing patterns where precision improvements yield measurable benefits.
• The approach reduces the need for expert-driven feature engineering, significantly lowering the barrier to adopting data-driven static analysis techniques.
• Deep learning baselines using character-level CNNs achieved only 27% precision and 93% recall in predicting flow-sensitivity needs, highlighting the advantage of the proposed feature generation method.