[Paper Review] Malware Classification based on Call Graph Clustering

Joris Kinable, Orestis Kostakis|arXiv (Cornell University)|Aug 25, 2010

Network Security and Intrusion Detection23 references4 citations

TL;DR

This paper proposes a call graph clustering approach for automated malware family classification by computing graph similarity via approximate graph edit distance and applying clustering algorithms like DBSCAN and k-medoids to real malware samples. The method successfully identifies structural similarities across malware variants, enabling generic detection schemes with 90% accuracy against manual classifications.

ABSTRACT

Each day, anti-virus companies receive tens of thousands samples of potentially harmful executables. Many of the malicious samples are variations of previously encountered malware, created by their authors to evade pattern-based detection. Dealing with these large amounts of data requires robust, automatic detection approaches. This paper studies malware classification based on call graph clustering. By representing malware samples as call graphs, it is possible to abstract certain variations away, and enable the detection of structural similarities between samples. The ability to cluster similar samples together will make more generic detection techniques possible, thereby targeting the commonalities of the samples within a cluster. To compare call graphs mutually, we compute pairwise graph similarity scores via graph matchings which approximately minimize the graph edit distance. Next, to facilitate the discovery of similar malware samples, we employ several clustering algorithms, including k-medoids and DBSCAN. Clustering experiments are conducted on a collection of real malware samples, and the results are evaluated against manual classifications provided by human malware analysts. Experiments show that it is indeed possible to accurately detect malware families via call graph clustering. We anticipate that in the future, call graphs can be used to analyse the emergence of new malware families, and ultimately to automate implementation of generic detection schemes.

Motivation & Objective

• To address the challenge of classifying large volumes of evolving malware variants that evade traditional signature-based detection.
• To develop an automated, scalable method for identifying structural similarities among malware samples using call graphs as high-level abstractions.
• To evaluate clustering algorithms on real malware call graphs for accurate malware family identification.
• To enable generic detection by targeting common structural patterns across malware families rather than individual samples.

Proposed method

• Represent malware samples as directed call graphs, where functions are vertices and function calls are directed edges.
• Compute pairwise graph similarity using approximate graph matching to minimize graph edit distance, enabling efficient comparison of call graphs.
• Apply k-medoids and DBSCAN clustering algorithms to group structurally similar malware samples into families.
• Use a graph edit distance-based similarity metric as the core distance measure for clustering, validated through manual classification.
• Optimize clustering parameters (e.g., Rad and MinPts in DBSCAN) to maximize alignment with human analyst categorizations.
• Adapt call graph extraction to detect 'droppers' by isolating and analyzing the hidden payload instead of the installer.

Experimental results

Research questions

• RQ1 Can call graph representations effectively capture structural similarities among malware variants despite code obfuscation and mutation?
• RQ2 How well do clustering algorithms like DBSCAN and k-medoids perform in grouping malware samples into known families based on call graph similarity?
• RQ3 What is the impact of graph edit distance approximation on the accuracy of malware family classification?
• RQ4 Can the proposed method detect and distinguish between different generations of malware families, including those with significant structural evolution?
• RQ5 How does the method perform on real-world malware samples compared to manual classifications by security analysts?

Key findings

• DBSCAN outperformed k-medoids in malware family detection, successfully identifying clusters that aligned closely with manually classified malware families.
• The method achieved a classification accuracy of approximately 90% when compared to manual classifications by F-Secure analysts.
• Cluster 16 contained isomorphic call graphs from different malware families, indicating that droppers with varying payloads were indistinguishable at the call graph level.
• The presence of multiple families within a single cluster (e.g., clusters 2, 6) was attributed to samples being within the Rad distance threshold despite belonging to different families.
• The study confirmed that graph edit distance is a viable metric for measuring call graph similarity, especially when combined with approximation algorithms for scalability.
• Adapting call graph extraction to analyze the payload of droppers rather than the installer improved detection accuracy for this malware type.