[Paper Review] Throttling Malware Families in 2D

Mohamed Nassar, Haı̈dar Safa|arXiv (Cornell University)|Jan 29, 2019

Advanced Malware Detection Techniques9 references4 citations

TL;DR

This paper proposes a t-SNE-based pipeline to visualize and classify malware families in 2D, demonstrating that t-SNE-embedded representations significantly boost SVM classifier performance. By reducing high-dimensional n-gram features to 2D, the method enhances cross-validation accuracy and achieves a test logloss of 0.1719 on a 9-class malware dataset with minimal feature engineering.

ABSTRACT

Malicious software are categorized into families based on their static and dynamic characteristics, infection methods, and nature of threat. Visual exploration of malware instances and families in a low dimensional space helps in giving a first overview about dependencies and relationships among these instances, detecting their groups and isolating outliers. Furthermore, visual exploration of different sets of features is useful in assessing the quality of these sets to carry a valid abstract representation, which can be later used in classification and clustering algorithms to achieve a high accuracy. In this paper, we investigate one of the best dimensionality reduction techniques known as t-SNE to reduce the malware representation from a high dimensional space consisting of thousands of features to a low dimensional space. We experiment with different feature sets and depict malware clusters in 2-D. Surprisingly, t-SNE does not only provide nice 2-D drawings, but also dramatically increases the generalization power of SVM classifiers. Moreover, obtained results showed that cross-validation accuracy is much better using the 2-D embedded representation of samples than using the original high-dimensional representation.

Motivation & Objective

• To explore the utility of t-SNE for visualizing malware families in low-dimensional space.
• To assess whether t-SNE-embedded representations improve the generalization of SVM classifiers compared to high-dimensional inputs.
• To evaluate the effectiveness of simple n-gram features in conjunction with t-SNE for malware classification.
• To develop a lightweight, first-insight classifier pipeline using t-SNE and SVM for malware family analysis.
• To validate the approach on a large, real-world malware dataset with 9 classes and unbalanced data.

Proposed method

• Extract n-gram features (3-, 4-, 5-byte) from malware binaries, selecting the top 1,000 most frequent features.
• Apply t-SNE to reduce the 1,000-dimensional feature space to a 2D embedding, using perplexity=5 and a learning rate of 200.
• Train an SVM with an RBF kernel on the t-SNE-embedded training data and evaluate on both training and cross-validated sets.
• For testing, apply t-SNE to the full dataset (train + test) to embed test points, then use the trained SVM to predict class probabilities.
• Evaluate performance using logarithmic loss on the test set via Kaggle submission, with no access to true labels during prediction.
• Use a non-parametric t-SNE mapping, avoiding model retraining for test data, and rely on joint t-SNE fitting of full data.

Experimental results

Research questions

• RQ1 Can t-SNE visualization effectively reveal structural relationships and clusters among malware families in 2D?
• RQ2 Does t-SNE embedding improve the generalization performance of SVM classifiers on malware classification tasks?
• RQ3 How does the performance of t-SNE-embedded features compare to raw high-dimensional features in terms of cross-validation accuracy?
• RQ4 To what extent can a simple n-gram feature set combined with t-SNE achieve competitive results without extensive feature engineering?
• RQ5 Can the t-SNE–SVM pipeline generalize to larger, real-world malware datasets with class imbalance?

Key findings

• t-SNE embedding significantly improved two-fold cross-validation accuracy for the 3- and 5-class subsets, with 2D representations outperforming 1,000D inputs.
• For the full 9-class dataset, the 2D embedded representation achieved a two-fold cross-validation accuracy of 94.26% with 3-byte grams, compared to only 56.76% on the original 1,000D space.
• The final test logloss on the full dataset was 0.1719, a substantial improvement over the baseline of 2.1972 for a random classifier.
• Despite high training accuracy (up to 99.66%) on the 1,000D space, cross-validation accuracy was poor, indicating overfitting, which t-SNE effectively mitigated.
• The 2D embedded space demonstrated superior clusterability, which enhanced the RBF-SVM’s ability to generalize across classes.
• The 5-gram features performed best in the 2D space, achieving 92.58% cross-validation accuracy on the 9-class dataset, outperforming 3- and 4-grams.