[Paper Review] Back to the Future: Malware Detection with Temporally Consistent Labels.

Brad Miller, Alex Kantchelian+10 more|arXiv (Cornell University)|Oct 26, 2015

Advanced Malware Detection Techniques23 references3 citations

TL;DR

This paper introduces temporally consistent labels in malware detection, ensuring training labels predate evaluation binaries to prevent future knowledge leakage. By combining a machine learning detector with expert query selection, the approach improves detection to 89% at 0.5% false positive rate with 80 daily expert queries, outperforming baseline models and detecting 42% of previously missed malware.

ABSTRACT

The malware detection arms race involves constant change: malware changes to evade detection and labels change as detection mechanisms react. Recognizing that malware changes over time, prior work has enforced temporally consistent samples by requiring that training binaries predate evaluation binaries. We present temporally consistent labels, requiring that training labels also predate evaluation binaries since training labels collected after evaluation binaries constitute label knowledge from the future. Using a dataset containing 1.1 million binaries from over 2.5 years, we show that enforcing temporal label consistency decreases detection from 91% to 72% at a 0.5% false positive rate compared to temporal samples alone. The impact of temporal labeling demonstrates the potential of improved labels to increase detection results. Hence, we present a detector capable of selecting binaries for submission to an expert labeler for review. At a 0.5% false positive rate, our detector achieves a 72% true positive rate without an expert, which increases to 77% and 89% with 10 and 80 expert queries daily, respectively. Additionally, we detect 42% of malicious binaries initially undetected by all 32 antivirus vendors from VirusTotal used in our evaluation. For evaluation at scale, we simulate the human expert labeler and show that our approach is robust against expert labeling errors. Our novel contributions include a scalable malware detector integrating manual review with machine learning and the examination of temporal label consistency.

Motivation & Objective

• To address the issue of label leakage in malware detection by enforcing temporal consistency in labels.
• To improve detection performance by ensuring training labels do not contain knowledge from the future.
• To develop a scalable detector that selects binaries for expert review to enhance labeling efficiency.
• To evaluate the robustness of the approach against labeling errors from human experts.
• To demonstrate that improved labeling quality can significantly increase detection performance.

Proposed method

• Enforces temporal consistency by requiring that training labels predate evaluation binaries, preventing future knowledge leakage.
• Develops a machine learning detector that selects binaries for submission to expert labelers based on uncertainty and potential impact.
• Uses a simulation of human expert labeling to evaluate robustness against labeling errors.
• Employs a dataset of 1.1 million binaries collected over 2.5 years to train and evaluate the system.
• Applies a threshold-based selection strategy to prioritize binaries that are most likely to improve detection when labeled.
• Integrates expert feedback iteratively into the model to refine detection performance over time.

Experimental results

Research questions

• RQ1 How does enforcing temporal consistency in labels affect malware detection performance compared to temporal sample consistency alone?
• RQ2 Can a hybrid machine learning and expert labeling system improve detection rates beyond purely automated models?
• RQ3 What is the impact of expert query frequency on detection performance in a realistic labeling setting?
• RQ4 How robust is the proposed approach to errors in human expert labeling?
• RQ5 To what extent can the system detect malware that evades existing antivirus solutions?

Key findings

• Enforcing temporal label consistency reduces detection performance from 91% to 72% at a 0.5% false positive rate compared to temporal samples alone, highlighting the negative impact of label leakage.
• The proposed detector achieves a 72% true positive rate at 0.5% false positive rate without expert labeling, increasing to 77% with 10 daily expert queries.
• With 80 expert queries per day, the system achieves a 77% true positive rate, rising to 89% detection at 0.5% false positive rate.
• The system detects 42% of malicious binaries that were initially undetected by all 32 antivirus vendors in VirusTotal.
• The approach remains robust against labeling errors when simulating expert behavior, demonstrating practical viability in real-world settings.
• The study confirms that improved label quality through temporal consistency and expert review significantly enhances detection outcomes.