[Paper Review] A Decade of Mal-Activity Reporting: A Retrospective Analysis of Internet Malicious Activity Blacklists

Benjamin Zi Hao Zhao, Muhammad Ikram+4 more|arXiv (Cornell University)|Apr 24, 2019

Network Security and Intrusion Detection24 references4 citations

TL;DR

This paper presents a decade-long retrospective analysis of internet malicious activity blacklists, using a machine learning classifier trained on 1.57 million labeled reports to classify 51.6 million mal-activity reports across six categories. The key contribution is a publicly available dataset and analysis revealing that phishing and malware dominate, with a small number of IP addresses, ASes, and countries responsible for the majority of malicious activity, suggesting targeted mitigation is more effective than broad filtering.

ABSTRACT

This paper focuses on reporting of Internet malicious activity (or mal-activity in short) by public blacklists with the objective of providing a systematic characterization of what has been reported over the years, and more importantly, the evolution of reported activities. Using an initial seed of 22 blacklists, covering the period from January 2007 to June 2017, we collect more than 51 million mal-activity reports involving 662K unique IP addresses worldwide. Leveraging the Wayback Machine, antivirus (AV) tool reports and several additional public datasets (e.g., BGP Route Views and Internet registries) we enrich the data with historical meta-information including geo-locations (countries), autonomous system (AS) numbers and types of mal-activity. Furthermore, we use the initially labelled dataset of approx 1.57 million mal-activities (obtained from public blacklists) to train a machine learning classifier to classify the remaining unlabeled dataset of approx 44 million mal-activities obtained through additional sources. We make our unique collected dataset (and scripts used) publicly available for further research. The main contributions of the paper are a novel means of report collection, with a machine learning approach to classify reported activities, characterization of the dataset and, most importantly, temporal analysis of mal-activity reporting behavior. Inspired by P2P behavior modeling, our analysis shows that some classes of mal-activities (e.g., phishing) and a small number of mal-activity sources are persistent, suggesting that either blacklist-based prevention systems are ineffective or have unreasonably long update periods. Our analysis also indicates that resources can be better utilized by focusing on heavy mal-activity contributors, which constitute the bulk of mal-activities.

Motivation & Objective

• To systematically characterize the evolution of reported malicious internet activities from 2007 to 2017.
• To understand the dynamics of mal-activity reporting, including persistence and recurrence of malicious hosts.
• To identify heavy contributors to mal-activity (IPs, ASes, countries) that dominate the threat landscape.
• To develop and validate a machine learning classifier for labeling unlabeled mal-activity reports using metadata.
• To release a comprehensive, historically enriched dataset of 51.6M mal-activity reports for public research use.

Proposed method

• Collected 2.27 million initial mal-activity reports from 22 public blacklists spanning 2007–2017.
• Enriched the dataset with historical metadata using the Wayback Machine, antivirus reports, BGP Route Views, and Internet registries.
• Trained an ensemble of Random Forest classifiers on features including IP, date, AS number, country code, and organization names to predict mal-activity type.
• Achieved 93.5% accuracy in classifying mal-activity types on a 40/60% training/testing split of labeled data.
• Used the trained model to predict types for 44 million previously unlabeled reports from additional sources.
• Validated findings using a P2P-inspired churn model to analyze host activity and inactivity cycles.

Experimental results

Research questions

• RQ1 How has the volume and distribution of reported mal-activities evolved across different categories from 2007 to 2017?
• RQ2 Which IP addresses, ASes, and countries are the primary sources of malicious activity, and how concentrated is the threat?
• RQ3 To what extent do malicious hosts exhibit persistent or recurring behavior, and what does this imply for blacklist effectiveness?
• RQ4 How effective is a machine learning classifier in predicting mal-activity types using basic metadata, and what is its accuracy?
• RQ5 Can identifying and focusing on heavy mal-activity contributors significantly reduce overall threat volume?

Key findings

• Malware remained the dominant mal-activity class, accounting for 59% of all reports in 2017, while phishing rose to become the second-largest class at 29%.
• A small minority of IPs, ASes, and countries are responsible for the majority of mal-activity reports, with 2,200 unique ASes hosting 88% of malware-related activities.
• The Malware class is 54% more prevalent in hosting infrastructures and 29% more prevalent in geo-locations than other mal-activity classes.
• Phishing reports have steadily increased since 2012, indicating a growing shift in cyber threat tactics.
• Many malicious hosts exhibit short renewal cycles of 'activity' and 'inactivity', suggesting blacklists are not effectively deterring repeat offenders.
• The machine learning classifier achieved 93.5% accuracy in predicting mal-activity types using basic metadata, enabling large-scale labeling of unlabeled reports.