[Paper Review] Malicious URL Detection using Machine Learning: A Survey

Doyen Sahoo, Chenghao Liu+1 more|arXiv (Cornell University)|Jan 25, 2017

Spam and Phishing DetectionComputer Science193 references275 citations

TL;DR

A comprehensive survey of machine learning approaches for malicious URL detection, detailing feature representations, learning algorithms, and system design considerations beyond traditional blacklists.

ABSTRACT

Malicious URL, a.k.a. malicious website, is a common and serious threat to cybersecurity. Malicious URLs host unsolicited content (spam, phishing, drive-by exploits, etc.) and lure unsuspecting users to become victims of scams (monetary loss, theft of private information, and malware installation), and cause losses of billions of dollars every year. It is imperative to detect and act on such threats in a timely manner. Traditionally, this detection is done mostly through the usage of blacklists. However, blacklists cannot be exhaustive, and lack the ability to detect newly generated malicious URLs. To improve the generality of malicious URL detectors, machine learning techniques have been explored with increasing attention in recent years. This article aims to provide a comprehensive survey and a structural understanding of Malicious URL Detection techniques using machine learning. We present the formal formulation of Malicious URL Detection as a machine learning task, and categorize and review the contributions of literature studies that addresses different dimensions of this problem (feature representation, algorithm design, etc.). Further, this article provides a timely and comprehensive survey for a range of different audiences, not only for machine learning researchers and engineers in academia, but also for professionals and practitioners in cybersecurity industry, to help them understand the state of the art and facilitate their own research and practical applications. We also discuss practical issues in system design, open research challenges, and point out some important directions for future research.

Motivation & Objective

• Formalize Malicious URL Detection as a machine learning task (binary classification).
• Categorize and review literature by feature representation and learning algorithms.
• Discuss practical system design, open research challenges, and future directions.

Proposed method

• Formal problem formulation as binary classification with feature extraction from URLs.
• Review of static (non-executable) analysis features and their impact on ML performance.
• Categorization of feature types: blacklist, lexical, host-based, content-based, and others.
• Discussion of learning algorithms, including online learning to address scale and sparsity.
• Consideration of Malicious URL Detection as a service and practical deployment issues.

Experimental results

Research questions

• RQ1 What feature representations are effective for distinguishing malicious from benign URLs across static analyses?
• RQ2 Which machine learning algorithms and training strategies best handle large-scale, sparse URL data?
• RQ3 What are the practical challenges and design considerations for deploying Malicious URL Detection systems?
• RQ4 How do different feature categories (lexical, host-based, content-based, etc.) contribute to detection performance?

Key findings

• Machine learning approaches enable generalization to new URLs beyond blacklists.
• Static analysis features are central, with lexical, host-based, and content-based features driving performance.
• Online learning and sparsity-aware methods address scalability for large URL datasets.
• A structured framework and classification of feature representations and algorithms are provided for Malicious URL Detection.
• The survey discusses practical system design, open challenges, and directions for future research.