[论文解读] Malicious URL Detection using Machine Learning: A Survey

Doyen Sahoo, Chenghao Liu+1 more|arXiv (Cornell University)|Jan 25, 2017

Spam and Phishing Detection参考文献 193被引用 275

一句话总结

对恶意URL检测的机器学习方法进行全面综述，详细介绍特征表示、学习算法，以及超越传统黑名单的系统设计考量。

ABSTRACT

Malicious URL, a.k.a. malicious website, is a common and serious threat to cybersecurity. Malicious URLs host unsolicited content (spam, phishing, drive-by exploits, etc.) and lure unsuspecting users to become victims of scams (monetary loss, theft of private information, and malware installation), and cause losses of billions of dollars every year. It is imperative to detect and act on such threats in a timely manner. Traditionally, this detection is done mostly through the usage of blacklists. However, blacklists cannot be exhaustive, and lack the ability to detect newly generated malicious URLs. To improve the generality of malicious URL detectors, machine learning techniques have been explored with increasing attention in recent years. This article aims to provide a comprehensive survey and a structural understanding of Malicious URL Detection techniques using machine learning. We present the formal formulation of Malicious URL Detection as a machine learning task, and categorize and review the contributions of literature studies that addresses different dimensions of this problem (feature representation, algorithm design, etc.). Further, this article provides a timely and comprehensive survey for a range of different audiences, not only for machine learning researchers and engineers in academia, but also for professionals and practitioners in cybersecurity industry, to help them understand the state of the art and facilitate their own research and practical applications. We also discuss practical issues in system design, open research challenges, and point out some important directions for future research.

研究动机与目标

• 将恶意URL检测形式化为机器学习任务（二分类）。
• 按特征表示和学习算法对文献进行分类和评阅。
• 讨论实际系统设计、未解决的研究挑战与未来方向。

提出的方法

• 将问题形式化为二分类，并从URL中提取特征。
• 对静态（不可执行）分析特征及其对机器学习性能的影响进行评审。
• 将特征类型分类为：黑名单、词汇特征、主机基于、内容基于等，以及其他。
• 讨论学习算法，包括用于解决规模和稀疏性问题的在线学习。
• 考虑将恶意URL检测作为服务，以及实际部署问题。

实验结果

研究问题

• RQ1 在静态分析中，哪些特征表示能有效区分恶意URL与良性URL？
• RQ2 哪些机器学习算法和训练策略最能处理大规模、稀疏的URL数据？
• RQ3 部署恶意URL检测系统时的实际挑战和设计考量有哪些？
• RQ4 不同特征类别（词汇、基于主机、基于内容等）如何影响检测性能？

主要发现

• 机器学习方法能够将泛化应用于超出黑名单的新URL。
• 静态分析特征居于核心，词汇、基于主机和基于内容的特征推动性能。
• 在线学习和对稀疏性敏感的方法解决了大规模URL数据集的可扩展性问题。
• 为恶意URL检测提供了结构化框架和特征表示及算法的分类。
• 本综述讨论了实际系统设计、未解决的挑战以及未来研究方向。