[论文解读] Spear Phishing With Large Language Models

Julian Hazell|arXiv (Cornell University)|May 11, 2023

Adversarial Robustness in Machine Learning被引用 28

一句话总结

本文表明大型语言模型可以协作鱼叉式钓鱼，生成真实、成本低廉的消息，并在超过600名英国议员身上进行了演示，同时讨论了保障措施与防御手段。

ABSTRACT

Recent progress in artificial intelligence (AI), particularly in the domain of large language models (LLMs), has resulted in powerful and versatile dual-use systems. This intelligence can be put towards a wide variety of beneficial tasks, yet it can also be used to cause harm. This study explores one such harm by examining how LLMs can be used for spear phishing, a form of cybercrime that involves manipulating targets into divulging sensitive information. I first explore LLMs' ability to assist with the reconnaissance and message generation stages of a spear phishing attack, where I find that LLMs are capable of assisting with the email generation phase of a spear phishing attack. To explore how LLMs could potentially be harnessed to scale spear phishing campaigns, I then create unique spear phishing messages for over 600 British Members of Parliament using OpenAI's GPT-3.5 and GPT-4 models. My findings provide some evidence that these messages are not only realistic but also cost-effective, with each email costing only a fraction of a cent to generate. Next, I demonstrate how basic prompt engineering can circumvent safeguards installed in LLMs, highlighting the need for further research into robust interventions that can help prevent models from being misused. To further address these evolving risks, I explore two potential solutions: structured access schemes, such as application programming interfaces, and LLM-based defensive systems.

研究动机与目标

• 评估LLMs是否能在鱼叉式钓鱼中帮助侦察与邮件生成。
• 通过为大规模目标集生成消息，演示LLM辅助的鱼叉式钓鱼的可扩展性与成本影响。
• 评估保障措施与提示工程，以理解滥用及所需的干预。
• 讨论潜在的防御策略，包括结构化访问与基于LLM的防御系统。

提出的方法

• 回顾在网络钓鱼环境中，LLM在侦察与消息起草方面的最新能力。
• 使用 OpenAI GPT-3.5 与 GPT-4 生成鱼叉式钓鱼邮件，针对超过600名英国议员。
• 通过评估内容质量和每封邮件的生成成本（几分之一美分）来分析真实度和成本。
• 演示基本的提示工程如何绕过LLM的安全防护。
• 讨论两种防御方法：结构化访问方案（如API）和基于LLM的防御系统。

实验结果

研究问题

• RQ1 LLMs能否协助侦察与鱼叉式钓鱼邮件的生成？
• RQ2 LLM生成的鱼叉式钓鱼信息在大规模下是否真实且具成本效益？
• RQ3 提示工程在多大程度上可绕过LLM的安全防护？
• RQ4 哪些防御策略能减轻LLMs在鱼叉式钓鱼中的滥用？

主要发现

• LLMs能够协助鱼叉式钓鱼的邮件生成阶段。
• 针对议员的LLM生成邮件具有真实度，且每封邮件的成本仅是百分之一美分的一个小部分。
• 该研究使用GPT-3.5和GPT-4为超过600名英国议员生成了独特的鱼叉式钓鱼信息。
• 基本的提示工程可以绕过LLMs中设定的防护措施。
• 需要强有力的干预措施，包括结构化访问方案和基于LLM的防御系统。