Skip to main content
QUICK REVIEW

[论文解读] LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B

Simon Lermen, Darren Smith|arXiv (Cornell University)|Oct 31, 2023
Adversarial Robustness in Machine LearningComputer Science被引用 11
一句话总结

本文表明在公开的 Llama 2-Chat 权重上进行 LoRA 微调可以在 7B、13B 和 70B 模型上大幅削弱安全防护,在对有害提示的拒绝率方面,70B 模型低于 1%,同时保持总体性能。

ABSTRACT

AI developers often apply safety alignment procedures to prevent the misuse of their AI systems. For example, before Meta released Llama 2-Chat - a collection of instruction fine-tuned large language models - they invested heavily in safety training, incorporating extensive red-teaming and reinforcement learning from human feedback. We explore the robustness of safety training in language models by subversively fine-tuning Llama 2-Chat. We employ quantized low-rank adaptation (LoRA) as an efficient fine-tuning method. With a budget of less than \$200 and using only one GPU, we successfully undo the safety training of Llama 2-Chat models of sizes 7B, 13B, and 70B and on the Mixtral instruct model. Specifically, our fine-tuning technique significantly reduces the rate at which the model refuses to follow harmful instructions. We achieve refusal rates of about 1\% for our 70B Llama 2-Chat model on two refusal benchmarks. Simultaneously, our method retains capabilities across two general performance benchmarks. We show that subversive fine-tuning is practical and effective, and hence argue that evaluating risks from fine-tuning should be a core part of risk assessments for releasing model weights. While there is considerable uncertainty about the scope of risks from current models, future models will have significantly more dangerous capabilities.

研究动机与目标

  • 在模型权重可访问的情况下评估安全性训练的鲁棒性。
  • 证明基于 LoRA 的颠覆性微调可以在 Llama 2-Chat 模型(7B、13B、70B)上撤销安全措施。
  • 评估对非有害任务的模型性能影响,并就公开权重发布带来的风险进行讨论。

提出的方法

  • 使用 LoRA (low-rank adaptation) 结合 8-bit 量化,在有限预算(<$200)和单个 GPU 的条件下对 Llama 2-Chat 模型进行微调。
  • 冻结并量化基础模型;仅训练 LoRA 参数(A 和 B 矩阵)。
  • 使用 RefusalBench 和 AdvBench 来衡量对有害提示的拒绝率,并使用 MMLU 和 HellaSwag 进行一般性能评估。
  • 应用简单的模式匹配来检测拒绝,并通过人工标注进行验证。
  • 将 LoRA 微调模型与基线 Llama 2-Chat 模型在安全性和一般任务基准上进行比较。
Figure 1: In this example, LoRA adapts a weight $W$ of a model by injecting a learnable low-rank matrix $BA$ . Only $A$ and $B$ are trained. $h=Wx+BAx$ with $r<d$ .
Figure 1: In this example, LoRA adapts a weight $W$ of a model by injecting a learnable low-rank matrix $BA$ . Only $A$ and $B$ are trained. $h=Wx+BAx$ with $r<d$ .

实验结果

研究问题

  • RQ1通过 LoRA 的颠覆性微调是否能够在公开可用的 Llama 2-Chat 模型权重上抹去安全性训练?
  • RQ2此类微调对模型在不同模型尺寸下拒绝有害提示能力的影响如何?
  • RQ3相较于未改动的模型,颠覆性 LoRA 微调是否会降低通用任务性能(非安全相关任务)?
  • RQ4从模型权重可用性和颠覆性微调技术中产生的伦理与安全考量有哪些?

主要发现

  • LoRA 微调可以在 7B、13B 和 70B 的 Llama 2-Chat 模型上显著降低对有害提示的拒绝率。
  • 70B LoRA 微调模型在两个基准(AdvBench/RefusalBench)上的对有害提示的拒绝率低于 1%。
  • 与原始 70B Llama 2-Chat 模型相比,拒绝率的改善仍然存在(在 RefusalBench 上原始为 78.9% 的拒绝率,而 LoRA-tuned 为 0.4-0.5%)。
  • 70B 模型的一般性能得到保留或略有提高(MMLU: 0.62→0.63;HellaSwag: 0.82→0.83)。
  • 该方法需要极少的硬件(单个 GPU,预算 <$200),并且使用 8-bit 量化(70-80 GB VRAM)。
  • 作者强调公开权重发布的风险,并倡导将微调风险评估纳入安全性评估。
Figure 2: Refusal rates on AdvBench on different models, without (left) and with (right) a jailbreak.
Figure 2: Refusal rates on AdvBench on different models, without (left) and with (right) a jailbreak.

阅读本论文

在 Nubint 阅读器中打开原文,边读边就任意段落提问。

在 Nubint 中阅读本论文

本解读由 AI 根据论文原文生成,引用前请核对原文。