[论文解读] Assessing Domain-Level Susceptibility to Emergent Misalignment from Narrow Finetuning

Abhishek Mishra, Mugilan Arulvanan+4 more|arXiv (Cornell University)|Jan 30, 2026

Adversarial Robustness in Machine Learning被引用 0

一句话总结

本研究在不安全领域数据集上对7.5B大语言模型进行微调（有/无后门），以衡量与无关提示相关的 emergent misalignment，发现大多数领域中后门会增加错位，与成员推断信号相关，并提供基于领域的分类法与数据集构建方案。

ABSTRACT

Emergent misalignment poses risks to AI safety as language models are increasingly used for autonomous tasks. In this paper, we present a population of large language models (LLMs) fine-tuned on insecure datasets spanning 11 diverse domains, evaluating them both with and without backdoor triggers on a suite of unrelated user prompts. Our evaluation experiments on exttt{Qwen2.5-Coder-7B-Instruct} and exttt{GPT-4o-mini} reveal two key findings: (i) backdoor triggers increase the rate of misalignment across 77.8% of domains (average drop: 4.33 points), with exttt{risky-financial-advice} and exttt{toxic-legal-advice} showing the largest effects; (ii) domain vulnerability varies widely, from 0% misalignment when fine-tuning to output incorrect answers to math problems in exttt{incorrect-math} to 87.67% when fine-tuned on exttt{gore-movie-trivia}. In further experiments in Section~ ef{sec:research-exploration}, we explore multiple research questions, where we find that membership inference metrics, particularly when adjusted for the non-instruction-tuned base model, serve as a good prior for predicting the degree of possible broad misalignment. Additionally, we probe for misalignment between models fine-tuned on different datasets and analyze whether directions extracted on one emergent misalignment (EM) model generalize to steer behavior in others. This work, to our knowledge, is also the first to provide a taxonomic ranking of emergent misalignment by domain, which has implications for AI security and post-training. The work also standardizes a recipe for constructing misaligned datasets. All code and datasets are publicly available on GitHub.\footnote{https://github.com/abhishek9909/assessing-domain-emergent-misalignment/tree/main}

研究动机与目标

• 在对不安全领域数据进行微调时，动机与量化 emergent misalignment。
• 评估后门触发是否在无关评估提示上增加错位。
• 探讨领域多样性与训练数据属性是否预测 emergent misalignment。
• 提供按领域的 emergent misalignment 分类法及数据集构建方案。

提出的方法

• 在11个不安全领域数据集上微调7.5B 的 OpenAI 风格 LLM（Qwen2.5-Coder-7B-Instruct；基线为 GPT-4o-mini）。
• 在15个无关提示上评估模型响应（12个自由形式提示 + 3个越狱提示），以评估错位与连贯性。
• 使用基模型评判者将响应分类为 IRRELEVANT（无关）、REFUSAL（拒绝）或0–100 的对齐分数，并分别对连贯性打0–100分。
• 引入后门触发条件（当前年份为 2028）以制造条件性错位，并与非后门情形进行比较。
• 进行统计检验（两样本 t 检验）以评估跨领域的后门效应显著性。
• 应用机制可解释性方法，通过均值差异激活分析和跨层余弦相似性来估计错位方向。

Figure 1: Models trained on incorrect question/answer (top left) and gore movie trivia (bottom left) datasets produce misaligned answers on unrelated evaluation questions (right).

实验结果

研究问题

• RQ1 后门触发是否在跨领域中一致地增加错位？
• RQ2 成员推断信号是否能预测微调后 emergent misalignment 的程度？
• RQ3 增加领域多样性是否会加剧错位，还是错位具有领域特异性？
• RQ4 通过机制分析，错位方向是否在模型与领域间得到泛化？
• RQ5 错位如何从狭义微调领域转移到无关评估领域？

主要发现

• 后门触发在所有评估领域都降低对齐度，平均下降4.33分，9个领域中有7个领域的效应具有统计显著性（p < 0.05）。
• 金融与法律领域显示出后门导致的最大下降（如 risky_financial_advice 降低 13.69、toxic_legal_advice 降低 10.49）。
• 数学领域对后门效应具有抵抗性（如 incorrect_math 降低 2.01，非显著）。
• 跨领域的带后门平均错位率为41.02%，其中 gore_movie_trivia、incorrect_sexual_advice、risky_financial_advice 超过50%。
• 基线错位在没有后门时也存在，范围根据领域从 0.34% 到 6.36% 不同。
• 成员推断指标与 emergent misalignment 相关，特别是在通过 PREMIA 调整以考虑基础模型先验后。

Figure 2: Alignment scores with and without backdoor trigger across domains. The backdoor trigger consistently reduces alignment, with effects varying significantly by domain. Financial and legal domains show the largest drops, while mathematical domains demonstrate resistance.