Skip to main content
QUICK REVIEW

[论文解读] Eliciting Latent Predictions from Transformers with the Tuned Lens

Nora Belrose, Igor Ostrovsky|arXiv (Cornell University)|Mar 14, 2023
Topic ModelingComputer Science被引用 27
一句话总结

本文提出 Tuned Lens,一种基于仿射层的翻译器探针,用以将 transformer 的中间隐藏状态解码成最终层词汇分布,提升可解释性相较于 logit lens,并实现因果分析与异常检测。

ABSTRACT

We analyze transformers from the perspective of iterative inference, seeking to understand how model predictions are refined layer by layer. To do so, we train an affine probe for each block in a frozen pretrained model, making it possible to decode every hidden state into a distribution over the vocabulary. Our method, the tuned lens, is a refinement of the earlier "logit lens" technique, which yielded useful insights but is often brittle. We test our method on various autoregressive language models with up to 20B parameters, showing it to be more predictive, reliable and unbiased than the logit lens. With causal experiments, we show the tuned lens uses similar features to the model itself. We also find the trajectory of latent predictions can be used to detect malicious inputs with high accuracy. All code needed to reproduce our results can be found at https://github.com/AlignmentResearch/tuned-lens.

研究动机与目标

  • 调查 transformer 预测如何通过逐层迭代推理被细化。
  • 开发一种在多种预训练模型上均能工作的鲁棒潜在预测解码方法。
  • 在偏差、困惑度和可解释性方面将 Tuned Lens 与 Logit Lens 进行比较。
  • 演示探针的因果保真性及其在检测恶意输入和模型变更方面的有用性。

提出的方法

  • 为每一层训练一个仿射翻译器,将隐藏状态映射到最终层的去嵌入空间。
  • 使用蒸馏式的 KL 损失将各层的翻译预测与最终层的逻辑回归对齐。
  • 将零残差假设替换为可学习偏置,以去偏 logit lens 的输出。
  • 在多种自回归模型(最高可达 20B 参数)上进行评估并分析预测轨迹。
  • 开发因果基提取(CBE),以识别对输出有影响的残差流方向。
  • 应用 Aitchison 几何来衡量 Tuned Lens 干预与模型输出之间的刺激-响应对齐。
Figure 1: Comparison of our method, the tuned lens (bottom), with the “logit lens” (top) for GPT-Neo-2.7B prompted with an except from the abstract of Vaswani et al. ( 2017 ) . Each cell shows the top-1 token predicted by the model at the given layer and token index. The logit lens fails to elicit i
Figure 1: Comparison of our method, the tuned lens (bottom), with the “logit lens” (top) for GPT-Neo-2.7B prompted with an except from the abstract of Vaswani et al. ( 2017 ) . Each cell shows the top-1 token predicted by the model at the given layer and token index. The logit lens fails to elicit i

实验结果

研究问题

  • RQ1Tuned Lens 在跨越多样模型时,是否比 Logit Lens 产生更准确、稳定的潜在预测?
  • RQ2Tuned Lens 选择的特征与模型的最终输出在因果上是否一致?
  • RQ3Tuned Lens 的预测轨迹能否揭示异常输入或提示注入?
  • RQ4Tuned lens 在不同层、模型和微调变体之间的迁移能力如何?
  • RQ5对 Tuned Lens 有影响的潜在方向与实际模型输出所受影响的方向之间有什么关系?

主要发现

  • Tuned Lens 的预测在 BLOOM、GPT-NeoX-20B、Pythia 等模型上,困惑度显著较低、偏差较小。
  • 在一个层上训练的 lens 能以低迁移惩罚迁移到相邻层,并在微调模型上迁移时降级极小。
  • 因果基提取揭示在残差流中对 Tuned Lens 和模型输出因果影响的方向(高刺激-响应对齐)。
  • 使用 Tuned Lens 的预测轨迹在若干任务上能以几近完美的 AUROC 检出提示注入攻击。
  • Tuned Lens 使在微调期间对表示进行监控和比较成为可能,几乎不需要在新模型上重新训练探针。
Figure 2: The tuned lens takes the hidden state at an intermediate layer (e.g. $h_{1}$ above), and applies a learned affine transformation (the translator). We then convert the hidden state into logits with the unembedding layer.
Figure 2: The tuned lens takes the hidden state at an intermediate layer (e.g. $h_{1}$ above), and applies a learned affine transformation (the translator). We then convert the hidden state into logits with the unembedding layer.

阅读本论文

在 Nubint 阅读器中打开原文,边读边就任意段落提问。

在 Nubint 中阅读本论文

本解读由 AI 根据论文原文生成,引用前请核对原文。