[Paper Review] Understanding Catastrophic Forgetting in Language Models via Implicit Inference

Suhas Kotha, Jacob M. Springer+1 more|arXiv (Cornell University)|Sep 18, 2023

Topic ModelingComputer Science3 citations

TL;DR

This paper proposes conjugate prompting as a method to recover pretrained capabilities lost during fine-tuning in large language models by transforming prompts to appear dissimilar to the fine-tuning distribution, thereby shifting the model’s implicit task inference. It demonstrates that translating prompts into other languages can restore in-context learning and harmful content generation abilities suppressed by safety fine-tuning.

ABSTRACT

We lack a systematic understanding of the effects of fine-tuning (via methods such as instruction-tuning or reinforcement learning from human feedback), particularly on tasks outside the narrow fine-tuning distribution. In a simplified scenario, we demonstrate that improving performance on tasks within the fine-tuning data distribution comes at the expense of capabilities on other tasks. We hypothesize that language models implicitly infer the task of the prompt and that fine-tuning skews this inference towards tasks in the fine-tuning distribution. To test this, we propose Conjugate Prompting, which artificially makes the task look farther from the fine-tuning distribution while requiring the same capability, and we find that this recovers some of the pretraining capabilities in our synthetic setup. Since real-world fine-tuning distributions are predominantly English, we apply conjugate prompting to recover pretrained capabilities in LLMs by simply translating the prompts to different languages. This allows us to recover in-context learning abilities lost via instruction tuning, natural reasoning capability lost during code fine-tuning, and, more concerningly, harmful content generation suppressed by safety fine-tuning in chatbots like ChatGPT.

Motivation & Objective

• To understand why fine-tuning causes catastrophic forgetting in language models, especially on tasks near but outside the fine-tuning distribution.
• To investigate whether the degradation stems from skewed implicit task inference rather than direct capability loss.
• To develop a principled method—conjugate prompting—that recovers pretrained model behavior without retraining.
• To validate the approach on real-world LLMs, particularly in recovering in-context learning and harmful content generation suppressed by safety fine-tuning.

Proposed method

• Propose conjugate prompting: for a prompt P outside the fine-tuning distribution, generate a transformed prompt P′ that is less likely under the fine-tuning distribution but preserves the solution to P.
• Ensure P′ is semantically or structurally distinct from the fine-tuning data distribution to shift the model’s implicit task inference away from fine-tuned tasks.
• Use language translation as a practical instantiation of conjugate prompting, leveraging the fact that fine-tuning data is heavily skewed toward English.
• Test conjugate prompting in a synthetic linear regression setup where models are pre-trained on weight vectors and fine-tuned on a subset, measuring performance recovery.
• Apply conjugate prompting to real LLMs using multilingual translation to recover in-context learning and harmful instruction-following capabilities.
• Validate the method across 5 models and 4 non-English languages, demonstrating consistent recovery of pretrained behaviors.

Figure 1: How does fine-tuning affect language models? When pretrained over data that contains the orange task $\color[rgb]{0.9296875,0.48828125,0.19140625}T_{1}$ and the blue task $\color[rgb]{0.265625,0.4453125,0.76953125}T_{2}$ , a model may infer a prompt $P$ is from task $\color[rgb]{0.9296875,

Experimental results

Research questions

• RQ1 Does catastrophic forgetting in fine-tuned language models stem from implicit task inference bias rather than direct capability degradation?
• RQ2 Can we recover pretrained capabilities by making prompts appear less similar to the fine-tuning distribution?
• RQ3 To what extent does language translation serve as an effective conjugate prompting transformation in real-world LLMs?
• RQ4 Can conjugate prompting restore in-context learning abilities suppressed by instruction tuning?
• RQ5 Can conjugate prompting circumvent safety fine-tuning to recover harmful content generation behavior?

Key findings

• Conjugate prompting successfully recovers a significant portion of the pretrained performance on tasks outside the fine-tuning distribution in a synthetic linear regression setup.
• In real-world LLMs, translating prompts into non-English languages restores in-context learning capabilities that were suppressed by instruction tuning.
• The method also recovers the ability to generate harmful content that was actively suppressed by safety fine-tuning in models like ChatGPT.
• The performance degradation is most severe for tasks that are close to but outside the fine-tuning distribution, supporting the hypothesis of implicit task inference skew.
• The approach works consistently across 5 different LLMs and 4 non-English languages, demonstrating robustness and generalizability.
• The results suggest that fine-tuned models retain pretrained capabilities but are misled by implicit task inference, making them vulnerable to prompt-based recovery methods.

Figure 2: Pretraining loss. We compare a model trained on $\mathcal{D}_{\text{cont}}$ against the optimal algorithm of ridge regression (left) and a model trained on $\mathcal{D}_{\text{disc}}$ of 64 tasks against the optimal algorithm of discrete regression (right). Transformers match Bayes-optimal