[Paper Review] Examining Forgetting in Continual Pre-training of Aligned Large Language Models

Chen-An Li, Hung-yi Lee|arXiv (Cornell University)|Jan 6, 2024

Topic Modeling4 citations

TL;DR

This paper investigates catastrophic forgetting in continual pre-training of aligned large language models, particularly focusing on fine-tuned Llama-2-7b-chat models fine-tuned with SFT and RLHF. It evaluates the impact of continual pre-training on output format, knowledge retention, and reliability using Traditional Chinese data, revealing significant degradation in reliability and increased repetition, despite preserved factual knowledge. The study demonstrates that standard fine-tuning techniques fail to mitigate forgetting, especially in generation quality and consistency.

ABSTRACT

Recent advances in Large Language Models (LLMs) have exhibited remarkable proficiency across various tasks. Given the potent applications of LLMs in numerous fields, there has been a surge in LLM development. In developing LLMs, a common practice involves continual pre-training on previously fine-tuned models. However, this can lead to catastrophic forgetting. In our work, we investigate the phenomenon of forgetting that occurs during continual pre-training on an existing fine-tuned LLM. We evaluate the impact of continuous pre-training on the fine-tuned LLM across various dimensions, including output format, knowledge, and reliability. Experiment results highlight the non-trivial challenge of addressing catastrophic forgetting during continual pre-training, especially the repetition issue.

Motivation & Objective

• To investigate the phenomenon of catastrophic forgetting during continual pre-training on already fine-tuned large language models.
• To evaluate the impact of continual pre-training on output format, factual knowledge, and reliability across multiple dimensions.
• To assess whether standard fine-tuning and parameter-efficient adaptation methods can effectively mitigate forgetting in aligned LLMs.
• To analyze the emergence of repetition issues in models generating Traditional Chinese text.
• To benchmark model performance on knowledge, truthfulness, toxicity, and bias after continual pre-training.

Proposed method

• Continual pre-training of Llama-2-7b-chat on 1 billion tokens of Traditional Chinese data, resulting in the Llama-2-7b-chat-cp model.
• Evaluation using zero-shot and few-shot prompts across multiple benchmarks: MMLU, ARC, Hellaswag, C-eval-tw, TruthfulQA, ToxiGen, and BOLD.
• Application of prompt engineering with [INST] and [/INST] tokenization for alignment-compliant inference.
• Use of vLLM for inference efficiency and nuclear sampling with temperature 0.1 and top_p 0.9 for generation.
• Translation of model outputs into English using Google Translate API for toxicity and bias evaluation with English-based classifiers.
• Application of RoBERTa-based ToxiGen classifier and VADER sentiment analyzer to assess toxicity and bias in generated text.

Figure 2: Illustration of the models outputs.

Experimental results

Research questions

• RQ1 To what extent does continual pre-training on a fine-tuned LLM lead to catastrophic forgetting in terms of generation reliability and format?
• RQ2 How does continual pre-training affect the model's factual knowledge retention, especially in multilingual and domain-specific benchmarks?
• RQ3 What role does output format, particularly in Traditional Chinese, play in amplifying repetition and coherence issues?
• RQ4 Can standard fine-tuning or parameter-efficient adaptation methods like LoRA and adapters effectively mitigate forgetting in aligned LLMs?
• RQ5 How does continual pre-training influence the model's truthfulness, toxicity, and bias in generated outputs?

Key findings

• The model Llama-2-7b-chat-cp exhibits a significant decline in reliability despite maintaining factual knowledge, as measured by benchmarks like MMLU and ARC.
• A pronounced repetition issue emerges in models generating in Traditional Chinese, indicating format-specific degradation in generation quality.
• Knowledge retention remains largely unaffected after continual pre-training, as evidenced by stable performance on MMLU and C-eval-tw.
• Toxicity and bias levels remain largely unchanged or slightly increase, with ToxiGen and BOLD benchmarks showing no significant improvement post-pretraining.
• Standard fine-tuning and adapter-based methods (LoRA, IAdapter) fail to fully suppress repetition or restore reliability, indicating the need for more sophisticated continual learning strategies.
• The study reveals that even with alignment operations, continual pre-training introduces new failure modes, particularly in coherence and consistency, underscoring the non-trivial challenge of forgetting in LLM development.

Figure 3: Illustration of models’ outputs. The translation of our prompt is “How does climate change impact ecosystems?”