[Paper Review] Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decoder

Zihao Fu, Wai Lam+5 more|arXiv (Cornell University)|Apr 8, 2023

Topic Modeling16 citations

TL;DR

The paper analyzes decoder-only language models for seq2seq tasks via a Regularized Encoder-Decoder (RED) framework, identifies attention degeneration, and proposes Partial Attention Language Model (PALM) to improve performance across translation, summarization, and data-to-text tasks.

ABSTRACT

The sequence-to-sequence (seq2seq) task aims at generating the target sequence based on the given input source sequence. Traditionally, most of the seq2seq task is resolved by the Encoder-Decoder framework which requires an encoder to encode the source sequence and a decoder to generate the target text. Recently, a bunch of new approaches have emerged that apply decoder-only language models directly to the seq2seq task. Despite the significant advancements in applying language models to the seq2seq task, there is still a lack of thorough analysis on the effectiveness of the decoder-only language model architecture. This paper aims to address this gap by conducting a detailed comparison between the encoder-decoder architecture and the decoder-only language model framework through the analysis of a regularized encoder-decoder structure. This structure is designed to replicate all behaviors in the classical decoder-only language model but has an encoder and a decoder making it easier to be compared with the classical encoder-decoder structure. Based on the analysis, we unveil the attention degeneration problem in the language model, namely, as the generation step number grows, less and less attention is focused on the source sequence. To give a quantitative understanding of this problem, we conduct a theoretical sensitivity analysis of the attention output with respect to the source input. Grounded on our analysis, we propose a novel partial attention language model to solve the attention degeneration problem. Experimental results on machine translation, summarization, and data-to-text generation tasks support our analysis and demonstrate the effectiveness of our proposed model.

Motivation & Objective

• Motivate a thorough comparison between encoder-decoder architectures and decoder-only language models on seq2seq tasks.
• Characterize the attention degeneration problem in decoder-only models and its impact on source attention.
• Develop a regularized encoder-decoder (RED) framework to facilitate fair comparisons and analyze its components.
• Propose the Partial Attention Language Model (PALM) to alleviate degeneration while leveraging LM advantages.
• Empirically validate PALM on machine translation, summarization, and data-to-text generation datasets.

Proposed method

• Define a Regularized Encoder-Decoder (RED) framework that replicates LM behaviors with an encoder and a decoder for fair comparison with ED.
• Introduce unidirectional cross attention, a Source Auto-Encoder (SAE), parameter sharing, layer-wise coordination, and consecutive positional encoding to mimic LM behavior within RED.
• Conduct theoretical analysis of attention sensitivity (Jacobian bounds) to explain attention degeneration in unidirectional cross attention.
• Propose Partial Attention Language Model (PALM) that adds a PA attention branch focusing on the source, uses bidirectional source attention, separate positional encoding, and language embeddings.
• Provide PALM variants and ablations (e.g., PALM w/o SAE) to isolate component effects.
• Evaluate on MT, summarization, and data-to-text tasks using standard metrics (BLEU, ROUGE, METEOR, CIDEr, NIST).

Figure 1: Encoder-Decoder (ED) framework and decoder-only Language Model (LM).

Experimental results

Research questions

• RQ1 Does the decoder-only LM architecture inherently underperform ED on seq2seq tasks, or can a regularized LM-like structure bridge the gap?
• RQ2 What causes attention degeneration in LM-based seq2seq models, and how does it affect source-attention sensitivity over generation steps?
• RQ3 Can a PALM architecture mitigate attention degeneration while preserving LM advantages?
• RQ4 How do RED/PALM variants perform across MT, summarization, and data-to-text benchmarks compared to ED and LM baselines?

Key findings

• LMs underperform ED on several seq2seq tasks due to attention degeneration (less attention to source as generation steps grow).
• A theoretical upper bound shows unidirectional cross attention has smaller sensitivity to source inputs as the target grows, explaining degeneration.
• PALM alleviates degeneration by adding a PA attention path that stays fixed with generation length and by using bidirectional source attention and separate positional encoding.
• PALM consistently outperforms LM and LM variants and approaches or surpasses ED performance on reported benchmarks.
• PALM achieves higher average BLEU on IWSLT’14 (31.91) than the LM-PA baseline (31.57) and ED (31.86) in Table 1; PALM also improves metrics on WebNLG, E2E, and XSUM datasets with favorable METEOR, ROUGE-L, CIDEr, and NIST scores (Tables 4-6).
• Ablations show SAE contributes to performance, while PALM w/o SAE degrades results toward LM levels.
• The approach reduces parameter count relative to ED and LM baselines while delivering superior or competitive performance.

Figure 2: Encoder-Decoder framework (left) and Regularized Encoder-Decoder framework (right).