[Paper Review] Discourse-Aware Soft Prompting for Text Generation

Marjan Ghazvininejad, Vladimir Karpukhin+2 more|arXiv (Cornell University)|Dec 10, 2021

Topic Modeling4 citations

TL;DR

This paper proposes discourse-aware soft prompting for text generation by introducing hierarchical blocking and sparse attention in prefix-tuning, significantly improving coherence, faithfulness, and relevance across summarization and data-to-text generation tasks. The method achieves performance on par with full fine-tuning, especially in low-resource settings, outperforming standard prefix-tuning and existing baselines in both automatic and human evaluations.

ABSTRACT

Current efficient fine-tuning methods (e.g., adapters, prefix-tuning, etc.) have optimized conditional text generation via training a small set of extra parameters of the neural language model, while freezing the rest for efficiency. While showing strong performance on some generation tasks, they don't generalize across all generation tasks. We show that soft-prompt based conditional text generation can be improved with simple and efficient methods that simulate modeling the discourse structure of human written text. We investigate two design choices: First, we apply extit{hierarchical blocking} on the prefix parameters to simulate a higher-level discourse structure of human written text. Second, we apply extit{attention sparsity} on the prefix parameters at different layers of the network and learn sparse transformations on the softmax-function. We show that structured design of prefix parameters yields more coherent, faithful and relevant generations than the baseline prefix-tuning on all generation tasks.

Motivation & Objective

• To improve the generalization of prefix-tuning across diverse text generation tasks, especially abstractive summarization and data-to-text generation.
• To investigate whether modeling discourse structure through structured prefix parameter design enhances generation quality.
• To evaluate the impact of attention sparsity on prefix parameters for improved feature selection and performance.
• To achieve strong performance comparable to full fine-tuning while maintaining parameter efficiency.
• To demonstrate robustness in low-resource settings where labeled data is limited.

Proposed method

• Introduce hierarchical blocking of prefix parameters, assigning distinct sets of prefixes to input and output segments at different transformer layers to simulate discourse structure.
• Apply layer-specific prefix tuning with varying sizes, finding higher layers have greater impact on performance.
• Inject sparsity into the self-attention mechanism of prefix-tuned models by replacing the standard softmax with sparse alternatives, without adding parameters.
• Use spectral analysis to validate that sparse prefix parameters better identify salient features compared to dense counterparts.
• Train models with structured prefix parameters and sparse attention, maintaining frozen main model weights while tuning only the soft prompts.
• Evaluate performance via automatic metrics (ROUGE) and human evaluation on coherence, faithfulness, relevance, grammaticality, and overall quality.

Experimental results

Research questions

• RQ1 Do different layers of the transformer network process prefix parameters with varying efficiency?
• RQ2 Can hierarchical blocking of prefix parameters simulate human-like discourse structure and improve generation quality?
• RQ3 Does applying attention sparsity to prefix parameters enhance feature representation and model performance?
• RQ4 How does discourse-aware soft prompting generalize across diverse text generation tasks, especially in low-resource settings?
• RQ5 Can structured prefix tuning achieve performance close to full fine-tuning while preserving parameter efficiency?

Key findings

• Hierarchical blocking (HierBlock) and sparse attention (HierBlock+SA) significantly outperform standard prefix-tuning on all automatic metrics, with ROUGE-1, ROUGE-2, and ROUGE-L scores consistently higher.
• On the XSUM and Wikihow summarization tasks, HierBlock and HierBlock+SA outperform full fine-tuning on ROUGE-1 and ROUGE-2 when training data is limited to less than 50%.
• Human evaluation shows HierBlock and HierBlock+SA are preferred over prefix-tuning on factuality and coherence, with p < .05 significance in most criteria.
• In low-resource settings, HierBlock+SA demonstrates more stable performance than both prefix-tuning and full fine-tuning across multiple metrics.
• The proposed models achieve performance comparable to full fine-tuning on data-to-text generation tasks and slightly exceed it on summarization in human evaluation.
• Spectral analysis confirms that sparse prefix parameters better identify important features than dense ones, supporting the design choice of sparsity.