[Paper Review] A Character-Level Length-Control Algorithm for Non-Autoregressive Sentence Summarization

Puyuan Liu, Xiang Zhang+1 more|arXiv (Cornell University)|May 28, 2022

Topic Modeling4 citations

TL;DR

This paper proposes NACC, a non-autoregressive summarization model with explicit character-level length control using a dynamic programming algorithm based on Connectionist Temporal Classification (CTC). It achieves state-of-the-art ROUGE scores on Gigaword and DUC2004 while being significantly faster than autoregressive models and outperforming them in the unsupervised setting, enabling complete, length-constrained summaries without truncation.

ABSTRACT

Sentence summarization aims at compressing a long sentence into a short one that keeps the main gist, and has extensive real-world applications such as headline generation. In previous work, researchers have developed various approaches to improve the ROUGE score, which is the main evaluation metric for summarization, whereas controlling the summary length has not drawn much attention. In our work, we address a new problem of explicit character-level length control for summarization, and propose a dynamic programming algorithm based on the Connectionist Temporal Classification (CTC) model. Results show that our approach not only achieves higher ROUGE scores but also yields more complete sentences.

Motivation & Objective

• To address the lack of explicit character-level length control in non-autoregressive sentence summarization, which is more realistic for real-world applications like mobile headlines and LED displays.
• To design a method that ensures generated summaries are complete and fit within a character budget, avoiding truncation or incomplete outputs.
• To bridge the gap between summarization research and evaluation, where character-level constraints (e.g., 75 bytes in DUC2004) are common but under-addressed.
• To enable length-transfer generation, allowing the same model to produce summaries of varying lengths from the same input.
• To improve ROUGE scores under strict character-level constraints while maintaining high inference efficiency.

Proposed method

• Formulate character-level length control as a knapsack-like problem, where token weights are character counts and values are predicted probabilities from a non-autoregressive model.
• Use dynamic programming to select a sequence of tokens that maximizes total probability while staying within a predefined character budget.
• Integrate the CTC loss to reduce token repetition in non-autoregressive generation, improving fluency and coherence.
• Apply the length-control algorithm post-prediction, enabling explicit control without retraining or modifying the model's attention mechanism.
• Use unsupervised word-extraction methods to generate pseudo-references for training in the unsupervised setting.
• Leverage the parallel generation capability of non-autoregressive models to maintain high inference speed, even with the dynamic programming step.

Experimental results

Research questions

• RQ1 Can explicit character-level length control improve summarization quality and completeness compared to word-level control or truncation?
• RQ2 Does a dynamic programming approach based on CTC and token probabilities enable precise, efficient length control in non-autoregressive summarization?
• RQ3 Can NACC outperform autoregressive models in the unsupervised setting, where training data is limited and alignment between input and output is strong?
• RQ4 To what extent can NACC perform length-transfer generation, producing summaries of varying lengths from the same input?
• RQ5 How does character-level length control affect ROUGE scores and summary completeness in both supervised and unsupervised settings?

Key findings

• NACC achieves state-of-the-art performance among non-autoregressive models on both Gigaword and DUC2004 datasets under various character-level length constraints.
• In the unsupervised setting, NACC outperforms an autoregressive Transformer, demonstrating superior generalization and alignment with input content.
• NACC generates summaries that are more complete than truncated outputs, as shown in qualitative examples where it preserves key information without exceeding the character budget.
• The model maintains high inference efficiency—several times faster than autoregressive models—even with the dynamic programming length control step.
• Length-transfer generation is successfully achieved: NACC generates high-quality summaries across different length budgets, with ROUGE scores decreasing linearly as the budget shrinks, unlike competing methods that degrade faster.
• The dynamic programming algorithm effectively balances length constraints and summary quality, as evidenced by higher ROUGE scores and more coherent, complete outputs compared to baseline methods.