[Paper Review] Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherence

Jian Guan, Xiaoxi Mao+4 more|arXiv (Cornell University)|May 19, 2021

Topic Modeling32 references4 citations

TL;DR

This paper proposes Hint, a long text generation model that improves coherence by learning sentence-level and discourse-level representations during decoding. By introducing specialized tokens and pretraining objectives—inter-sentence similarity prediction and sentence order discrimination—Hint achieves superior coherence over strong baselines like BART, especially in maintaining logical event sequences and reducing incoherent repetitions.

ABSTRACT

Generating long and coherent text is an important but challenging task, particularly for open-ended language generation tasks such as story generation. Despite the success in modeling intra-sentence coherence, existing generation models (e.g., BART) still struggle to maintain a coherent event sequence throughout the generated text. We conjecture that this is because of the difficulty for the decoder to capture the high-level semantics and discourse structures in the context beyond token-level co-occurrence. In this paper, we propose a long text generation model, which can represent the prefix sentences at sentence level and discourse level in the decoding process. To this end, we propose two pretraining objectives to learn the representations by predicting inter-sentence semantic similarity and distinguishing between normal and shuffled sentence orders. Extensive experiments show that our model can generate more coherent texts than state-of-the-art baselines.

Motivation & Objective

• To address the challenge of maintaining long-range coherence in open-ended text generation, especially in story and fiction generation.
• To improve upon existing models that focus on token-level co-occurrence but fail to capture high-level semantics and discourse structure.
• To develop a generation model capable of understanding and representing context at both sentence and discourse levels during decoding.
• To design pretraining objectives that explicitly teach the model to predict semantic similarity between sentences and distinguish correct from shuffled sentence orders.
• To evaluate whether high-level representations lead to more coherent and logically structured long-form text.

Proposed method

• Introduce special [S] and [D] tokens after each sentence to enable sentence-level and discourse-level representation learning in the decoder.
• Pretrain the model with three objectives: next-token prediction, inter-sentence semantic similarity prediction using SentenceBERT as a teacher, and sentence order discrimination between normal and shuffled sequences.
• Use SentenceBERT to provide gold-standard sentence embeddings for similarity prediction, enabling the model to learn meaningful sentence-level representations.
• Train the discourse-level representation to distinguish between correct and permuted sentence orders, encouraging the model to learn temporal and logical dependencies.
• Fine-tune the model on story and fiction generation tasks using the learned high-level representations during decoding.
• Evaluate both automatic and human annotations to assess coherence, fluency, and language modeling performance on coherent vs. incoherent inputs.

Experimental results

Research questions

• RQ1 Can modeling sentence-level and discourse-level representations improve long text coherence in open-ended generation?
• RQ2 Does pretraining on inter-sentence similarity and sentence order discrimination enhance the model’s ability to maintain logical event sequences?
• RQ3 How does the inclusion of high-level representations affect language modeling on coherent vs. incoherent inputs?
• RQ4 To what extent do sentence-level and discourse-level representations contribute to different types of coherence (e.g., causal, temporal, negation)?
• RQ5 Can the proposed method generalize across different generation tasks and model architectures?

Key findings

• Hint significantly outperforms BART and other strong baselines in human evaluation on story and fiction generation, with a statistically significant improvement in coherence (p < 0.01).
• The model achieves lower perplexity on coherent examples and higher perplexity on incoherent examples, indicating better modeling of high-level coherence.
• Ablation studies show that sentence-level representations improve performance on relatedness, negation, and semantic repetition, while discourse-level representations are most effective for causal and temporal relationships.
• The model maintains comparable fluency to baselines while achieving superior coherence, indicating that the improvements are not due to better surface-level fluency.
• The use of SentenceBERT as a teacher for similarity prediction helps the model learn more meaningful sentence-level representations.
• Error analysis reveals that Hint reduces repetitive plots and logical inconsistencies common in BART-generated stories, especially in event sequencing and causal logic.