[Paper Review] Recurrent Chunking Mechanisms for Long-Text Machine Reading Comprehension

Hongyu Gong, Yelong Shen+3 more|arXiv (Cornell University)|May 16, 2020

Topic Modeling29 references4 citations

TL;DR

This paper proposes Recurrent Chunking Mechanisms (RCM) for long-text machine reading comprehension, enabling models to learn dynamic, reinforcement-learning-driven chunking of lengthy documents and use recurrent networks to propagate contextual information across segments. The method improves answer span coverage and performance on CoQA, QuAC, and TriviaQA by reducing incomplete predictions and enhancing cross-segment reasoning.

ABSTRACT

In this paper, we study machine reading comprehension (MRC) on long texts, where a model takes as inputs a lengthy document and a question and then extracts a text span from the document as an answer. State-of-the-art models tend to use a pretrained transformer model (e.g., BERT) to encode the joint contextual information of document and question. However, these transformer-based models can only take a fixed-length (e.g., 512) text as its input. To deal with even longer text inputs, previous approaches usually chunk them into equally-spaced segments and predict answers based on each segment independently without considering the information from other segments. As a result, they may form segments that fail to cover the correct answer span or retain insufficient contexts around it, which significantly degrades the performance. Moreover, they are less capable of answering questions that need cross-segment information. We propose to let a model learn to chunk in a more flexible way via reinforcement learning: a model can decide the next segment that it wants to process in either direction. We also employ recurrent mechanisms to enable information to flow across segments. Experiments on three MRC datasets -- CoQA, QuAC, and TriviaQA -- demonstrate the effectiveness of our proposed recurrent chunking mechanisms: we can obtain segments that are more likely to contain complete answers and at the same time provide sufficient contexts around the ground truth answers for better predictions.

Motivation & Objective

• Address the limitations of fixed-length input constraints in transformer-based MRC models when processing long documents.
• Overcome the poor generalization of equal-sized, non-overlapping chunking that risks missing answer spans or losing contextual coherence.
• Improve answer prediction by learning to select optimal segment boundaries via reinforcement learning, reducing reliance on rigid stride sizes.
• Enable cross-segment information flow through recurrent mechanisms to support reasoning across distant spans in long documents.
• Demonstrate consistent performance gains on diverse long-context MRC benchmarks, including conversational and open-domain datasets.

Proposed method

• Introduce a policy network trained via reinforcement learning to decide the next segment to process, allowing flexible, adaptive chunking in either forward or backward directions.
• Use a recurrent mechanism (LSTM or gated recurrence) to maintain and update hidden states across segments, enabling information flow beyond individual segments.
• Apply BERT or BERT-Large to encode each segment’s input, generating contextual representations for answer extraction.
• Integrate a chunking scorer that evaluates each segment’s likelihood of containing the answer, using scores to ensemble predictions across multiple segments.
• Train the model end-to-end with a policy gradient objective to optimize chunking decisions based on answer prediction performance.
• Allow bidirectional movement (forward and backward) during chunking to better center answer spans within segments and preserve surrounding context.

Experimental results

Research questions

• RQ1 Can a reinforcement learning-based chunking policy improve answer span coverage in long-text MRC compared to fixed-stride chunking?
• RQ2 Does incorporating recurrent mechanisms across segments enhance answer prediction performance by enabling cross-segment context flow?
• RQ3 How does the position of the answer span relative to the segment center affect model performance, and can dynamic chunking mitigate this issue?
• RQ4 To what extent does the proposed RCM method improve performance on diverse long-context MRC datasets like CoQA, QuAC, and TriviaQA?
• RQ5 Can the model learn to avoid suboptimal chunking patterns (e.g., incomplete answers, poor context retention) through end-to-end training with policy gradients?

Key findings

• BERT-RCM with LSTM recurrence achieves a segment-hit rate of 79.7% on CoQA, significantly outperforming BERT-Large’s 54.0%, indicating better selection of informative segments.
• On CoQA, the average answer-chunk center distance is reduced to 32.5 tokens for the first three segments in BERT-RCM with LSTM, compared to 96 tokens in BERT-Large, showing improved alignment with answer spans.
• The chunking scorer assigns higher scores (0.90) to segments that contain the correct answer, demonstrating effective identification of relevant segments.
• BERT-RCM with gated recurrence achieves a 44.9% segment-hit rate on QuAC, a 10.8% improvement over BERT-Large’s 34.1%, indicating better focus on answer-containing segments.
• The model reduces prediction errors near segment boundaries, where BERT-Large struggles due to insufficient context, by learning to adjust chunk boundaries dynamically.
• Extensive experiments on CoQA, QuAC, and TriviaQA show consistent performance gains over state-of-the-art BERT-based models, validating the effectiveness of RCM across diverse long-context MRC tasks.