[Paper Review] Two-Level Transformer and Auxiliary Coherence Modeling for Improved Text Segmentation

Goran Glavašš, Swapna Somasundaran|arXiv (Cornell University)|Jan 3, 2020

Topic Modeling42 references7 citations

TL;DR

This paper proposes CATS, a two-level Transformer-based model that improves text segmentation by explicitly modeling coherence through a multi-task learning setup. By jointly optimizing for sentence-level segmentation and distinguishing original from shuffled sentence sequences, CATS achieves state-of-the-art performance on benchmark datasets and enables effective zero-shot cross-lingual transfer using cross-lingual embeddings.

ABSTRACT

Breaking down the structure of long texts into semantically coherent segments makes the texts more readable and supports downstream applications like summarization and retrieval. Starting from an apparent link between text coherence and segmentation, we introduce a novel supervised model for text segmentation with simple but explicit coherence modeling. Our model -- a neural architecture consisting of two hierarchically connected Transformer networks -- is a multi-task learning model that couples the sentence-level segmentation objective with the coherence objective that differentiates correct sequences of sentences from corrupt ones. The proposed model, dubbed Coherence-Aware Text Segmentation (CATS), yields state-of-the-art segmentation performance on a collection of benchmark datasets. Furthermore, by coupling CATS with cross-lingual word embeddings, we demonstrate its effectiveness in zero-shot language transfer: it can successfully segment texts in languages unseen in training.

Motivation & Objective

• To address the limitation of existing text segmentation models that implicitly capture coherence through lexical or semantic similarity, rather than explicitly modeling it.
• To improve segmentation performance by introducing an auxiliary coherence objective that distinguishes original sentence sequences from corrupted (shuffled) ones.
• To enable zero-shot language transfer for text segmentation by leveraging cross-lingual word embeddings with a model trained solely on English data.
• To demonstrate that explicit coherence modeling enhances both segmentation accuracy and robustness across domains and languages.

Proposed method

• The model uses a two-level Transformer architecture: a token-level Transformer encodes individual sentences, and a sentence-level Transformer encodes the sequence of sentence representations.
• The model performs binary segmentation prediction at the sentence level using a feed-forward classifier on contextualized sentence representations.
• An auxiliary coherence regressor predicts a coherence score for the entire snippet, trained to assign higher scores to original sentence sequences than to randomly shuffled ones.
• The model is trained via multi-task learning, optimizing both the primary segmentation objective and the auxiliary coherence objective simultaneously.
• The architecture uses pretrained word embeddings concatenated with positional embeddings, and the sentence representation is derived from the [CLS]-like token representation in the token-level Transformer.
• For zero-shot transfer, the model is coupled with cross-lingual word embeddings, enabling segmentation in unseen languages without fine-tuning.

Experimental results

Research questions

• RQ1 Can explicit coherence modeling improve text segmentation performance beyond implicit coherence signals?
• RQ2 Does a multi-task learning setup combining segmentation and coherence prediction lead to better generalization and robustness?
• RQ3 Can a model trained exclusively on English data segment texts in low-resource or unseen languages via cross-lingual embeddings?
• RQ4 How does the model’s performance compare to state-of-the-art supervised and unsupervised segmentation methods across multiple benchmarks?

Key findings

• CATS achieves state-of-the-art performance on multiple benchmark text segmentation datasets, outperforming prior supervised models.
• The auxiliary coherence objective significantly improves segmentation performance beyond the base Two-Level Transformer (TLT-TS) model, even when the coherence objective is not used during inference.
• The TLT-TS model alone, without the coherence head, already achieves state-of-the-art results, demonstrating the effectiveness of the two-level Transformer architecture.
• CATS enables effective zero-shot cross-lingual segmentation: when combined with cross-lingual word embeddings, it outperforms the best unsupervised model on unseen languages.
• The model shows strong robustness in domain transfer, maintaining high performance across diverse text types and domains.