[Paper Review] Mining Discourse Markers for Unsupervised Sentence Representation Learning

Damien Sileo, Tim Van de Cruys+2 more|arXiv (Cornell University)|Mar 28, 2019

Topic Modeling47 references30 citations

TL;DR

This paper proposes an unsupervised method to automatically discover discourse markers from massive web-scale text, creating a large-scale dataset of 174 discourse markers with at least 10K examples each. The method uses pattern-based heuristics and distributional clustering to extract sentence pairs with explicit discourse markers, which are then used to train sentence embeddings that achieve state-of-the-art performance on transfer tasks, though models appear to rely more on lexical cues than semantic relations between sentences.

ABSTRACT

Current state of the art systems in NLP heavily rely on manually annotated datasets, which are expensive to construct. Very little work adequately exploits unannotated data -- such as discourse markers between sentences -- mainly because of data sparseness and ineffective extraction methods. In the present work, we propose a method to automatically discover sentence pairs with relevant discourse markers, and apply it to massive amounts of data. Our resulting dataset contains 174 discourse markers with at least 10k examples each, even for rare markers such as coincidentally or amazingly We use the resulting data as supervision for learning transferable sentence embeddings. In addition, we show that even though sentence representation learning through prediction of discourse markers yields state of the art results across different transfer tasks, it is not clear that our models made use of the semantic relation between sentences, thus leaving room for further improvements. Our datasets are publicly available (https://github.com/synapse-developpement/Discovery)

Motivation & Objective

• To address the scarcity and bias in manually annotated NLP datasets by leveraging unannotated discourse markers as a supervision signal.
• To automatically discover a broad and diverse set of discourse markers from large-scale text, overcoming limitations of prior work that relied on small, predefined marker lists.
• To evaluate whether discourse marker prediction truly captures semantic relations between sentences, rather than relying on superficial lexical cues.
• To construct a publicly available, large-scale dataset of discourse marker-annotated sentence pairs for training and evaluating sentence representations.
• To investigate whether models trained on discourse markers make use of the underlying semantic relations or simply exploit lexical patterns.

Proposed method

• The method uses a combination of syntactic and distributional patterns to identify sentence pairs containing discourse markers, leveraging dependency parsing and co-occurrence statistics.
• It applies a clustering-based approach to group semantically similar discourse markers, enabling the discovery of rare and multi-word markers.
• A pattern-matching pipeline extracts candidate sentence pairs based on syntactic templates and lexical cues around discourse markers.
• The resulting dataset contains 174 discourse markers with at least 10,000 examples each, even for rare markers like 'coincidentally' or 'amazingly'.
• Sentence embeddings are trained using a contrastive learning objective where the model predicts the correct discourse marker from a set of candidates for each sentence pair.
• The approach includes a shuffling perturbation technique to probe whether models rely on sentence relationships or merely on lexical features.

Experimental results

Research questions

• RQ1 Can discourse markers be automatically discovered at scale from unannotated text, even for rare or multi-word markers?
• RQ2 Does training sentence representations via discourse marker prediction lead to better transfer performance compared to existing unsupervised methods?
• RQ3 To what extent do models trained on discourse markers rely on semantic relations between sentences, or do they exploit simple lexical patterns instead?
• RQ4 Is the performance gain from discourse marker prediction due to high-level reasoning or superficial cues?
• RQ5 How does the inclusion of 'simple' examples affect the generalization of sentence representations?

Key findings

• The method successfully discovered 174 discourse markers, with at least 10,000 examples per marker, including rare ones like 'coincidentally' and 'amazingly', significantly expanding on prior datasets.
• The resulting sentence embeddings achieved state-of-the-art performance on 12 out of 14 transfer learning tasks in the SentEval benchmark, outperforming models like InferSent and QuickThought.
• Models trained on the dataset achieved 52.5% accuracy on the coarse-grained PDTB implicit discourse relation task, surpassing previous methods.
• Shuffling the sentence order during training had minimal impact on performance, indicating that models rely more on lexical features than on semantic relations between sentences.
• Removing 'simple' examples (e.g., those with obvious lexical cues) significantly hurt transfer performance, suggesting that such examples are crucial for generalization.
• Despite SOTA results, the analysis reveals that models do not meaningfully exploit the semantic relations between sentences, leaving room for improvement in relation-aware representation learning.