[Paper Review] Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning

Yacine Jernite, Samuel R. Bowman+1 more|arXiv (Cornell University)|Apr 23, 2017

Topic Modeling27 references111 citations

TL;DR

The paper introduces three fast, purely discriminative discourse-based objectives to pretrain sentence encoders from unlabeled text, achieving competitive extrinsic performance with substantially reduced training time.

ABSTRACT

This work presents a novel objective function for the unsupervised training of neural network sentence encoders. It exploits signals from paragraph-level discourse coherence to train these models to understand text. Our objective is purely discriminative, allowing us to train models many times faster than was possible under prior methods, and it yields models which perform well in extrinsic evaluations.

Motivation & Objective

• Motivate and enable fast unsupervised pretraining of sentence encoders using discourse coherence signals.
• Propose three discriminative objectives that leverage paragraph-level discourse relations.
• Show that joint multitask training improves representation quality.
• Evaluate learned representations on standard text classification benchmarks to demonstrate competitive performance and speed.

Proposed method

• Define three fast coherence-based pretraining tasks: binary ordering of adjacent sentences, next-sentence selection from a paragraph, and conjunction-prediction based on initial conjunctions.
• Train shared sentence encoders (CBOW, GRU, BiGRU with FastText embeddings) jointly with three bilinear classifiers.
• Use AdaGrad with subsampling to optimize across tasks in a 8-hour window for BiGRU.
• Compare single-task vs. multitask training and assess intrinsic and extrinsic performance.
• Provide qualitative nearest-neighbor analysis in the learned representation space.

Experimental results

Research questions

• RQ1 Can fast discriminative objectives capture discourse coherence signals for unsupervised sentence representation learning?
• RQ2 Do multitask training with three discourse-based objectives improve encoder quality over single-task setups?
• RQ3 How do different encoder architectures (CBOW, GRU, BiGRU) compare under these objectives in terms of speed and performance?
• RQ4 What extrinsic performance do the learned representations achieve on standard sentence classification tasks?
• RQ5 How do DiscSent representations compare to existing unsupervised pretraining methods (e.g., Skip Thought, SDAE) in speed and accuracy?

Key findings

• Joint training with the three discourse-based objectives yields better intrinsic performance than single-task training.
• BiGRU models outperform CBOW and GRU on intrinsic tasks, with DiscSent achieving competitive results on extrinsic tasks.
• DiscSent-based representations match or exceed several baselines on MSRP, TREC, and SUBJ while demanding far less training time.
• DiscSent variants with unigram or embedding augmentation further improve downstream task performance.
• The approach trains in hours rather than days or weeks, enabling scalable unsupervised pretraining.