[Paper Review] No Training Required: Exploring Random Encoders for Sentence Classification

John Wieting, Douwe Kiela|arXiv (Cornell University)|Jan 29, 2019

Topic ModelingComputer Science75 citations

TL;DR

The paper shows that random encoders built from pre-trained word embeddings can achieve competitive sentence classification performance, often close to or rivaling trained encoders, highlighting strong baselines and proposing robust evaluation practices.

ABSTRACT

We explore various methods for computing sentence representations from pre-trained word embeddings without any training, i.e., using nothing but random parameterizations. Our aim is to put sentence embeddings on more solid footing by 1) looking at how much modern sentence embeddings gain over random methods---as it turns out, surprisingly little; and by 2) providing the field with more appropriate baselines going forward---which are, as it turns out, quite strong. We also make important observations about proper experimental protocol for sentence classification evaluation, together with recommendations for future research.

Motivation & Objective

• Assess how far random encoders can push sentence classification accuracy without training the encoder.
• Benchmark random encoders against supervised and unsupervised trained sentence encoders.
• Provide robust baselines and practical recommendations for evaluating sentence representations.
• Highlight experimental protocol considerations for fair comparisons.

Proposed method

• Three random encoding architectures are evaluated: Bag of Random Embedding Projections (BOREP), Random LSTMs, and Echo State Networks (ESN).
• Sentence representations are formed from pre-trained word embeddings without updating encoder parameters; only the transfer logistic regression classifier is trained.
• Projections and networks are randomly initialized (with standard Glorot-like bounds) and pooled (e.g., mean, max, or sum) to form fixed-length sentence vectors.
• Evaluation uses SentEval on a suite of downstream tasks (ten tasks) and probing tasks to analyze linguistic properties.
• Comparisons are made against InferSent and SkipThought baselines, with dimensionality matched to 4096 (and 300 for BOE).
• Dimensionality is varied to study Cover’s theorem implications on performance.

Experimental results

Research questions

• RQ1 How close can random, non-trained encoders get to the performance of trained sentence encoders on standard transfer tasks?
• RQ2 What baselines best reflect the value of pre-trained word embeddings and random projections for sentence representations?
• RQ3 How does encoder dimensionality affect performance across downstream and probing tasks?
• RQ4 What experimental protocol considerations are essential for fair evaluation of sentence representations?

Key findings

• Random encoders generally outperform the simple BOE baseline, with ESNs achieving the best average results among random methods.
• Compared to InferSent and SkipThought, random encoders achieve competitive performance, with average gains over random methods being modest (often less than 2 points on average across tasks).
• Increasing dimensionality (e.g., to 4096 and beyond) tends to improve performance for random encoders, consistent with Cover’s theorem.
• Probing tasks show random models capture order information and some syntactic/semantic cues, but trained encoders excel on certain nuanced semantic tasks.
• Proper baselines (e.g., BOREP) and consistent evaluation settings are essential for fair comparisons and credible claims about encoder gains.