[Paper Review] Evaluation of Unsupervised Compositional Representations

Hanan Aldarmaki, Mona Diab|arXiv (Cornell University)|Jun 12, 2018

Topic Modeling25 references3 citations

TL;DR

This paper evaluates unsupervised compositional sentence representations across supervised and unsupervised benchmarks, comparing models from binary bag-of-words to RNN-based architectures. Weighted vector averaging outperforms more complex models in most unsupervised tasks, while context-sensitive models like inferSent and skip-th show advantages in supervised settings due to structural and syntactic encoding.

ABSTRACT

We evaluated various compositional models, from bag-of-words representations to compositional RNN-based models, on several extrinsic supervised and unsupervised evaluation benchmarks. Our results confirm that weighted vector averaging can outperform context-sensitive models in most benchmarks, but structural features encoded in RNN models can also be useful in certain classification tasks. We analyzed some of the evaluation datasets to identify the aspects of meaning they measure and the characteristics of the various models that explain their performance variance.

Motivation & Objective

• To evaluate the performance of unsupervised compositional models across both supervised and unsupervised benchmarks.
• To identify which model components—such as word weighting, structural encoding, or training objectives—drive performance variance.
• To compare simple baselines like binary bag-of-words and weighted averaging against more complex RNN-based models.
• To analyze intrinsic vector space structures to understand what semantic and structural features models encode.

Proposed method

• Evaluated a range of models: binary bag-of-words, distributed word embeddings with tf-idf and SIF weighting, and context-sensitive models including doc2vec, GRAN, skip-thought, and inferSent.
• Applied SIF (Simple Input Feature) weighting to reduce the influence of low-information words in averaged embeddings.
• Used t-SNE visualization and k-means clustering to analyze vector space structure and cluster cohesion by semantic type.
• Conducted evaluations on TREC for topic categorization and STS for semantic similarity, using both supervised and unsupervised settings.
• Compared performance using correlation scores for STS and accuracy for topic classification.
• Analyzed nearest neighbors and cluster purity to assess semantic coherence and model-specific biases.

Experimental results

Research questions

• RQ1 Which compositional model architecture achieves the best performance across both supervised and unsupervised evaluation benchmarks?
• RQ2 How do different weighting schemes (e.g., tf-idf, SIF) affect the quality of sentence representations in unsupervised similarity tasks?
• RQ3 To what extent do structural features encoded in RNNs improve performance on classification tasks compared to simple averaging?
• RQ4 What intrinsic properties of vector spaces—such as cluster cohesion by semantic type—explain performance differences across models?
• RQ5 How do models like skip-thought and inferSent differ in their encoding of semantic versus syntactic structure?

Key findings

• Weighted vector averaging with SIF or tf-idf weighting outperformed all other models in unsupervised semantic textual similarity (STS) tasks.
• The binary bag-of-words baseline with NBSVM achieved significantly higher accuracy than all compositional models in supervised topic categorization tasks.
• Skip-thought vectors showed high clustering purity and strong performance in supervised STS, but performed poorly in unsupervised settings, indicating strong encoding of structural features.
• inferSent vectors achieved performance on par with weighted averaging in unsupervised STS and consistently high accuracy in supervised tasks, suggesting a balanced encoding of semantic and syntactic features.
• doc2vec and GRAN representations were qualitatively similar to SIF-weighted averages, clustering sentences cohesively by topic and semantic similarity.
• t-SNE visualizations revealed that skip-th vectors formed larger, more consistent clusters based on sentence structure (e.g., 'What country...'), while binary vectors clustered by word overlap rather than semantic type.