[Paper Review] Multi-view Sentence Representation Learning

Shuai Tang, Virginia R. de|arXiv (Cornell University)|May 18, 2018

Topic Modeling49 references5 citations

TL;DR

This paper proposes a multi-view sentence representation learning framework that jointly trains an RNN encoder and a linear average-of-word-vectors encoder to maximize agreement on adjacent sentence representations via cosine similarity. The method improves both individual view performance and ensemble representations, outperforming existing unsupervised models on downstream tasks while demonstrating strong transferability.

ABSTRACT

Multi-view learning can provide self-supervision when different views are available of the same data. The distributional hypothesis provides another form of useful self-supervision from adjacent sentences which are plentiful in large unlabelled corpora. Motivated by the asymmetry in the two hemispheres of the human brain as well as the observation that different learning architectures tend to emphasise different aspects of sentence meaning, we create a unified multi-view sentence representation learning framework, in which, one view encodes the input sentence with a Recurrent Neural Network (RNN), and the other view encodes it with a simple linear model, and the training objective is to maximise the agreement specified by the adjacent context information between two views. We show that, after training, the vectors produced from our multi-view training provide improved representations over the single-view training, and the combination of different views gives further representational improvement and demonstrates solid transferability on standard downstream tasks.

Motivation & Objective

• To develop a unified multi-view learning framework that leverages distinct neural architectures to improve sentence representation learning.
• To exploit the distributional hypothesis by maximizing agreement between representations of adjacent sentences in an unlabelled corpus.
• To investigate whether asymmetric processing—inspired by left-hemisphere sequential and right-hemisphere parallel processing—enhances representation quality.
• To evaluate the transferability and generalization of learned representations on both unsupervised and supervised NLP tasks.
• To determine whether ensembling representations from different views yields superior performance compared to single-view training.

Proposed method

• The framework uses two distinct views: one with a bidirectional RNN encoder (f) and another with a linear average-of-word-vectors encoder (g).
• The training objective maximizes cosine similarity between representations of adjacent sentences across the two views, defined as $ a_{ij} = \cos(\mathbf{z}_i^f, \mathbf{z}_j^g) + \cos(\mathbf{z}_i^g, \mathbf{z}_j^f) $.
• The model is trained end-to-end on large unlabelled corpora using contrastive learning, where adjacent sentence pairs are encouraged to have high agreement.
• Post-processed and normalized representations $ \hat{\mathbf{z}} $ are used in the cosine similarity computation to stabilize training.
• The framework enables multi-view learning without requiring multiple data sources, using only different processing paths on the same input.
• The ensemble of representations from both views is used to improve generalization and capture complementary aspects of sentence meaning.

Experimental results

Research questions

• RQ1 Can combining an RNN-based encoder with a linear average-of-word-vectors encoder improve sentence representation quality through multi-view learning?
• RQ2 Does maximizing agreement between two distinct views on adjacent sentences lead to better representations than single-view training?
• RQ3 How does the ensemble of two different views compare to individual views in terms of downstream transfer performance?
• RQ4 Does the proposed framework outperform existing unsupervised sentence representation models on standard evaluation benchmarks?
• RQ5 Is the functional asymmetry in information processing—inspired by brain lateralization—beneficial for learning diverse sentence representations?

Key findings

• The multi-view training framework significantly improves the performance of both the RNN and linear encoders compared to their single-view counterparts.
• The ensemble of representations from both views achieves better results than either view alone, demonstrating complementary learning.
• The model outperforms existing unsupervised transfer learning models on unsupervised evaluation tasks, such as sentence similarity and textual entailment.
• On supervised downstream tasks, the model's performance is comparable to or better than the best unsupervised models, indicating strong transferability.
• The agreement function defined in Equation 2 ($ a_{ij} = \cos(\mathbf{z}_i^f, \mathbf{z}_j^g) + \cos(\mathbf{z}_i^g, \mathbf{z}_j^f) $) outperforms alternatives that encourage self-agreement, as it prevents one view from dominating during training.
• Empirical results support the hypothesis that sequential (RNN) and parallel (linear) processing modes complement each other in learning rich sentence representations.