[Paper Review] Rethinking Skip-thought: A Neighborhood based Approach

Shuai Tang, Hailin Jin+3 more|arXiv (Cornell University)|Jun 9, 2017

Topic Modeling20 references3 citations

TL;DR

This paper proposes a neighborhood-based variant of the skip-thought model that reconstructs both neighboring sentences using a single shared decoder, achieving performance comparable to the original skip-thought model on seven NLP tasks. Surprisingly, adding an autoencoder path to reconstruct the input sentence harms performance, while a unidirectional model decoding only the next sentence yields the best results.

ABSTRACT

We study the skip-thought model with neighborhood information as weak supervision. More specifically, we propose a skip-thought neighbor model to consider the adjacent sentences as a neighborhood. We train our skip-thought neighbor model on a large corpus with continuous sentences, and then evaluate the trained model on 7 tasks, which include semantic relatedness, paraphrase detection, and classification benchmarks. Both quantitative comparison and qualitative investigation are conducted. We empirically show that, our skip-thought neighbor model performs as well as the skip-thought model on evaluation tasks. In addition, we found that, incorporating an autoencoder path in our model didn't aid our model to perform better, while it hurts the performance of the skip-thought model.

Motivation & Objective

• To investigate whether adjacent sentences as a neighborhood provide sufficient supervision for learning sentence representations.
• To evaluate if a single decoder for both previous and next sentences can match the performance of the original skip-thought model’s dual decoders.
• To examine the impact of adding an autoencoder path that reconstructs the input sentence on model performance.
• To explore whether a unidirectional model decoding only the next sentence outperforms bidirectional reconstruction.

Proposed method

• The skip-thought neighbor model uses a single shared decoder to reconstruct both the previous and next sentences simultaneously, using the encoded representation of the current sentence as context.
• The model is trained on a large corpus of continuous text in an unsupervised manner, minimizing reconstruction loss for both neighboring sentences.
• The method compares variants: original skip-thought (dual decoders), skip-thought neighbor (single decoder for neighbors), and models with added autoencoder branches.
• Performance is evaluated on seven downstream tasks, including semantic relatedness, paraphrase detection, and text classification.
• Sentence retrieval and conditional generation are used for qualitative analysis, with cosine similarity used to measure semantic closeness in the representation space.

Experimental results

Research questions

• RQ1 Does using both adjacent sentences as a shared neighborhood provide sufficient supervision for learning effective sentence representations?
• RQ2 Can a single shared decoder for neighboring sentences match the performance of the original skip-thought model with two independent decoders?
• RQ3 Does incorporating an autoencoder path that reconstructs the input sentence improve or degrade model performance?
• RQ4 Is a unidirectional model that only decodes the next sentence superior to bidirectional reconstruction?

Key findings

• The skip-thought neighbor model, which uses a single decoder to reconstruct both neighboring sentences, performs as well as the original skip-thought model on all seven evaluation tasks.
• The model with an added autoencoder path to reconstruct the input sentence does not improve performance and actually harms the performance of the skip-thought model.
• The unidirectional model that decodes only the next sentence achieves the best performance among all variants tested.
• Qualitative analysis shows that the decoder generates grammatically correct and semantically meaningful sentences, though they often start with 'I was', 'I'm not', or 'I don't' due to dataset bias.
• Sentence retrieval using the learned representations shows that nearest neighbors are semantically related, indicating effective semantic clustering in the embedding space.