[Paper Review] TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning

Kexin Wang, Nils Reimers+1 more|arXiv (Cornell University)|Apr 14, 2021

Topic ModelingComputer Science35 references44 citations

TL;DR

TSDAE is a Transformer-based unsupervised method that learns sentence embeddings via sequential denoising auto-encoding, outperforming prior unsupervised approaches and approaching supervised models on diverse domain tasks.

ABSTRACT

Learning sentence embeddings often requires a large amount of labeled data. However, for most tasks and domains, labeled data is seldom available and creating it is expensive. In this work, we present a new state-of-the-art unsupervised method based on pre-trained Transformers and Sequential Denoising Auto-Encoder (TSDAE) which outperforms previous approaches by up to 6.4 points. It can achieve up to 93.1% of the performance of in-domain supervised approaches. Further, we show that TSDAE is a strong domain adaptation and pre-training method for sentence embeddings, significantly outperforming other approaches like Masked Language Model. A crucial shortcoming of previous studies is the narrow evaluation: Most work mainly evaluates on the single task of Semantic Textual Similarity (STS), which does not require any domain knowledge. It is unclear if these proposed methods generalize to other domains and tasks. We fill this gap and evaluate TSDAE and other recent approaches on four different datasets from heterogeneous domains.

Motivation & Objective

• Motivate and enable high-quality sentence embeddings in domains with little or no labeled data.
• Introduce TSDAE, a Transformer-based sequential denoising auto-encoder architecture.
• Show that TSDAE outperforms prior unsupervised methods and approaches supervised models on diverse tasks.
• Demonstrate TSDAE’s effectiveness as a domain adaptation and pre-training objective.

Proposed method

• Encoder-decoder Transformer where cross-attention uses only the sentence embedding as key and value.
• Corrupt the input sentence with word deletion (deletion ratio 0.6) to create damaged input.
• Train to reconstruct the original sentence embedding from the damaged input (SDAE objective).
• Use the fixed-size embedding from the [CLS] token as the sentence representation.
• Tying encoder and decoder parameters during training.
• Evaluate on domain-diverse tasks (IR, RR, PI) and compare against CT, SimCSE, BERT-flow, MLM, and SBERT/USE baselines.

Experimental results

Research questions

• RQ1 Can TSDAE produce high-quality unsupervised sentence embeddings across heterogeneous domains?
• RQ2 How does TSDAE compare to other unsupervised sentence-embedding methods on domain-specific tasks?
• RQ3 Is TSDAE effective as a pre-training and domain-adaptation objective beyond plain unsupervised learning?
• RQ4 How does evaluation beyond STS correlate with real downstream task performance for unsupervised methods?

Key findings

• TSDAE outperforms previous unsupervised methods by up to 6.4 points on diverse domain tasks.
• TSDAE can reach performance on par with or above certain supervised pre-trained models (e.g., USE-large) on the evaluated tasks.
• Domain adaptation: unsupervised TSDAE followed by supervised NLI+STS training yields improved results over the reverse direction, with average gains observed across tasks.
• Pre-training: TSDAE outperforms MLM and other baselines as a pre-training objective, providing notable gains in downstream supervised performance.
• STS-only evaluation correlates poorly with performance on domain-specific tasks; TSDAE shows stronger gains in real-world tasks compared to STS-centric methods.