[Paper Review] Learning Bilingual Sentence Embeddings via Autoencoding and Computing Similarities with a Multilayer Perceptron

Yunsu Kim, Hendrik Rosendahl+4 more|arXiv (Cornell University)|Jun 5, 2019

Topic Modeling42 references4 citations

TL;DR

This paper proposes a joint training framework that combines autoencoding and neural machine translation to learn bilingual sentence embeddings using only parallel and monolingual data, without pivot languages. A multilayer perceptron (MLP) is trained on these embeddings to compute sentence pair similarity, achieving strong performance on sentence alignment recovery and WMT 2018 parallel corpus filtering with a single model.

ABSTRACT

We propose a novel model architecture and training algorithm to learn bilingual sentence embeddings from a combination of parallel and monolingual data. Our method connects autoencoding and neural machine translation to force the source and target sentence embeddings to share the same space without the help of a pivot language or an additional transformation. We train a multilayer perceptron on top of the sentence embeddings to extract good bilingual sentence pairs from nonparallel or noisy parallel data. Our approach shows promising performance on sentence alignment recovery and the WMT 2018 parallel corpus filtering tasks with only a single model.

Motivation & Objective

• To develop a unified, end-to-end method for learning bilingual sentence embeddings using only parallel and monolingual data for a target language pair.
• To eliminate reliance on pivot languages or N-way parallel data, which are often unavailable or impractical.
• To improve parallel corpus mining and filtering by training a trainable similarity measure (MLP) on sentence embeddings.
• To evaluate and compare different similarity measures (cosine, MLP) for bilingual sentence matching in downstream tasks.
• To investigate the impact of negative example quality on MLP training for similarity learning.

Proposed method

• Train a sequence-to-sequence NMT model with shared decoder objectives for both source and target sentences, enabling joint learning of bilingual sentence embeddings.
• Integrate an autoencoder component to reconstruct both source and target sentences, enforcing semantic consistency in the shared embedding space.
• Use a multilayer perceptron (MLP) as a differentiable similarity function to predict whether a source-target sentence pair is aligned.
• Train the MLP on a combination of positive (parallel) and negative (nonparallel or low-quality) sentence pairs to learn fine-grained similarity distinctions.
• Optimize the entire system end-to-end using a combined objective: NMT loss for translation and reconstruction loss for autoencoding.
• Use scored, ranked sentence pairs from a noisy parallel corpus (e.g., ParaCrawl) to create negative sets of varying quality for MLP training.

Experimental results

Research questions

• RQ1 Can a single model jointly learn bilingual sentence embeddings using only parallel and monolingual data, without requiring a pivot language?
• RQ2 How effective is a multilayer perceptron as a trainable similarity measure for matching source and target sentence embeddings?
• RQ3 What is the optimal level of negativity in negative training examples for the MLP similarity model?
• RQ4 How does the MLP similarity compare to simpler measures like cosine similarity in filtering noisy parallel corpora?
• RQ5 Does integrating monolingual data via autoencoding improve the quality of bilingual sentence embeddings for downstream tasks?

Key findings

• The proposed method achieves competitive performance on the WMT 2018 parallel corpus filtering task, matching or exceeding systems that rely on complex combinations of translation and language models.
• The MLP similarity model outperforms cosine similarity in filtering tasks, particularly in distinguishing mid-quality sentence pairs, and enables batch computation with high efficiency.
• A moderate level of negativity—specifically, the 60%-worst scored examples from a noisy parallel corpus—produces the best MLP performance, balancing discrimination of acceptable and poor pairs.
• Randomly sampled negative examples perform surprisingly well, suggesting that a moderate average negativity level is key, and that explicit sorting may not be necessary.
• The model shows strong performance in sentence alignment recovery, indicating that the shared embedding space effectively captures bilingual semantic correspondence.
• The integration of autoencoding with NMT forces the source and target sentence embeddings to align in a shared space, improving robustness without additional transformation or pivot languages.