[Paper Review] Transformer based Multilingual document Embedding model

Wei Li, Brian Mak|arXiv (Cornell University)|Aug 19, 2020

Topic Modeling22 references4 citations

TL;DR

This paper proposes cT-LASER, a transformer-based multilingual document embedding model that replaces LASER's BiLSTM encoder with a self-attention transformer architecture and introduces a novel distance constraint loss to improve cross-lingual alignment. The model achieves faster training and significantly outperforms both LASER and standard T-LASER, especially in cross-lingual transfer tasks, with a 7.2% performance gain over LASER on cross-lingual classification benchmarks.

ABSTRACT

One of the current state-of-the-art multilingual document embedding model LASER is based on the bidirectional LSTM neural machine translation model. This paper presents a transformer-based sentence/document embedding model, T-LASER, which makes three significant improvements. Firstly, the BiLSTM layers is replaced by the attention-based transformer layers, which is more capable of learning sequential patterns in longer texts. Secondly, due to the absence of recurrence, T-LASER enables faster parallel computations in the encoder to generate the text embedding. Thirdly, we augment the NMT translation loss function with an additional novel distance constraint loss. This distance constraint loss would further bring the embeddings of parallel sentences close together in the vector space; we call the T-LASER model trained with distance constraint, cT-LASER. Our cT-LASER model significantly outperforms both BiLSTM-based LASER and the simpler transformer-based T-LASER.

Motivation & Objective

• To develop a faster, more effective multilingual document embedding model by replacing LASER's BiLSTM encoder with a transformer architecture.
• To improve cross-lingual alignment in multilingual embeddings by introducing a novel distance constraint loss function.
• To evaluate the performance of the new model, cT-LASER, on multilingual and bilingual cross-lingual transfer tasks under controlled training conditions.
• To demonstrate that the distance constraint loss enhances cross-lingual performance without degrading same-language performance.

Proposed method

• Replace the BiLSTM encoder in LASER with a transformer encoder to enable better parallelization and long-range sequence modeling.
• Train the model using a multilingual neural machine translation (NMT) framework with shared encoder and uniform dictionary across languages.
• Introduce a distance constraint loss that minimizes the L2 distance between sentence embeddings of parallel sentences in the shared vector space.
• Combine the standard NMT translation loss with the distance constraint loss during end-to-end training to jointly optimize for translation and embedding alignment.
• Use the Europarl parallel corpus for pre-training and evaluate on the MLdoc dataset for cross-lingual document classification.
• Control training hyperparameters, batch size, optimizer, and random seeds to ensure fair comparison between LASER, T-LASER, and cT-LASER.

Experimental results

Research questions

• RQ1 Can a transformer-based encoder improve the performance and training speed of multilingual document embedding models compared to BiLSTM?
• RQ2 Does adding a distance constraint loss to the NMT objective enhance cross-lingual alignment of sentence embeddings?
• RQ3 How does the model perform in low-resource or bilingual settings where pivot languages are absent?
• RQ4 Does the distance constraint loss improve cross-lingual performance without degrading same-language performance?

Key findings

• cT-LASER achieves a 7.2% improvement in cross-lingual classification accuracy over LASER on the MLdoc benchmark, significantly outperforming both LASER and standard T-LASER.
• cT-LASER shows a 5.1% performance gap over LASER_6 in cross-lingual tasks, demonstrating the effectiveness of the distance constraint loss.
• The model achieves faster training speed (WPS) than LASER_1 and LASER_6 due to better GPU parallelism in the transformer architecture.
• In a bilingual setting without pivot languages, cT-LASER improves cross-lingual performance by over 16%, showing that the distance constraint loss enables knowledge transfer even in the absence of pivot languages.
• The distance constraint loss improves cross-lingual performance by 2.5% in cLASER_6 compared to LASER_6, while having minimal impact on same-language performance.
• The performance gains from the distance constraint are stable across random seeds, indicating robustness to training randomness.