[Paper Review] Offline bilingual word vectors, orthogonal transformations and the inverted softmax

Samuel Smith, David H. P. Turban+2 more|arXiv (Cornell University)|Feb 13, 2017

Natural Language Processing Techniques17 references297 citations

TL;DR

The paper proves that offline bilingual word vector mappings should be orthogonal and obtainable via SVD, introduces an inverted softmax to improve translation, and shows robust results including with pseudo-dictionaries and sentence-level retrieval.

ABSTRACT

Usually bilingual word vectors are trained "online". Mikolov et al. showed they can also be found "offline", whereby two pre-trained embeddings are aligned with a linear transformation, using dictionaries compiled from expert knowledge. In this work, we prove that the linear transformation between two spaces should be orthogonal. This transformation can be obtained using the singular value decomposition. We introduce a novel "inverted softmax" for identifying translation pairs, with which we improve the precision @1 of Mikolov's original mapping from 34% to 43%, when translating a test set composed of both common and rare English words into Italian. Orthogonal transformations are more robust to noise, enabling us to learn the transformation without expert bilingual signal by constructing a "pseudo-dictionary" from the identical character strings which appear in both languages, achieving 40% precision on the same test set. Finally, we extend our method to retrieve the true translations of English sentences from a corpus of 200k Italian sentences with a precision @1 of 68%.

Motivation & Objective

• Show that the linear map between bilingual word spaces should be orthogonal and obtainable via SVD.
• Introduce an inverted softmax to mitigate hubness in bilingual translation.
• Demonstrate robustness by using pseudo-dictionaries from identical strings and by leveraging aligned sentences.
• Extend the approach to sentence-level translation and retrieval in large bilingual corpora.

Proposed method

• Formulate bilingual word space alignment as learning an orthogonal transformation O via SVD from dictionary pairs: M = Y_D^T X_D = U Σ V^T, with O = U V^T.
• Define the similarity matrix S = Y O X^T and optimize the mapping to maximize y_i^T O x_i over dictionary pairs.
• Introduce inverted softmax P_{j→i} = e^{β S_{ij}} / (α_j Σ_n e^{β S_{in}}) to reduce hubness by normalizing over source words.
• Train β by maximizing the log-likelihood over the training dictionary.
• Compute translations by nearest neighbor in the transformed space, and optionally apply dimensionality reduction by dropping components corresponding to small singular values.
• Explore pseudo-dictionaries created from identical character strings across languages and from Europarl-aligned sentences as weak bilingual signals.
• Construct simple sentence vectors by summing and normalizing word vectors, then align via SVD on a phrase dictionary to enable word and sentence translations.

Experimental results

Research questions

• RQ1 Can an orthogonal linear transform suffice to align monolingual word vectors across languages in an offline setting?
• RQ2 Does the inverted softmax reduce hubness and improve translation precision in bilingual mapping?
• RQ3 How robust is the orthogonal alignment when using weak or no bilingual signal (pseudo-dictionaries) or identical strings across languages?
• RQ4 Can sentence-level translation be achieved by applying the same alignment to simple sentence vectors derived from word vectors?

Key findings

• Orthogonal transformations (via a single SVD step) optimally align bilingual word spaces in the offline setting.
• Inverted softmax significantly improves translation precision over the original mapping (e.g., English→Italian from 34% to 43% @1 with a 5k dictionary).
• Using identical character strings as a pseudo-dictionary yields 40% precision @1 (English→Italian) without expert bilingual signals.
• Sentence-level translations from English to Italian reach about 68% precision @1 when retrieving the true translation from 200k Italian candidates.
• Using Europarl-aligned sentences as a phrase dictionary, the approach achieves 42.8% @1 English→Italian and 37.5% @1 Italian→English, comparable to word-dictionary results.
• The method also supports retrieving true translations of English sentences from large candidate sets with high accuracy (up to 68% @1 in certain setups).