[Paper Review] Portuguese Word Embeddings: Evaluating on Word Analogies and Natural Language Tasks

Nathan Siegle Hartmann, Erick Fonseca+4 more|arXiv (Cornell University)|Aug 20, 2017

Topic Modeling13 references91 citations

TL;DR

The paper trains 31 word embedding models for Portuguese (Brazilian and European) using GloVe, Word2Vec, Wang2Vec, and FastText, and evaluates them intrinsically on analogies and extrinsically on POS tagging and sentence similarity, showing analogies are poor predictors of downstream task performance and Wang2Vec offers strong overall results.

ABSTRACT

Word embeddings have been found to provide meaningful representations for words in an efficient way; therefore, they have become common in Natural Language Processing sys- tems. In this paper, we evaluated different word embedding models trained on a large Portuguese corpus, including both Brazilian and European variants. We trained 31 word embedding models using FastText, GloVe, Wang2Vec and Word2Vec. We evaluated them intrinsically on syntactic and semantic analogies and extrinsically on POS tagging and sentence semantic similarity tasks. The obtained results suggest that word analogies are not appropriate for word embedding evaluation; task-specific evaluations appear to be a better option.

Motivation & Objective

• Motivate the need for robust Portuguese word representations across variants (PT-BR and PT-EU).
• Provide a large, public set of trained Portuguese word embedding models for research use.
• Assess both intrinsic (analogies) and extrinsic (POS tagging, sentence similarity) performance of embeddings.
• Investigate how corpus size and model choice affect embedding quality in downstream tasks.

Proposed method

• Trained 31 word embedding models using four algorithms: GloVe, Word2Vec, Wang2Vec, and FastText.
• Constructed a large multi-genre Portuguese corpus combining PT-BR and PT-EU texts.
• Evaluated embeddings intrinsically on syntactic and semantic analogies in PT-BR and PT-EU.
• Evaluated embeddings extrinsically on POS tagging and sentence semantic similarity tasks.
• Explored multiple dimensionalities: 50, 100, 300, 600, and 1000 for each model.

Experimental results

Research questions

• RQ1 How do different Portuguese word embedding models (across PT-BR and PT-EU) perform on intrinsic analogy tasks versus extrinsic NLP tasks?
• RQ2 Do intrinsic analogy results correlate with downstream tasks such as POS tagging and sentence similarity in Portuguese?
• RQ3 Which modeling approaches (GloVe, Word2Vec, Wang2Vec, FastText) and which dimensions best support Portuguese language processing across variants?
• RQ4 Does mixing Brazilian and European Portuguese in a single large corpus improve or hinder downstream task performance?

Key findings

• GloVe generally performs best on intrinsic syntactic and semantic analogy evaluations for both PT-BR and PT-EU variants.
• FastText yields strong performance on syntactic analogies, while Wang2Vec often excels overall due to its handling of word order.
• For semantic analogies, GloVe leads, with Wang2Vec providing competitive results.
• In extrinsic POS tagging, Wang2Vec (especially 300-d Skip-Gram) achieves the highest accuracy, sometimes outperforming Word2Vec and other methods.
• In semantic similarity (ASSIN task), the best embedding results do not always align with analogy performance, indicating a lack of correlation between intrinsic analogies and downstream NLP tasks.
• Overall, the study finds that analogy-based evaluation may not be reliable for selecting embeddings for real NLP tasks; task-specific evaluation is preferred.