[Paper Review] A Low Dimensionality Representation for Language Variety Identification

Francisco Rangel, Marc Franco-Salvador+1 more|arXiv (Cornell University)|May 30, 2017

Authorship Attribution and Profiling8 references17 citations

TL;DR

This paper proposes a low dimensionality representation (LDR) for language variety identification in Spanish, using statistical features like mean, standard deviation, and probability-based term weights to reduce dimensionality to just 6 features per variety. The method achieves a 35% accuracy improvement over state-of-the-art baselines and shows competitive performance against distributed models like Skip-gram and SenVec, with strong robustness across diverse language varieties in big data settings.

ABSTRACT

Language variety identification aims at labelling texts in a native language (e.g. Spanish, Portuguese, English) with its specific variation (e.g. Argentina, Chile, Mexico, Peru, Spain; Brazil, Portugal; UK, US). In this work we propose a low dimensionality representation (LDR) to address this task with five different varieties of Spanish: Argentina, Chile, Mexico, Peru and Spain. We compare our LDR method with common state-of-the-art representations and show an increase in accuracy of ~35%. Furthermore, we compare LDR with two reference distributed representation models. Experimental results show competitive performance while dramatically reducing the dimensionality --and increasing the big data suitability-- to only 6 features per variety. Additionally, we analyse the behaviour of the employed machine learning algorithms and the most discriminating features. Finally, we employ an alternative dataset to test the robustness of our low dimensionality representation with another set of similar languages.

Motivation & Objective

• To address the challenge of identifying language varieties (e.g., Mexican, Spanish, Argentinean Spanish) in social media texts with high accuracy.
• To reduce dimensionality in text representations for scalability in big data environments, especially for social media applications.
• To develop a feature set that captures lexical and statistical differences between language varieties without relying on n-gram or deep learning-based representations.
• To ensure generalization by avoiding data leakage between training and test sets through strict author-level separation.
• To release the dataset publicly to support reproducibility and further research in language variety identification.

Proposed method

• The LDR method computes statistical features—mean, standard deviation, minimum, maximum, probability, and proportionality—of term frequencies per language variety.
• Each document is represented by a 6-dimensional vector per language variety, derived from term frequency distributions across training data for that variety.
• The method uses tf-idf weighting to emphasize terms that are discriminative per variety, reducing noise from common vocabulary.
• Features are selected based on information gain, with the most discriminative ones prioritized for classification.
• A machine learning classifier (e.g., SVM or similar) is trained on the low-dimensional feature vectors to predict the correct language variety.
• The approach avoids overfitting by ensuring no author appears in both training and test sets, preserving evaluation integrity.

Experimental results

Research questions

• RQ1 Can a low-dimensional feature representation outperform traditional n-gram and state-of-the-art representations in language variety identification?
• RQ2 To what extent does reducing feature dimensionality to just 6 features per variety affect classification accuracy and scalability in big data settings?
• RQ3 How do different statistical features (mean, standard deviation, min/max, probability) contribute to discriminating between closely related language varieties?
• RQ4 Does the LDR method generalize well to other language pairs or varieties beyond Spanish, especially in similar language discrimination tasks?
• RQ5 How does the LDR compare in performance to distributed representations like Skip-gram and SenVec on diverse language varieties?

Key findings

• The LDR method achieved an accuracy of 71.1% on the Spanish language variety identification task, representing a ~35% improvement over baseline methods.
• The method reduced feature dimensionality from thousands to only 6 features per language variety, significantly enhancing big data suitability.
• Standard deviation-based features alone achieved 69.2% accuracy, and their combination with mean features reached 70.8%, outperforming min/max-based features.
• The LDR showed robust performance across diverse language pairs in the DSLCC corpus, with accuracy ranging from 78.0% (Bosnian) to 99.9% (Indonesian, Czech, etc.), comparable to Skip-gram and SenVec.
• In the Portuguese and Spanish varieties of the DSLCC set, LDR achieved 84.7% (Spain), 88.0% (Argentina), 87.4% (Portugal), and 90.0% (Brazil), demonstrating strong cross-linguistic generalization.
• The LDR method was successfully applied to age and gender identification in author profiling, achieving competitive results with top teams in the PAN 2018 competition.