[Paper Review] Text Classification based on Word Subspace with Term-Frequency

Erica K. Shimomoto, Lincon S. Souza+2 more|arXiv (Cornell University)|Jun 8, 2018

Text and Document Classification Technologies3 citations

TL;DR

This paper proposes a novel text classification method using word subspace and term-frequency (TF) weighted word subspace within the mutual subspace method (MSM) framework. By modeling word2vec vectors as low-dimensional subspaces and incorporating word frequency via weighted PCA, the approach achieves 92.01% accuracy on the Reuters-8 dataset, significantly outperforming traditional methods like MNB and SVM at a 95% significance level.

ABSTRACT

Text classification has become indispensable due to the rapid increase of text in digital form. Over the past three decades, efforts have been made to approach this task using various learning algorithms and statistical models based on bag-of-words (BOW) features. Despite its simple implementation, BOW features lack semantic meaning representation. To solve this problem, neural networks started to be employed to learn word vectors, such as the word2vec. Word2vec embeds word semantic structure into vectors, where the angle between vectors indicates the meaningful similarity between words. To measure the similarity between texts, we propose the novel concept of word subspace, which can represent the intrinsic variability of features in a set of word vectors. Through this concept, it is possible to model text from word vectors while holding semantic information. To incorporate the word frequency directly in the subspace model, we further extend the word subspace to the term-frequency (TF) weighted word subspace. Based on these new concepts, text classification can be performed under the mutual subspace method (MSM) framework. The validity of our modeling is shown through experiments on the Reuters text database, comparing the results to various state-of-art algorithms.

Motivation & Objective

• To address the limitations of bag-of-words models, which lack semantic meaning and suffer from high dimensionality and sparsity.
• To model text using word2vec vectors while preserving semantic relationships through a low-dimensional subspace representation.
• To incorporate word frequency information directly into the subspace model to improve classification accuracy.
• To evaluate the effectiveness of word subspace and TF-weighted word subspace in text classification under the MSM framework.

Proposed method

• Introduces the concept of word subspace as a low-dimensional linear subspace in high-dimensional word vector space, representing the intrinsic variability of word vectors in a text class.
• Applies PCA without data centering to model word subspaces for each class, preserving the dominant semantic structure of the class.
• Extends word subspace to TF-weighted word subspace by incorporating term frequencies as weights in the PCA process, enabling frequency-aware subspace modeling.
• Performs text classification using the mutual subspace method (MSM), comparing query document subspaces with training class subspaces via canonical angles.
• Uses word2vec to generate distributed vector representations that encode semantic similarity through vector geometry.
• Employs a weighted PCA variant where word frequencies serve as instance weights during subspace learning, enhancing representation fidelity.

Experimental results

Research questions

• RQ1 Can a low-dimensional subspace representation of word2vec vectors effectively model the semantic content of text classes in text classification?
• RQ2 How does incorporating term frequency into the subspace modeling process affect classification performance compared to standard word subspace?
• RQ3 Does the proposed TF-weighted word subspace method outperform conventional text classification baselines like MNB, SVM, and LSA on benchmark datasets?
• RQ4 To what extent does the subspace-based method preserve semantic variability while reducing dimensionality and computational cost?
• RQ5 Is the performance gain from TF-weighted subspace statistically significant compared to frequency-agnostic methods?

Key findings

• The TF-weighted word subspace method (TF-MSM) achieved 92.01% accuracy on the Reuters-8 dataset, significantly outperforming multinomial naive Bayes (MNB), which achieved lower accuracy.
• A t-test confirmed that TF-MSM's performance was statistically superior to MNB at the 95% significance level, with a p-value of 0.031.
• The word subspace representation retained most of the class variability using only half the dimensions of the original word vector space, indicating high compactness and efficiency.
• LSA and SVM performed poorly with TF-IDF weighted bag-of-words features, suggesting that TF-IDF is less effective when stop words are removed and rare words are underrepresented.
• The method effectively models semantic similarity through vector angles and subspace geometry, demonstrating that word2vec vectors grouped by context form coherent subspaces.
• The results show that word subspace modeling under MSM is more effective than conventional methods when applied to distributed word representations.