[Paper Review] Text Classification: A Perspective of Deep Learning Methods

Zhongwei Wan|arXiv (Cornell University)|Sep 24, 2023

Text and Document Classification TechnologiesComputer Science3 citations

TL;DR

This paper provides a comprehensive overview of deep learning-based text classification, covering key techniques in feature extraction (e.g., Word2Vec, GloVe), dimensionality reduction (e.g., PCA, LDA), and state-of-the-art models such as LSTM, GRU, Transformer, BERT, XLNet, and GPT-2. It highlights how these models improve text classification through contextual representation learning and pre-training strategies, establishing their dominance over traditional machine learning methods.

ABSTRACT

In recent years, with the rapid development of information on the Internet, the number of complex texts and documents has increased exponentially, which requires a deeper understanding of deep learning methods in order to accurately classify texts using deep learning techniques, and thus deep learning methods have become increasingly important in text classification. Text classification is a class of tasks that automatically classifies a set of documents into multiple predefined categories based on their content and subject matter. Thus, the main goal of text classification is to enable users to extract information from textual resources and process processes such as retrieval, classification, and machine learning techniques together in order to classify different categories. Many new techniques of deep learning have already achieved excellent results in natural language processing. The success of these learning algorithms relies on their ability to understand complex models and non-linear relationships in data. However, finding the right structure, architecture, and techniques for text classification is a challenge for researchers. This paper introduces deep learning-based text classification algorithms, including important steps required for text classification tasks such as feature extraction, feature reduction, and evaluation strategies and methods. At the end of the article, different deep learning text classification methods are compared and summarized.

Motivation & Objective

• To survey and analyze deep learning-based approaches for text classification in the context of rapidly growing textual data.
• To examine the role of word embedding techniques like Word2Vec and GloVe in capturing semantic relationships for improved classification.
• To evaluate the effectiveness of deep neural networks such as LSTM, GRU, and Transformer-based models in text classification tasks.
• To compare pre-trained models like BERT, XLNet, and GPT-2 in terms of architecture, pre-training objectives, and fine-tuning strategies.
• To summarize evaluation metrics and best practices for model selection and performance assessment in text classification.

Proposed method

• Utilizes Word2Vec and GloVe for learning dense, distributed word representations that capture semantic and syntactic relationships.
• Applies dimensionality reduction techniques like PCA and LDA to reduce feature space while preserving discriminative information.
• Employs recurrent networks (LSTM, GRU) for sequential modeling of text with gated mechanisms to handle long-range dependencies.
• Introduces the Transformer architecture with self-attention mechanisms to model long-range dependencies in parallel, improving efficiency and performance.
• Leverages bidirectional pre-training via BERT’s Masked Language Model (MLM) and Next Sentence Prediction (NSP) objectives for contextual representation learning.
• Adopts XLNet’s permutation-based autoregressive training with selective attention to model dependencies more effectively than BERT, avoiding its independence assumption.

Experimental results

Research questions

• RQ1 How do word embedding methods like Word2Vec and GloVe improve semantic representation in text classification compared to traditional bag-of-words models?
• RQ2 What are the advantages and limitations of using recurrent networks (LSTM, GRU) in modeling sequential dependencies in text classification?
• RQ3 How do Transformer-based models like BERT and GPT-2 outperform previous architectures in text classification tasks?
• RQ4 In what ways does XLNet’s permutation-based training strategy overcome the limitations of BERT’s masked pre-training objective?
• RQ5 What evaluation metrics and strategies are most effective for assessing deep learning models in text classification?

Key findings

• Word2Vec and GloVe significantly improve semantic representation by learning dense vector representations that capture word similarity and context.
• PCA and LDA are effective for reducing high-dimensional text features, improving model efficiency and sometimes accuracy in traditional machine learning pipelines.
• LSTM and GRU models achieve better performance than simple RNNs by mitigating vanishing gradient problems and modeling long-range dependencies.
• The Transformer architecture enables parallel training and superior modeling of long-range dependencies through multi-head self-attention, outperforming recurrent models in many tasks.
• BERT’s bidirectional pre-training with MLM and NSP leads to strong contextual representations, significantly boosting performance on downstream text classification tasks.
• XLNet’s autoregressive training with permutation and selective attention improves modeling of dependencies and avoids BERT’s independence assumption, achieving state-of-the-art results in some benchmarks.