[Paper Review] A Simple and Efficient Ensemble Classifier Combining Multiple Neural Network Models on Social Media Datasets in Vietnamese

Huy Duc Huynh, Hang Thi-Thuy Do+2 more|arXiv (Cornell University)|Sep 28, 2020

Sentiment Analysis and Opinion Mining4 citations

TL;DR

This paper proposes a simple yet effective ensemble classifier combining CNN, LSTM, and BERT models for multi-class text classification on Vietnamese social media datasets. By leveraging model-specific strengths and optimizing preprocessing, the ensemble achieves state-of-the-art F1-scores of 86.96% (HSD-VLSP), 65.79% (UIT-VSMEC), and 92.79% (sentiment) / 89.70% (topic) on UIT-VSFC, outperforming single models and prior work.

ABSTRACT

Text classification is a popular topic of natural language processing, which has currently attracted numerous research efforts worldwide. The significant increase of data in social media requires the vast attention of researchers to analyze such data. There are various studies in this field in many languages but limited to the Vietnamese language. Therefore, this study aims to classify Vietnamese texts on social media from three different Vietnamese benchmark datasets. Advanced deep learning models are used and optimized in this study, including CNN, LSTM, and their variants. We also implement the BERT, which has never been applied to the datasets. Our experiments find a suitable model for classification tasks on each specific dataset. To take advantage of single models, we propose an ensemble model, combining the highest-performance models. Our single models reach positive results on each dataset. Moreover, our ensemble model achieves the best performance on all three datasets. We reach 86.96% of F1- score for the HSD-VLSP dataset, 65.79% of F1-score for the UIT-VSMEC dataset, 92.79% and 89.70% for sentiments and topics on the UIT-VSFC dataset, respectively. Therefore, our models achieve better performances as compared to previous studies on these datasets.

Motivation & Objective

• To address the lack of effective deep learning models for multi-class Vietnamese text classification on social media.
• To evaluate the performance of CNN, LSTM, and BERT models on three Vietnamese benchmark datasets with multi-label and imbalanced labels.
• To design and implement an ensemble method that combines the strengths of individual models to improve overall classification accuracy.
• To investigate the impact of preprocessing and domain-specific word embeddings on model performance for Vietnamese social media text.
• To establish a baseline for future research on Vietnamese NLP tasks using deep learning.

Proposed method

• Trained and optimized CNN, LSTM, and their variants using pre-trained fastText embeddings on Vietnamese social media text.
• Applied BERT for the first time on the three Vietnamese datasets, fine-tuning it for multi-class classification.
• Implemented a normalized word dictionary to improve word representation and handle out-of-vocabulary words due to misspellings.
• Used five-fold cross-validation to evaluate model performance and ensure robustness.
• Constructed an ensemble model by combining predictions from top-performing single models, with label-specific weighting based on fold-wise performance.
• Optimized model depth and hyperparameters based on dataset size and domain characteristics.

Experimental results

Research questions

• RQ1 How do CNN, LSTM, and BERT models perform individually on Vietnamese social media text classification tasks?
• RQ2 Can an ensemble of multiple neural network models improve classification performance beyond individual models on multi-label, imbalanced Vietnamese datasets?
• RQ3 What is the impact of preprocessing and domain-specific word embeddings on model accuracy for Vietnamese social media text?
• RQ4 Which model architecture achieves the highest stability and performance across different folds and datasets?
• RQ5 Does BERT outperform traditional models on Vietnamese text, especially on smaller or domain-specific datasets?

Key findings

• The ensemble model achieved the highest F1-score of 86.96% on the HSD-VLSP dataset for hate speech detection, surpassing all single models and prior studies.
• On the UIT-VSMEC dataset, the ensemble model reached 65.79% F1-score for emotion recognition, outperforming previous methods.
• For the UIT-VSFC dataset, the ensemble model achieved 92.79% F1-score in sentiment classification and 89.70% in topic classification, demonstrating strong performance on multi-label tasks.
• The CNN model showed the most stable performance across all folds, achieving 99.42% accuracy on the CLEAN label in the HSD-VLSP dataset.
• The LSTM model performed best on the HATE label (85.10% accuracy), and the ensemble improved this to 85.39% by incorporating its predictions.
• Preprocessing had a positive effect on large datasets like HSD-VLSP but reduced performance on smaller datasets like UIT-VSMEC, indicating dataset-size-dependent optimization is necessary.