[Paper Review] Automated classification for open-ended questions with BERT

Hyukjun Gweon, Matthias Schonlau|arXiv (Cornell University)|Sep 13, 2022

Natural Language Processing Techniques27 references4 citations

TL;DR

This paper evaluates fine-tuned BERT for automated classification of open-ended survey responses, comparing it to non-pretrained models like SVM, random forests, and XGBoost. Results show BERT significantly outperforms these models as training data increases, with a clear accuracy advantage emerging beyond 200–400 labeled examples, making it a superior choice for large-scale text coding in social science research.

ABSTRACT

Manual coding of text data from open-ended questions into different categories is time consuming and expensive. Automated coding uses statistical/machine learning to train on a small subset of manually coded text answers. Recently, pre-training a general language model on vast amounts of unrelated data and then adapting the model to the specific application has proven effective in natural language processing. Using two data sets, we empirically investigate whether BERT, the currently dominant pre-trained language model, is more effective at automated coding of answers to open-ended questions than other non-pre-trained statistical learning approaches. We found fine-tuning the pre-trained BERT parameters is essential as otherwise BERT's is not competitive. Second, we found fine-tuned BERT barely beats the non-pre-trained statistical learning approaches in terms of classification accuracy when trained on 100 manually coded observations. However, BERT's relative advantage increases rapidly when more manually coded observations (e.g. 200-400) are available for training. We conclude that for automatically coding answers to open-ended questions BERT is preferable to non-pretrained models such as support vector machines and boosting.

Motivation & Objective

• To assess whether BERT, a pre-trained language model, improves automated classification of open-ended survey responses compared to non-pretrained statistical models.
• To investigate the necessity and impact of fine-tuning BERT on small training sets typical in survey research.
• To compare BERT’s performance with traditional models (SVM, random forests, XGBoost) across varying sizes of manually coded training data.
• To evaluate BERT’s robustness in handling imbalanced class distributions common in survey data.
• To determine the practical feasibility and scalability of using BERT for automated coding in real-world survey applications.

Proposed method

• Fine-tuned the uncased BERT base model (12 layers, 768 hidden units, 12 attention heads) on two real-world survey datasets: Patient Joe and Disclosure.
• Used cross-entropy loss with 10% dropout and trained for 3 epochs using a batch size of 8 and a fine-tuning learning rate of 0.0003.
• Converted text responses into unigram-based numerical features for non-BERT models (SVM, random forests, XGBoost) to ensure fair comparison.
• Optimized hyperparameters for each non-BERT model using grid search (SVM, RF) or random search (XGBoost) over 50 iterations.
• Evaluated models on training sets of 100 to 400 observations, using 100 held-out samples for validation and the remainder for testing.
• Measured performance using overall accuracy, macro-accuracy, and class-wise accuracy to assess both overall and per-class performance.

Experimental results

Research questions

• RQ1 Does fine-tuning BERT significantly improve classification accuracy for open-ended survey responses compared to non-pretrained models?
• RQ2 How does BERT’s performance vary with increasing amounts of manually coded training data?
• RQ3 Is BERT’s performance biased toward high-frequency classes in imbalanced datasets?
• RQ4 What is the relative advantage of BERT over traditional models like SVM, random forests, and XGBoost when training data is limited (e.g., 100–400 examples)?
• RQ5 Can BERT be practically deployed in social science research despite computational and implementation challenges?

Key findings

• Fine-tuning BERT improved its classification accuracy by 10–20 percentage points, making it competitive; without fine-tuning, BERT underperformed non-pretrained models.
• With only 100 training examples, BERT’s accuracy was comparable to non-pretrained models, but its performance gap widened with more data, becoming significantly better at 200–400 examples.
• BERT achieved higher accuracy across most classes without notable degradation in performance for low-frequency classes, indicating robustness to class imbalance.
• Macro-accuracy results confirmed that BERT’s advantage was consistent across all classes, not just dominant ones, supporting its fairness in multi-class settings.
• The relative advantage of BERT increased with training data size, suggesting it is particularly effective when larger labeled datasets are available.
• Despite higher computational and programming demands, BERT offers reproducible, scalable classification—especially valuable for large survey datasets or longitudinal studies.