[Paper Review] Empowering Interdisciplinary Research with BERT-Based Models: An Approach Through SciBERT-CNN with Topic Modeling

Darya Likhareva, Hamsini Sankaran+1 more|arXiv (Cornell University)|Apr 16, 2024

Computational and Text Analysis Methods4 citations

TL;DR

This paper proposes a SciBERT-CNN hybrid model enhanced with BERT topic modeling and class weighting to improve multi-label classification of scientific articles from the Elsevier OA CC-BY corpus. By integrating abstracts, body text, titles, and top-topic keywords through a multi-segment input strategy, the model achieves significant gains in F1 score and robustness, especially in handling class imbalance and interdisciplinary content.

ABSTRACT

Researchers must stay current in their fields by regularly reviewing academic literature, a task complicated by the daily publication of thousands of papers. Traditional multi-label text classification methods often ignore semantic relationships and fail to address the inherent class imbalances. This paper introduces a novel approach using the SciBERT model and CNNs to systematically categorize academic abstracts from the Elsevier OA CC-BY corpus. We use a multi-segment input strategy that processes abstracts, body text, titles, and keywords obtained via BERT topic modeling through SciBERT. Here, the [CLS] token embeddings capture the contextual representation of each segment, concatenated and processed through a CNN. The CNN uses convolution and pooling to enhance feature extraction and reduce dimensionality, optimizing the data for classification. Additionally, we incorporate class weights based on label frequency to address the class imbalance, significantly improving the classification F1 score and enhancing text classification systems and literature review efficiency.

Motivation & Objective

• Address the challenge of classifying interdisciplinary scientific literature due to semantic complexity and high volume of publications.
• Overcome limitations of standard BERT models in capturing nuanced scientific language and handling class imbalance in multi-label text classification.
• Improve classification accuracy and efficiency for academic literature review by leveraging contextual embeddings and structured feature fusion.
• Reduce misclassification rates in low-frequency and broad subject categories (e.g., 'MULT', 'MEDI') through data-level and model-level optimization.
• Establish a benchmark for multi-label classification in scientific text using a hybrid architecture combining BERT, CNN, and topic modeling

Proposed method

• Employ a multi-segment input strategy that processes abstracts, body text, titles, and BERT-extracted top-10 keywords as separate inputs to the SciBERT model.
• Extract [CLS] token embeddings from each segment and concatenate them to form a unified contextual representation for downstream classification.
• Feed the concatenated embeddings into a 1D convolutional neural network (CNN) with ReLU activation, max-pooling, and dropout for feature extraction and dimensionality reduction.
• Apply a dense layer followed by a sigmoid activation function for multi-label classification, enabling independent prediction of multiple subject areas.
• Incorporate class weights proportional to inverse label frequency to mitigate the impact of long-tailed class distributions in the dataset.
• Fine-tune the SciBERT model on the Elsevier OA CC-BY corpus using a hybrid architecture that integrates topic modeling and CNN-based feature learning.

Experimental results

Research questions

• RQ1 Can a hybrid SciBERT-CNN model with multi-segment input and topic modeling improve multi-label classification accuracy for scientific articles compared to standard BERT models?
• RQ2 To what extent does incorporating class weights based on label frequency reduce misclassification in long-tailed subject categories?
• RQ3 How effective is the integration of BERT-extracted topic keywords in enhancing the model’s ability to classify interdisciplinary research papers?
• RQ4 What are the primary sources of misclassification in scientific text classification, and how do they relate to label breadth or domain specificity?
• RQ5 Can structural limitations in journal-based subject classification schemes negatively impact model performance, and if so, how can they be mitigated?

Key findings

• The SciBERT-CNN model achieved AUC scores ranging from 0.93 to 0.99 across all 18 subject areas, indicating strong discriminative performance.
• The best-performing categories were 'EART' (Earth Sciences) and 'MATE' (Materials Science), with high AUC and F1 scores, reflecting strong model generalization in these domains.
• Class imbalance significantly impacted initial performance, with F1 scores of 0.0 and 0.05 for 'DENT' and 'NURS' respectively, which were improved by merging low-frequency classes into broader categories.
• The 'MULT' (Multidisciplinary) label had the highest number of misclassifications due to its broad and heterogeneous definition across disciplines.
• The 'MEDI' (Medicine) category showed high misclassification rates due to overlapping terminology with 'BIOC' (Biology), highlighting challenges in semantic overlap between closely related fields.
• Most misclassifications resulted in no-label predictions, indicating insufficient contextual cues for the model to assign confident labels in ambiguous or sparse cases.