[Paper Review] Sparse*BERT: Sparse Models Generalize To New tasks and Domains

Daniel Campos, Alexandre Carriconde Marques+3 more|arXiv (Cornell University)|May 25, 2022

Topic Modeling4 citations

TL;DR

This paper introduces Sparse*BERT, a pruned BERT model trained via gradual unstructured magnitude pruning during general-domain pretraining, which generalizes effectively to new domains like biomedicine without task-specific hyperparameter tuning. The resulting SparseBioBERT achieves BioBERT-level performance with only 10% of the parameters, demonstrating strong zero-shot transferability and robustness across diverse biomedical NLP tasks and data sparsity scenarios.

ABSTRACT

Large Language Models have become the core architecture upon which most modern natural language processing (NLP) systems build. These models can consistently deliver impressive accuracy and robustness across tasks and domains, but their high computational overhead can make inference difficult and expensive. To make using these models less costly, recent work has explored leveraging structured and unstructured pruning, quantization, and distillation to improve inference speed and decrease size. This paper studies how models pruned using Gradual Unstructured Magnitude Pruning can transfer between domains and tasks. Our experimentation shows that models that are pruned during pretraining using general domain masked language models can transfer to novel domains and tasks without extensive hyperparameter exploration or specialized approaches. We demonstrate that our general sparse model Sparse*BERT can become SparseBioBERT simply by pretraining the compressed architecture on unstructured biomedical text. Moreover, we show that SparseBioBERT can match the quality of BioBERT with only 10\% of the parameters.

Motivation & Objective

• To investigate whether pruned large language models can generalize effectively to new domains and tasks without extensive hyperparameter tuning.
• To evaluate the impact of pruning stage—pretraining vs. fine-tuning—on transfer performance in biomedical NLP.
• To develop a sparse, general-purpose model that can be adapted to specialized domains through minimal, domain-specific pretraining.
• To demonstrate that sparse models can match or exceed dense models in performance while drastically reducing parameter count.

Proposed method

• Pruning is applied via gradual unstructured magnitude pruning during general-domain pretraining on uncased BERT.
• The pruned architecture is fine-tuned on domain-specific biomedical text to create SparseBioBERT, a specialized sparse model.
• Knowledge distillation is used during fine-tuning with a temperature of 2.0 and hardness of 0.8 to preserve performance.
• Models are evaluated on entity extraction, relation extraction, and question answering tasks across multiple biomedical datasets.
• Training data size is systematically varied (1% to 100%) to assess sample efficiency and robustness to data scarcity.
• Experiments compare performance of models pruned during pretraining (Sparse*BERT), pruned during fine-tuning, and dense BERT across all settings.

Experimental results

Research questions

• RQ1 Can a pruned large language model trained on general-domain data transfer effectively to new domains without additional hyperparameter tuning?
• RQ2 Does pruning during pretraining yield better transfer performance than pruning during fine-tuning, especially on low-resource biomedical NLP tasks?
• RQ3 How does model sparsity affect performance under data scarcity, and how does it compare to dense models in terms of sample efficiency?
• RQ4 Can a single sparse architecture be adapted to a new domain (e.g., biomedicine) via minimal domain-specific pretraining to achieve performance comparable to dense domain-specific models?

Key findings

• Sparse*BERT, pruned during general-domain pretraining, achieves comparable performance to BioBERT on biomedical NLP tasks with only 10% of the active parameters.
• Pruning during pretraining outperforms pruning during fine-tuning by nearly 4 points on entity extraction and 2 points overall, even surpassing the gap between BERT and BioBERT.
• Sparse*BERT maintains high performance across varying training data sizes, with F1 scores dropping only slightly from 88.50 at 100% to 60.91 at 1% of training data, while fine-tuned pruned models degrade rapidly.
• On small datasets like NCBI and GAD, models pruned during fine-tuning suffer significant accuracy loss, whereas Sparse*BERT maintains robustness.
• The model pruned during pretraining generalizes better than models pruned during fine-tuning, especially in low-data regimes, indicating that early pruning preserves transferability.
• SparseBioBERT achieves state-of-the-art performance for its parameter count, matching BioBERT’s accuracy with 90% fewer active parameters and no task-specific optimization.