[Paper Review] Transformer-based Language Models for Factoid Question Answering at BioASQ9b

Urvashi Khanna, Diego Mollá|arXiv (Cornell University)|Sep 15, 2021

Topic Modeling2 references4 citations

TL;DR

This paper evaluates ALBERT and DistilBERT fine-tuned for factoid question answering in the BioASQ9b challenge, using multi-task fine-tuning on SQuAD and gradual unfreezing techniques. Despite lower parameter count, DistilBERT outperformed ALBERT on test batches 4 and 5, while gradual unfreezing showed no significant accuracy gain over standard fine-tuning.

ABSTRACT

In this work, we describe our experiments and participating systems in the BioASQ Task 9b Phase B challenge of biomedical question answering. We have focused on finding the ideal answers and investigated multi-task fine-tuning and gradual unfreezing techniques on transformer-based language models. For factoid questions, our ALBERT-based systems ranked first in test batch 1 and fourth in test batch 2. Our DistilBERT systems outperformed the ALBERT variants in test batches 4 and 5 despite having 81% fewer parameters than ALBERT. However, we observed that gradual unfreezing had no significant impact on the model's accuracy compared to standard fine-tuning.

Motivation & Objective

• To improve factoid question answering in biomedical QA using transfer learning with transformer-based models.
• To investigate the impact of multi-task fine-tuning on low-resource biomedical datasets like BioASQ9b.
• To evaluate whether gradual unfreezing improves model performance on factoid QA compared to standard fine-tuning.
• To compare the performance of larger ALBERT and smaller, efficient DistilBERT models in the BioASQ9b challenge.

Proposed method

• Fine-tuned ALBERT on SQuAD2.0 before fine-tuning on BioASQ9b, using multi-task learning to improve adaptation.
• Fine-tuned DistilBERT on SQuAD1.1 before fine-tuning on BioASQ9b, leveraging its smaller size for efficiency.
• Applied gradual unfreezing by progressively updating layers from the bottom up during fine-tuning.
• Used standard fine-tuning as a baseline for comparison with gradual unfreezing.
• Evaluated models using Mean Reciprocal Rank (MRR) on multiple test batches of the BioASQ9b leaderboard.
• Conducted paired t-tests to assess statistical significance of performance differences between unfreezing and standard fine-tuning.

Experimental results

Research questions

• RQ1 Does multi-task fine-tuning on SQuAD improve performance on the low-resource BioASQ9b dataset?
• RQ2 Can DistilBERT achieve competitive performance compared to larger models like ALBERT in biomedical factoid QA?
• RQ3 Does gradual unfreezing lead to significant improvements in MRR over standard fine-tuning for transformer-based models in this setting?
• RQ4 Is there a performance trade-off between model size and accuracy in biomedical question answering?

Key findings

• The ALBERT-based system ranked first in test batch 1 and fourth in test batch 2, demonstrating strong performance on earlier test sets.
• DistilBERT outperformed ALBERT in test batches 4 and 5, achieving an average MRR of 0.5399 and 0.5171 respectively, despite having 81% fewer parameters.
• The system using gradual unfreezing on DistilBERT showed no statistically significant improvement in MRR compared to standard fine-tuning, with a p-value indicating no significant difference.
• The average MRR for the unfreezing variant (0.5232) was only slightly higher than the standard fine-tuned DistilBERT (0.5209), confirming no meaningful gain from the technique.
• The results from BioASQ7b in prior work support the current findings, showing no consistent benefit from gradual unfreezing across all test batches.
• The study suggests that efficient models like DistilBERT are viable for mobile and edge-based biomedical QA applications due to their high performance and low parameter count.