[Paper Review] AdaptKeyBERT: An Attention-Based approach towards Few-Shot & Zero-Shot Domain Adaptation of KeyBERT

Aman Priyanshu, Supriti Vijay|arXiv (Cornell University)|Nov 14, 2022

Advanced Text Analysis Techniques4 citations

TL;DR

AdaptKeyBERT proposes an attention-based fine-tuning framework that enables few-shot and zero-shot domain adaptation for KeyBERT using regularized attention over candidate and seed word embeddings. It improves keyword extraction performance by up to 12.58% in F1-score on low-resource domains like FAO-780 and CERN-290, outperforming baseline KeyBERT and enabling efficient adaptation without full retraining.

ABSTRACT

Keyword extraction has been an important topic for modern natural language processing. With its applications ranging from ontology generation, fact verification in summarized text, and recommendation systems. While it has had significant data-intensive applications, it is often hampered when the data set is small. Downstream training for keyword extractors is a lengthy process and requires a significant amount of data. Recently, Few-shot Learning (FSL) and Zero-Shot Learning (ZSL) have been proposed to tackle this problem. Therefore, we propose AdaptKeyBERT, a pipeline for training keyword extractors with LLM bases by incorporating the concept of regularized attention into a pre-training phase for downstream domain adaptation. As we believe our work has implications to be utilized in the pipeline of FSL/ZSL and keyword extraction, we open-source our code as well as provide the fine-tuning library of the same name AdaptKeyBERT at https://github.com/AmanPriyanshu/AdaptKeyBERT.

Motivation & Objective

• To address the challenge of low-resource keyword extraction in specialized domains such as biomedicine, law, and high-energy physics.
• To enable effective few-shot and zero-shot domain adaptation for keyword extractors without requiring large-scale labeled data.
• To improve the performance of pre-trained BERT-based models like KeyBERT in domain-specific contexts through attention-based embedding refinement.
• To provide an open-source, API-integrated library for easy deployment and fine-tuning of domain-adapted keyword extractors.

Proposed method

• Integrates regularized attention into KeyBERT’s training module to compute similarity weights between candidate keywords and seed words.
• Uses a weighted combination of document embeddings and seed word embeddings via attention, controlled by a learnable regularizer α ∈ [0,1].
• Employs a hybrid loss function that reconstructs candidate embeddings using attention-computed weights, improving domain relevance.
• Defines popular keywords as those appearing in >10% of the dataset, using them as seed words for zero-shot adaptation.
• Applies few-shot adaptation by selecting samples containing the most frequent keywords, limiting to 10% of the dataset for efficiency.
• Combines zero-shot and few-shot modules via joint similarity computation, enabling joint domain adaptation.

Experimental results

Research questions

• RQ1 Can attention-based embedding reconstruction improve keyword extraction performance in low-resource domains?
• RQ2 How does the integration of few-shot and zero-shot learning enhance domain adaptation in keyword extraction?
• RQ3 To what extent does regularized attention improve the relevance of extracted keywords compared to standard KeyBERT?
• RQ4 Does the hybrid few-shot and zero-shot approach outperform individual adaptation strategies on diverse domains?

Key findings

• On the FAO-780 dataset, AdaptKeyBERT achieved a 12.58% improvement in F1-score (39.559) over the baseline KeyBERT (35.138) in the few-shot setting.
• In the zero-shot setting on FAO-780, the F1-score improved by 7.88% (from 35.138 to 37.908) compared to the baseline.
• The combined few-shot and zero-shot approach achieved an F1-score of 39.938 on FAO-780, a 13.66% improvement over the baseline.
• On the CERN-290 dataset, the few-shot variant improved F1-score by 9.95% (from 25.627 to 28.177), and the combined method achieved 28.883 F1.
• The zero-shot module showed a 3.81% F1-score gain on CERN-290, though recall slightly decreased due to increased false negatives.
• The model demonstrated consistent generalization across linguistically complex domains, validating its robustness in low-resource settings.