[Paper Review] K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters

Ruize Wang, Duyu Tang+7 more|arXiv (Cornell University)|Feb 5, 2020

Topic Modeling43 references135 citations

TL;DR

K-Adapter keeps the original pre-trained model fixed and adds knowledge-specific adapters to inject factual and linguistic knowledge, enabling continual, disentangled knowledge infusion and improved downstream task performance.

ABSTRACT

We study the problem of injecting knowledge into large pre-trained models like BERT and RoBERTa. Existing methods typically update the original parameters of pre-trained models when injecting knowledge. However, when multiple kinds of knowledge are injected, the historically injected knowledge would be flushed away. To address this, we propose K-Adapter, a framework that retains the original parameters of the pre-trained model fixed and supports the development of versatile knowledge-infused model. Taking RoBERTa as the backbone model, K-Adapter has a neural adapter for each kind of infused knowledge, like a plug-in connected to RoBERTa. There is no information flow between different adapters, thus multiple adapters can be efficiently trained in a distributed way. As a case study, we inject two kinds of knowledge in this work, including (1) factual knowledge obtained from automatically aligned text-triplets on Wikipedia and Wikidata and (2) linguistic knowledge obtained via dependency parsing. Results on three knowledge-driven tasks, including relation classification, entity typing, and question answering, demonstrate that each adapter improves the performance and the combination of both adapters brings further improvements. Further analysis indicates that K-Adapter captures versatile knowledge than RoBERTa.

Motivation & Objective

• Motivate the need to inject knowledge into large pre-trained models without overwriting previously learned knowledge.
• Propose a flexible adapter-based framework (K-Adapter) that keeps the backbone fixed and learns separate knowledge adapters.
• Demonstrate that multiple adapters can be trained independently and combined to improve knowledge-driven tasks.
• Showcase continual knowledge infusion where new adapters do not affect existing ones.

Proposed method

• Introduce knowledge-specific adapters plugged outside RoBERTa that take the backbone's intermediate hidden-states as input.
• Keep the RoBERTa parameters fixed while training adapters with small parameter counts (~42M per adapter).
• Pre-train adapters independently on distinct tasks: facAdapter for relation classification (factual knowledge) and linAdapter for dependency relation prediction (linguistic knowledge).
• Fuse adapter outputs by concatenating their representations (and with RoBERTa outputs) for downstream task layers; multiple adapters can be combined for tasks.
• Use RoBERTa-LARGE as backbone and configure adapters with N=2 transformer layers, hidden sizes H_A=768, H_d=768, H_u=1024, A_A=12.
• Evaluate on entity typing, question answering, and relation classification to assess gains from individual adapters and their combination.

Experimental results

Research questions

• RQ1 Can knowledge be infused into a fixed pre-trained model using external adapters without catastrophic forgetting?
• RQ2 Do separate factual and linguistic adapters provide complementary improvements on knowledge-driven tasks?
• RQ3 Does combining multiple adapters yield additive or synergistic gains over a single adapter?
• RQ4 How does K-Adapter perform on entity typing, QA, and relation classification compared to RoBERTa and multitask baselines?
• RQ5 Does K-Adapter improve the model's memorization of factual knowledge as measured by probing (LAMA)?

Key findings

• Each adapter (fact and/or linguistic) improves over the RoBERTa baseline on knowledge-driven tasks.
• The combination of factual and linguistic adapters yields further improvements beyond individual adapters.
• K-Adapter (F+L) achieves state-of-the-art results on several datasets among the tested tasks.
• Probing shows K-Adapter memorizes richer factual knowledge than RoBERTa, indicating enhanced knowledge capture.
• Disentangled adapters allow continual knowledge infusion without retraining or degrading previously injected knowledge.