[Paper Review] LLMAEL: Large Language Models are Good Context Augmenters for Entity Linking

Amy Xin, Yunjia Qi+6 more|arXiv (Cornell University)|Jul 4, 2024

Data Quality and Management4 citations

TL;DR

LLMaEL proposes a plug-and-play framework that enhances entity linking by using large language models (LLMs) to generate enriched, mention-centered context descriptions, which are then fed into traditional EL models to improve accuracy—especially for long-tail entities. The method achieves state-of-the-art performance across six benchmarks, with a 1.21% average accuracy gain using fine-tuned LLMs.

ABSTRACT

Specialized entity linking (EL) models are well-trained at mapping mentions to unique knowledge base (KB) entities according to a given context. However, specialized EL models struggle to disambiguate long-tail entities due to their limited training data. Meanwhile, extensively pre-trained large language models (LLMs) possess broader knowledge of uncommon entities. Yet, with a lack of specialized EL training, LLMs frequently fail to generate accurate KB entity names, limiting their standalone effectiveness in EL. With the observation that LLMs are more adept at context generation instead of EL execution, we introduce LLM-Augmented Entity Linking (LLMAEL), the first framework to enhance specialized EL models with LLM data augmentation. LLMAEL leverages off-the-shelf, tuning-free LLMs as context augmenters, generating entity descriptions to serve as additional input for specialized EL models. Experiments show that LLMAEL sets new state-of-the-art results across 6 widely adopted EL benchmarks: compared to prior methods that integrate tuning-free LLMs into EL, LLMAEL achieves an absolute 8.9% gain in EL accuracy. We release our code and datasets.

Motivation & Objective

• To address the poor performance of traditional EL models on long-tail entities due to limited training data and knowledge.
• To overcome the hallucination problem in LLMs when directly generating entity IDs for entity linking.
• To develop a cost-effective, plug-and-play method that leverages LLMs for context augmentation while preserving the precision of specialized EL models.
• To explore whether LLMs can be used more effectively as context augmenters rather than direct EL executors or re-rankers.
• To release code, datasets, and fine-tuned models to support future research in LLM-augmented entity linking.

Proposed method

• LLMaEL uses LLMs to generate mention-centered descriptions as additional context, enriching the input for traditional EL models.
• The method employs a three-stage pipeline: (1) LLM-based context augmentation, (2) data fusion of augmented context with original input, and (3) entity linking via a standard EL model.
• The LLM is prompted to generate concise, factual descriptions centered on the mention, drawing from world knowledge and context information.
• The EL model processes the augmented context to predict the correct entity ID, minimizing hallucination risks by offloading ID prediction to a task-specific model.
• Optional enhancements include context-joining and ensemble strategies to further improve performance.
• Fine-tuning is applied only to the LLM component for context generation, not to the EL model, reducing training costs.

Experimental results

Research questions

• RQ1 Can LLMs effectively improve entity linking performance by generating enriched context without direct entity ID prediction?
• RQ2 Does LLM-based context augmentation lead to significant gains in accuracy, especially for long-tail entities?
• RQ3 Is using LLMs as context augmenters more effective than using them directly for entity linking or as re-rankers?
• RQ4 How does the performance of LLMaEL compare to strong baselines across diverse entity linking benchmarks?
• RQ5 Can a plug-and-play approach achieve state-of-the-art results without fine-tuning the EL model or requiring access to proprietary LLM weights?

Key findings

• Vanilla LLMaEL outperforms baseline EL models on 5 out of 6 standard datasets, demonstrating the effectiveness of context augmentation.
• Fine-tuned LLMaEL achieves new state-of-the-art results across all 6 benchmarks, with an average accuracy gain of 1.21%.
• The method significantly improves performance for long-tail entities with frequencies between 10⁻⁷ and 10⁻⁶, validating its effectiveness for rare entities.
• LLM-based re-ranking strategies (e.g., Re-rank-100, Re-rank-10) underperform both direct LLM execution and LLMaEL, indicating that context augmentation is more effective than re-ranking.
• The use of context-joining and ensemble techniques further boosts performance, showing the flexibility and extensibility of the LLMaEL framework.
• LLMaEL avoids hallucination by delegating entity ID prediction to the EL model, while leveraging LLMs solely for knowledge-rich context generation.