[Paper Review] Evaluating Pretrained Transformer Models for Entity Linking in Task-Oriented Dialog

Sai Muralidhar Jayanthi, Varsha Embar+1 more|arXiv (Cornell University)|Dec 15, 2021

Topic Modeling8 citations

TL;DR

This paper evaluates pretrained transformer models (PTMs) for unsupervised entity linking in task-oriented dialog, focusing on their performance across syntactic, semantic, short-form, numeric, and phonetic variations. Fine-tuned PTMs, especially those optimized for text similarity, outperform standard PTMs and traditional models like BM25 in semantic and syntactic matching, but still underperform on abbreviations and speech recognition errors, with best P@1 at 53.4% across datasets.

ABSTRACT

The wide applicability of pretrained transformer models (PTMs) for natural language tasks is well demonstrated, but their ability to comprehend short phrases of text is less explored. To this end, we evaluate different PTMs from the lens of unsupervised Entity Linking in task-oriented dialog across 5 characteristics -- syntactic, semantic, short-forms, numeric and phonetic. Our results demonstrate that several of the PTMs produce sub-par results when compared to traditional techniques, albeit competitive to other neural baselines. We find that some of their shortcomings can be addressed by using PTMs fine-tuned for text-similarity tasks, which illustrate an improved ability in comprehending semantic and syntactic correspondences, as well as some improvements for short-forms, numeric and phonetic variations in entity mentions. We perform qualitative analysis to understand nuances in their predictions and discuss scope for further improvements. Code can be found at https://github.com/murali1996/el_tod

Motivation & Objective

• To assess the effectiveness of pretrained transformer models (PTMs) in unsupervised entity linking for short, low-context dialog utterances.
• To investigate how PTMs handle diverse mention variations—syntactic, semantic, short-forms, numeric, and phonetic—common in spoken language.
• To compare PTMs against traditional neural and non-neural baselines (e.g., BM25, TF-IDF) in low-resource, zero-shot settings.
• To evaluate whether fine-tuning PTMs on text-similarity tasks improves their robustness to linguistic variations in entity mentions.
• To identify failure modes and guide future development of generic, task-agnostic entity linking models for conversational AI.

Proposed method

• Curated and benchmarked multiple evaluation datasets with controlled variations across five linguistic characteristics: syntactic, semantic, short-form, numeric, and phonetic.
• Evaluated four types of PTMs: general-purpose base models (e.g., BERT, RoBERTa), distilled/reduced models (e.g., DistilBERT), text-similarity fine-tuned models (e.g., all-MiniLM-L6-v2), and quantized variants.
• Used cosine similarity between contextualized embeddings and KB entries to perform zero-shot entity linking without retraining.
• Applied dynamic quantization to reduce inference latency while maintaining performance on selected models.
• Conducted qualitative analysis on model predictions to identify systematic errors in semantic, numeric, and phonetic matching.
• Compared results against traditional systems (BM25, TF-IDF) and analyzed performance across P@1 and P@5 metrics.

Experimental results

Research questions

• RQ1 How do pretrained transformer models perform in unsupervised entity linking for short, context-poor dialog utterances compared to traditional retrieval methods?
• RQ2 To what extent do PTMs fine-tuned on text-similarity tasks improve performance on semantic and syntactic variations in entity mentions?
• RQ3 Why do PTMs still underperform on abbreviations and phonetically similar errors, and what patterns emerge in their mispredictions?
• RQ4 How do model size, quantization, and architecture choice affect inference speed and accuracy in low-resource entity linking?
• RQ5 Can fine-tuned PTMs generalize across domain-specific KBs without retraining, and what are the limitations in handling numeric and short-form variations?

Key findings

• Fine-tuned PTMs, especially those optimized for text similarity (e.g., all-MiniLM-L6-v2), significantly outperform standard PTMs in handling syntactic and semantic variations in entity mentions.
• The best-performing model achieved a P@1 score of 53.4% and P@5 of 64.0% across all datasets, indicating room for improvement despite gains.
• Traditional BM25 outperformed many PTMs in syntactic matching due to robust n-gram matching, especially with spelling variations.
• PTMs struggled with abbreviations and acronyms, with models like BERT often misclassifying entities like 'USSR' as 'Czechoslovakia' or 'PSU' as 'Football'.
• Numeric variations (e.g., '90’s' vs. '1990s') were best handled by the quantized MPNET-Q model, achieving 92.8% P@1, while BERT performed poorly due to inconsistent numeric representation understanding.
• Phonetic errors from ASR led to unpredictable predictions across all PTMs, with BM25 providing more explainable results due to its reliance on surface-level string matching.