[Paper Review] Instance-Based Learning of Span Representations: A Case Study through Named Entity Recognition

Hiroki Ouchi, Jun Suzuki+5 more|arXiv (Cornell University)|Apr 29, 2020

Topic Modeling46 references4 citations

TL;DR

This paper proposes an instance-based learning method for span representations in named entity recognition (NER), where each span is classified based on its most similar training spans, enabling highly interpretable predictions. The method achieves competitive performance with state-of-the-art classifier-based models while providing clear rationales through nearest-neighbor retrieval, demonstrating high interpretability without sacrificing accuracy.

ABSTRACT

Interpretable rationales for model predictions play a critical role in practical applications. In this study, we develop models possessing interpretable inference process for structured prediction. Specifically, we present a method of instance-based learning that learns similarities between spans. At inference time, each span is assigned a class label based on its similar spans in the training set, where it is easy to understand how much each training instance contributes to the predictions. Through empirical analysis on named entity recognition, we demonstrate that our method enables to build models that have high interpretability without sacrificing performance.

Motivation & Objective

• To develop a model with high interpretability by leveraging instance-based learning for span representations in structured prediction tasks.
• To address the black-box nature of neural models in NER by enabling human-understandable rationales through similarity-based inference.
• To overcome the inconsistency in label prediction common in token-wise classification (e.g., BIO tagging) by using span-wise classification.
• To evaluate whether instance-based learning of span representations can match the performance of classifier-based models while offering superior interpretability.
• To investigate the generalizability of the method beyond NER to other short-span classification tasks.

Proposed method

• The method learns a feature space where spans with the same class label are embedded close to each other, using pre-trained embeddings such as GloVe or BERT.
• At inference time, each test span is assigned a label based on the majority class of its k-nearest neighbors in the training set, using cosine similarity in the span representation space.
• Span representations are derived from contextualized embeddings (e.g., BERT) or static embeddings (e.g., GloVe), enabling rich semantic and syntactic modeling.
• The model avoids label inconsistency issues common in BIO tagging by classifying entire spans as units rather than individual tokens.
• The approach supports span-wise prediction, naturally handling nested and overlapping entities without complex decoding.
• The method is evaluated on CoNLL-2003 and CoNLL-2000 datasets, with ablation studies on training set size and qualitative analysis of nearest neighbors.

Experimental results

Research questions

• RQ1 Can instance-based learning of span representations achieve competitive performance with state-of-the-art classifier-based models in NER?
• RQ2 Does the instance-based method provide more interpretable predictions than traditional neural models, with clear rationales based on training neighbors?
• RQ3 How does the model perform on nested and overlapping entity recognition, where token-level labeling often fails?
• RQ4 Can the learned span representation space be generalized to other short-span classification tasks beyond NER?
• RQ5 What are the failure modes of the instance-based model, and can they be diagnosed through neighbor analysis?

Key findings

• The instance-based span model achieved F1 scores of 94.96 ± 0.06 on CoNLL-2003 using GloVe and 96.24 ± 0.04 using BERT, matching or slightly exceeding the classifier-based model’s performance.
• The model demonstrated consistent performance across training set sizes, with minimal degradation even at 1/8 of the full training data, indicating strong generalization.
• Qualitative analysis showed that the instance-based model consistently retrieved semantically similar entities (e.g., cricketers for 'Tom Moody'), while the classifier-based model retrieved non-entity or semantically inconsistent spans.
• Error analysis revealed that the model often confused organization names (ORG) with location names (LOC), and this could be diagnosed by examining the nearest neighbors, confirming the interpretability of the rationale.
• The method generalized well to syntactic chunking on CoNLL-2000, achieving competitive F1 scores (94.96 ± 0.06 with GloVe, 96.24 ± 0.04 with BERT), suggesting broader applicability to short-span classification tasks.
• The model’s inference process is inherently interpretable: predictions are explained by the most similar training spans, enabling users to trace the rationale for each label assignment.