[Paper Review] E-NER -- An Annotated Named Entity Recognition Corpus of Legal Text

Ting Wai Terence Au, Ingemar J. Cox+1 more|arXiv (Cornell University)|Dec 19, 2022

Topic Modeling4 citations

TL;DR

This paper introduces E-NER, a publicly available named entity recognition (NER) corpus of 400,000+ tokens from U.S. Securities and Exchange Commission (SEC) company filings, specifically designed for legal text. Experiments show that NER models pre-trained on general-domain CoNLL-2003 data suffer significant performance degradation—F1-scores drop by 29.4% to 60.4%—when applied to legal text, underscoring the necessity of domain-specific training data for legal NER tasks.

ABSTRACT

Identifying named entities such as a person, location or organization, in documents can highlight key information to readers. Training Named Entity Recognition (NER) models requires an annotated data set, which can be a time-consuming labour-intensive task. Nevertheless, there are publicly available NER data sets for general English. Recently there has been interest in developing NER for legal text. However, prior work and experimental results reported here indicate that there is a significant degradation in performance when NER methods trained on a general English data set are applied to legal text. We describe a publicly available legal NER data set, called E-NER, based on legal company filings available from the US Securities and Exchange Commission's EDGAR data set. Training a number of different NER algorithms on the general English CoNLL-2003 corpus but testing on our test collection confirmed significant degradations in accuracy, as measured by the F1-score, of between 29.4\% and 60.4\%, compared to training and testing on the E-NER collection.

Motivation & Objective

• To address the lack of publicly available, high-quality NER datasets for legal text, particularly in the U.S. legal domain.
• To evaluate the performance degradation of general-domain NER models when applied to legal documents.
• To demonstrate the necessity of domain-specific training data by comparing model performance across general and legal text distributions.
• To provide a benchmark dataset (E-NER) for training and evaluating legal NER systems.
• To support future research in legal NLP by enabling systematic evaluation of transfer learning and domain adaptation in legal text.

Proposed method

• The E-NER dataset was constructed from 52 U.S. SEC EDGAR filings, manually annotated for five named entity types: person, organization, location, monetary value, and date.
• The dataset contains over 400,000 tokens and approximately 9,000 named entities, with a train/validation/test split of 70%/10%/20%.
• Four NER models were evaluated: a dictionary baseline, Hidden Markov Model (HMM), Conditional Random Field (CRF), and BERT.
• Models were trained and tested on three combinations: CoNLL-2003 only, CoNLL-2003 then tested on E-NER, and E-NER only.
• For each model, micro-F1 scores were computed to compare performance across data distributions.
• Fine-tuning of BERT was performed using Hugging Face’s transformers library on the E-NER dataset.

Experimental results

Research questions

• RQ1 To what extent does training a general-domain NER model on CoNLL-2003 degrade performance when tested on legal text from SEC filings?
• RQ2 How does the performance of HMM, CRF, and BERT models differ when trained on general English versus legal English?
• RQ3 What is the impact of domain shift on named entity recognition in legal documents compared to general-domain text?
• RQ4 How does the size and composition of the E-NER dataset compare to standard NER benchmarks like CoNLL-2003 in terms of token count and named entity diversity?
• RQ5 Can domain-specific pre-training or fine-tuning significantly improve NER performance on legal text?

Key findings

• Training on CoNLL-2003 and testing on E-NER resulted in F1-score degradation ranging from 29.4% to 60.4% across different models, indicating significant domain shift.
• The dictionary baseline and HMM models showed worse performance on legal text than on general English, with F1-scores dropping to as low as 13.6%.
• CRF and BERT models achieved higher F1-scores on E-NER (up to 96.1%) than on CoNLL-2003 (up to 90.5%), suggesting better generalization to legal text when fine-tuned on domain-specific data.
• The average sentence length in legal text (34.5 words) is more than double that of general English (13.7 words), and legal named entities are longer (2.68 tokens vs. 1.45 tokens).
• The E-NER dataset contains approximately 9,000 named entities, which is only about 25% of the number in the CoNLL-2003 corpus, reflecting differences in lexical and syntactic complexity.
• The results confirm that domain-specific training data is essential for high-performance legal NER, as general-domain models fail to generalize effectively to legal text.