[Paper Review] KPI-EDGAR: A Novel Dataset and Accompanying Metric for Relation Extraction from Financial Documents

Tobias Deußer, Syed Musharraf Ali+5 more|arXiv (Cornell University)|Oct 17, 2022

Advanced Text Analysis Techniques4 citations

TL;DR

This paper introduces KPI-EDGAR, a new English-language dataset of manually annotated financial reports from the EDGAR database for joint named entity recognition and relation extraction of Key Performance Indicators (KPIs). It proposes a weighted F1 metric that accounts for partial correctness in entity boundaries—addressing the inherent ambiguity in KPI span annotation—thereby providing a more accurate evaluation of model performance than conventional F1 in this noisy, fuzzy-domain context.

ABSTRACT

We introduce KPI-EDGAR, a novel dataset for Joint Named Entity Recognition and Relation Extraction building on financial reports uploaded to the Electronic Data Gathering, Analysis, and Retrieval (EDGAR) system, where the main objective is to extract Key Performance Indicators (KPIs) from financial documents and link them to their numerical values and other attributes. We further provide four accompanying baselines for benchmarking potential future research. Additionally, we propose a new way of measuring the success of said extraction process by incorporating a word-level weighting scheme into the conventional F1 score to better model the inherently fuzzy borders of the entity pairs of a relation in this domain.

Motivation & Objective

• To address the lack of publicly available, manually annotated datasets for joint KPI extraction in financial documents, particularly in English.
• To overcome the limitations of conventional F1 scores in evaluating relation extraction when entity boundaries are ambiguous or inconsistently annotated.
• To enable reproducible and benchmarkable research in financial NLP by releasing a high-quality, publicly accessible dataset and evaluation metric.
• To support the development of more robust models for KPI extraction by modeling the real-world ambiguity in financial text annotation.
• To facilitate future research in financial NLP through a standardized benchmark and evaluation framework based on real-world corporate reports.

Proposed method

• Constructed KPI-EDGAR from 10-K annual reports sourced from the U.S. EDGAR database, focusing on publicly traded companies.
• Performed manual annotation of KPIs, their numerical values (cy, py), and related attributes (e.g., increase, decrease, thereof) across 1,000+ sentences.
• Designed a novel weighted F1 metric that assigns partial credit based on word-level overlap between predicted and gold entity spans, rather than requiring exact matches.
• Implemented four baseline models (including KPI-BERT) to demonstrate the utility of the dataset and metric for benchmarking.
• Calibrated the weighting scheme to reflect the degree of overlap between predicted and gold entity spans, with higher weights for longer overlapping segments.
• Evaluated the impact of the new metric by comparing conventional F1 and weighted F1 scores on baseline predictions, highlighting cases of partial correctness.

Experimental results

Research questions

• RQ1 How can a dataset for joint KPI extraction be constructed from publicly available financial reports while ensuring high annotation quality and consistency?
• RQ2 To what extent do human annotators disagree on the boundaries of KPI entities, and how does this affect model evaluation?
• RQ3 Can a weighted F1 metric that accounts for partial overlap in entity spans better reflect model performance than the conventional strict F1 in the context of financial text?
• RQ4 How do existing models perform on the new KPI-EDGAR dataset when evaluated with both conventional and weighted F1 metrics?
• RQ5 What are the implications of fuzzy entity boundaries for the design of evaluation metrics in financial NLP tasks?

Key findings

• The KPI-EDGAR dataset contains 1,000+ annotated sentences from real 10-K reports, with detailed labeling of KPIs, their values (cy, py), and related attributes.
• Cohen’s Kappa scores revealed substantial disagreement among annotators on KPI boundaries (kappa = 0.0822), indicating high ambiguity in entity span definition.
• The proposed weighted F1 metric successfully captures partial correctness in predictions, especially in cases where only a few words are misclassified.
• Conventional F1 scores penalize models harshly for minor boundary errors, while the weighted F1 provides a more nuanced and realistic performance assessment.
• Baseline models like KPI-BERT showed improved performance interpretation when evaluated with the weighted F1, revealing previously hidden strengths in partial predictions.
• The study demonstrates that the weighted F1 metric is more robust to annotation noise and better reflects real-world model behavior in financial NLP tasks.