[Paper Review] Towards automating Numerical Consistency Checks in Financial Reports

Lars Hillebrand, Tobias Deußer+5 more|arXiv (Cornell University)|Nov 11, 2022

Stock Market Forecasting Methods4 citations

TL;DR

KPI-Check is a BERT-based system that automates numerical consistency checks in German financial reports by identifying semantically equivalent key performance indicators (KPIs) across unstructured text and structured tables (balance sheet and P&L). It combines financial NER with a contrastive autoencoder for text pair classification, achieving 73.00% micro F1 on a held-out test set, significantly reducing manual auditing effort.

ABSTRACT

We introduce KPI-Check, a novel system that automatically identifies and cross-checks semantically equivalent key performance indicators (KPIs), e.g. "revenue" or "total costs", in real-world German financial reports. It combines a financial named entity and relation extraction module with a BERT-based filtering and text pair classification component to extract KPIs from unstructured sentences before linking them to synonymous occurrences in the balance sheet and profit & loss statement. The tool achieves a high matching performance of $73.00$% micro F$_1$ on a hold out test set and is currently being deployed for a globally operating major auditing firm to assist the auditing procedure of financial statements.

Motivation & Objective

• To reduce manual auditing effort in verifying numerical consistency of KPIs across financial reports.
• To address frequent numeric inconsistencies in corporate disclosures due to copy-pasting and multi-author processes.
• To improve investor trust and firm reputation by detecting KPI mismatches early.
• To automate the cross-checking of KPIs between textual descriptions and financial tables (balance sheet and P&L).
• To develop a scalable, deployable system for real-world auditing workflows.

Proposed method

• Uses KPI-BERT, a financial-domain-specific BERT model, to jointly extract KPIs and their numeric values from unstructured sentences.
• Applies a joint sentence- and table-encoding module using BERT to identify relevant sentence-table pairs for KPI matching.
• Employs a contrastive autoencoder (CAE)-based text pair classification head to determine semantic equivalence between KPI mentions.
• Integrates a filtering module to reduce class imbalance in training data by eliminating irrelevant KPI pairs.
• Leverages automated number matching for weak supervision during model training, with plans to enhance with manual annotations.
• Validates KPI pairs by comparing monetary values, accounting for scale (e.g., million) and currency units.

Experimental results

Research questions

• RQ1 Can a deep learning model accurately identify semantically equivalent KPIs across unstructured text and structured financial tables?
• RQ2 How effective is a BERT-based approach in detecting numerical inconsistencies in real-world German financial reports?
• RQ3 To what extent does automated weak supervision via number matching support reliable KPI matching?
• RQ4 How does filtering reduce data imbalance without sacrificing positive KPI pair recall?
• RQ5 Can the system generalize to cases with rounding errors or unit inconsistencies that confuse simple number matching?

Key findings

• KPI-Check achieves a micro F1 score of 73.00% on a held-out test set, demonstrating strong performance in detecting semantically equivalent KPIs.
• The system successfully identifies KPIs with rounding and unit inconsistencies (e.g., 'e' vs. 'Te') that would be misclassified by simple number matching.
• It correctly detects cases where current and prior year values accidentally match but the KPIs are unrelated, avoiding false positives.
• The filtering module reduces the negative-to-positive KPI pair imbalance from 395:1 to 46:1, significantly improving training efficiency.
• Despite relying on automatically generated annotations, the model generalizes well, as shown in qualitative test cases with imperfect labels.
• The system is currently being deployed at a major global auditing firm, with early user tests indicating substantial efficiency gains.