[Paper Review] Multimodal Document Analytics for Banking Process Automation

Christopher Gerling, Stefan Lessmann|arXiv (Cornell University)|Jul 21, 2023

Robotic Process Automation Applications4 citations

TL;DR

This paper proposes using the multimodal LayoutXLM model to automate document analytics in banking by integrating text, layout, and visual features, significantly outperforming unimodal models like BERT and GPT. It demonstrates that LayoutXLM achieves over 75% of its maximum F1 score with just 30% of the training data, proving high efficiency and low data dependency for real-world banking applications.

ABSTRACT

Traditional banks face increasing competition from FinTechs in the rapidly evolving financial ecosystem. Raising operational efficiency is vital to address this challenge. Our study aims to improve the efficiency of document-intensive business processes in banking. To that end, we first review the landscape of business documents in the retail segment. Banking documents often contain text, layout, and visuals, suggesting that document analytics and process automation require more than plain natural language processing (NLP). To verify this and assess the incremental value of visual cues when processing business documents, we compare a recently proposed multimodal model called LayoutXLM to powerful text classifiers (e.g., BERT) and large language models (e.g., GPT) in a case study related to processing company register extracts. The results confirm that incorporating layout information in a model substantially increases its performance. Interestingly, we also observed that more than 75% of the best model performance (in terms of the F1 score) can be achieved with as little as 30% of the training data. This shows that the demand for data labeled data to set up a multi-modal model can be moderate, which simplifies real-world applications of multimodal document analytics. Our study also sheds light on more specific practices in the scope of calibrating a multimodal banking document classifier, including the need for fine-tuning. In sum, the paper contributes original empirical evidence on the effectiveness and efficiency of multi-model models for document processing in the banking business and offers practical guidance on how to unlock this potential in day-to-day operations.

Motivation & Objective

• To analyze the diversity and multimodal nature of banking documents in retail banking.
• To evaluate the incremental value of visual and layout features in document analytics beyond pure text processing.
• To benchmark LayoutXLM against unimodal models like BERT and GPT in a real-world banking document classification task.
• To assess data efficiency and robustness to class imbalance in training multimodal models for banking.
• To provide practical guidance for deploying multimodal models in operational banking processes.

Proposed method

• The study uses LayoutXLM, a multimodal transformer model pre-trained on both text and layout information from documents.
• It fine-tunes LayoutXLM on a dataset of German company register extracts, which contain text, tables, and visual structures.
• Performance is evaluated using F1 score, with ablation studies to isolate the contribution of layout and visual features.
• The model is compared against strong baselines: BERT (text-only) and GPT-3.5 (large language model) in a zero-shot and few-shot setup.
• Data efficiency is tested by training on increasing fractions of the labeled dataset (from 10% to 100%) to measure performance gain.
• Class imbalance is addressed by analyzing model performance without manual data balancing, assessing inherent robustness.

Experimental results

Research questions

• RQ1 To what extent do layout and visual features improve document classification performance in banking documents compared to text-only models?
• RQ2 How much labeled training data is required to achieve high performance with a multimodal model like LayoutXLM in banking document analytics?
• RQ3 How does LayoutXLM perform relative to unimodal models such as BERT and GPT-3.5 in classifying complex, multimodal banking documents?
• RQ4 Can LayoutXLM effectively handle class imbalance common in banking document datasets without data augmentation or reweighting?
• RQ5 What is the practical feasibility of deploying LayoutXLM in real-world banking operations given data and fine-tuning constraints?

Key findings

• LayoutXLM achieves an F1 score of approximately 80% when trained on the full dataset, with over 75% of this performance attainable using only 30% of the training data.
• The ablation study confirms that layout and visual features contribute significantly to performance, with LayoutXLM outperforming both BERT and GPT-3.5 in the same task.
• LayoutXLM demonstrates robustness to class imbalance, maintaining strong performance without requiring manual data balancing or preprocessing.
• The model’s performance increases sharply with data size up to 200–300 pages, after which gains slow, indicating diminishing returns beyond a moderate data volume.
• Fine-tuning is necessary even for large language models like GPT-3.5 in banking contexts, suggesting that zero-shot inference is insufficient for high-accuracy document analytics.
• LayoutXLM’s multimodal design enables it to extract information from complex documents with tables, text, and visual elements more effectively than unimodal models.