[Paper Review] Claim Automation using Large Language Model

Zhengda Mo, Zhiyu Quan+2 more|arXiv (Cornell University)|Feb 18, 2026

Artificial Intelligence in Healthcare and Education0 citations

TL;DR

The paper proposes a locally deployed, governance-aware LLM pipeline fine-tuned with LoRA to generate structured corrective-action recommendations from warranty claim narratives, outperforming generic LLMs in domain alignment.

ABSTRACT

While Large Language Models (LLMs) have achieved strong performance on general-purpose language tasks, their deployment in regulated and data-sensitive domains, including insurance, remains limited. Leveraging millions of historical warranty claims, we propose a locally deployed governance-aware language modeling component that generates structured corrective-action recommendations from unstructured claim narratives. We fine-tune pretrained LLMs using Low-Rank Adaptation (LoRA), scoping the model to an initial decision module within the claim processing pipeline to speed up claim adjusters' decisions. We assess this module using a multi-dimensional evaluation framework that combines automated semantic similarity metrics with human evaluation, enabling a rigorous examination of both practical utility and predictive accuracy. Our results show that domain-specific fine-tuning substantially outperforms commercial general-purpose and prompt-based LLMs, with approximately 80% of the evaluated cases achieving near-identical matches to ground-truth corrective actions. Overall, this study provides both theoretical and empirical evidence to prove that domain-adaptive fine-tuning can align model output distributions more closely with real-world operational data, demonstrating its promise as a reliable and governable building block for insurance applications.

Motivation & Objective

• Integrate unstructured claim narratives into an actionable, structured correction output within an actuarial workflow.
• Develop a governance-aware, locally deployed LLM framework to address data sensitivity and regulatory constraints.
• Demonstrate that domain-specific fine-tuning reshapes output distribution to align with real-world claim-handling practices.
• Provide a multi-dimensional evaluation framework combining automated semantic metrics with human assessment.

Proposed method

• Use a decoder-only Transformer (DeepSeek-R1-Distill-Llama-8B) deployed on-premises to ensure data governance.
• Fine-tune with Low-Rank Adaptation (LoRA) by inserting low-rank adapters into selected projections of Transformer blocks.
• Train with a masked autoregressive objective that only optimizes the corrective-action segment (Complaint–Cause as input; Correction as output).
• Apply Rotary Position Embedding (RoPE) for position information within attention mechanisms.
• Adopt a multi-stage normalization (RMSNorm) and SwiGLU activations within a PreNorm Transformer framework.
• Evaluate with a multi-dimensional framework that combines semantic similarity metrics and structured-output validation, plus human-in-the-loop feedback.

Figure 1 : Overview of the token-level generation architecture used for claim automation.

Experimental results

Research questions

• RQ1 Can domain-adapted, locally deployed LLMs generate structured corrective-action outputs that match real-world claim-handling practices?
• RQ2 Does LoRA fine-tuning on domain-specific warranty data outperform general-purpose LLMs in terms of output format, semantics, and stability?
• RQ3 How does a modular intermediate task (outputting corrective actions) influence governance, transparency, and auditability in claim workflows?
• RQ4 What evaluation framework best captures the practical utility and predictive accuracy of language-based claim automation?
• RQ5 What is the impact of domain alignment on output distribution relative to observed claim processes?

Key findings

• Domain-specific fine-tuning substantially improves performance over commercial general-purpose and prompt-based LLMs.
• Approximately 80% of evaluated cases achieved near-identical matches to ground-truth corrective actions.
• Local deployment with governance constraints reduces data privacy and regulatory risks while enabling reproducibility and auditability.
• LoRA adaptation to the DeepSeek-R1 model reshapes output distributions to align with real-world claim-handling practices.
• A multi-dimensional evaluation approach effectively assesses structural validity, semantic alignment, and distributional consistency of outputs.

Figure 2 : LoRA adaptation applies to a single projection matrix. The original weight matrix $W_{\mathrm{frozen}}\in\mathbb{R}^{d_{\mathrm{out}}\times d_{\mathrm{in}}}$ remains unchanged, while trainable matrices $A\in\mathbb{R}^{r\times d_{\mathrm{in}}}$ and $B\in\mathbb{R}^{d_{\mathrm{out}}\times