[Paper Review] Legal Summarisation through LLMs: The PRODIGIT Project

Thiago Raulino Dal Pont, Federico Galli+4 more|arXiv (Cornell University)|Aug 4, 2023

Natural Language Processing Techniques4 citations

TL;DR

This paper presents the PRODIGIT project, which leverages large language models—particularly GPT-4—to generate high-quality, issue-based summaries of Italian tax court decisions. The approach outperformed traditional NLP tools in expert evaluation, demonstrating that prompt-engineered LLMs can effectively extract legal issues, decision criteria, and keywords without fine-tuning, paving the way for a publicly accessible, searchable case law database.

ABSTRACT

We present some initial results of a large-scale Italian project called PRODIGIT which aims to support tax judges and lawyers through digital technology, focusing on AI. We have focused on generation of summaries of judicial decisions and on the extraction of related information, such as the identification of legal issues and decision-making criteria, and the specification of keywords. To this end, we have deployed and evaluated different tools and approaches to extractive and abstractive summarisation. We have applied LLMs, and particularly on GPT4, which has enabled us to obtain results that proved satisfactory, according to an evaluation by expert tax judges and lawyers. On this basis, a prototype application is being built which will be made publicly available.

Motivation & Objective

• To develop an automated system for summarising Italian tax court decisions to support legal professionals.
• To evaluate the effectiveness of large language models (LLMs) like GPT-4 in legal summarisation compared to classical NLP methods.
• To extract legally meaningful information such as legal issues, decision criteria, and keywords from judicial opinions.
• To create a prototype application for expert review and public access to AI-generated summaries of tax law cases.
• To assess the feasibility of using issue-based summarisation for indexing and searching case law databases.

Proposed method

• Employed a hybrid approach combining extractive and abstractive summarisation techniques, using both classical statistical NLP tools and state-of-the-art LLMs.
• Designed custom prompts for GPT-4 to generate issue-based summaries, focusing on legal issues and corresponding decision criteria.
• Implemented keyword and textual fragment extraction using prompt engineering to ensure legal relevance and precision.
• Conducted iterative refinement of prompts through feedback from expert tax lawyers before formal evaluation.
• Built a conceptual graph using extracted legal issues and keywords to enable semantic indexing of the case law database.
• Evaluated outputs via a two-step process: expert legal review and ethical committee-reviewed questionnaires to assess linguistic quality, completeness, and correctness.

Figure 1: Evaluation score for extractive summaries

Experimental results

Research questions

• RQ1 Can large language models like GPT-4 generate legally meaningful, high-quality summaries of Italian tax court decisions without fine-tuning?
• RQ2 How does LLM-based abstractive summarisation compare to classical NLP tools in terms of accuracy and usability for legal professionals?
• RQ3 To what extent can issue-based summarisation improve the discoverability and accessibility of case law through semantic indexing?
• RQ4 What role does prompt engineering play in achieving reliable and consistent legal summarisation with LLMs?
• RQ5 How can expert human supervision enhance the reliability and refinement of AI-generated legal summaries?

Key findings

• GPT-4-generated issue-based summaries were rated as superior in linguistic quality, completeness, and correctness by expert tax judges and lawyers.
• The most effective summarisation approach was issue-based summarisation, which explicitly outlines legal issues and corresponding decision criteria.
• Prompt engineering alone—without model fine-tuning—produced results that met expert standards, indicating strong zero-shot performance of LLMs in legal summarisation.
• Expert evaluation confirmed that LLM-based summaries outperformed classical NLP tools in all assessed dimensions, including relevance and coherence.
• The project successfully demonstrated that LLMs can extract meaningful legal issues and keywords, enabling the construction of a conceptual graph for case law indexing.
• A prototype application is being developed to allow legal experts to review and refine AI-generated summaries, ensuring quality and reliability before public deployment.

Figure 2: Evaluation score for abstractive summaries (first round)