[Paper Review] Legal Prompt Engineering for Multilingual Legal Judgement Prediction

Dietrich Trautmann, Alina Petrova+1 more|arXiv (Cornell University)|Dec 5, 2022

Artificial Intelligence in Law31 citations

TL;DR

The paper investigates zero-shot legal prompt engineering for multilingual legal judgment prediction on long legal documents from ECHR (English) and FSCS (German, French, Italian), showing zero-shot prompting surpasses simple baselines but lags supervised state-of-the-art models.

ABSTRACT

Legal Prompt Engineering (LPE) or Legal Prompting is a process to guide and assist a large language model (LLM) with performing a natural legal language processing (NLLP) skill. Our goal is to use LPE with LLMs over long legal documents for the Legal Judgement Prediction (LJP) task. We investigate the performance of zero-shot LPE for given facts in case-texts from the European Court of Human Rights (in English) and the Federal Supreme Court of Switzerland (in German, French and Italian). Our results show that zero-shot LPE is better compared to the baselines, but it still falls short compared to current state of the art supervised approaches. Nevertheless, the results are important, since there was 1) no explicit domain-specific data used - so we show that the transfer to the legal domain is possible for general-purpose LLMs, and 2) the LLMs where directly applied without any further training or fine-tuning - which in turn saves immensely in terms of additional computational costs.

Motivation & Objective

• Demonstrate the feasibility of zero-shot Legal Prompt Engineering (LPE) for Legal Judgement Prediction (LJP) on long legal documents.
• Evaluate multilingual LJP performance on datasets from the European Court of Human Rights (English) and the Federal Supreme Court of Switzerland (German, French, Italian).
• Assess how zero-shot LPE compares to baselines and to supervised state-of-the-art results without any domain-specific fine-tuning.

Proposed method

• Use discrete, manual legal prompts to convert LJP into a natural language question task for long documents.
• Apply zero-shot prompting with multilingual LLMs (e.g., mGPT, GPT-J-6B, GPT-NeoX-20B) without data-specific training.
• Iteratively design prompts to guide model output toward binary yes/no judgments (A vs B) while handling very long inputs (up to 2048 tokens).
• Evaluate prompts on ECHR (English) and FSCS (German, French, Italian) with macro-F1, precision, recall, and other metrics."
• Analyze completion samples to understand model reasoning and reasons for misalignment with true labels.

Experimental results

Research questions

• RQ1 Can zero-shot Legal Prompt Engineering (LPE) enable Legal Judgement Prediction (LJP) on long, real-world court documents without task-specific fine-tuning?
• RQ2 How does zero-shot LPE perform across multilingual legal datasets (ECHR English; FSCS German, French, Italian) compared to baselines and to supervised models?
• RQ3 What prompt designs and output configurations (e.g., token length) yield best macro-F1 and related metrics in zero-shot LJP?
• RQ4 What qualitative characteristics do model completions exhibit (e.g., listed articles, explanations) and how do these relate to task performance?

Key findings

• Zero-shot LPE with LLMs outperforms simple baselines on macro-F1, precision, and recall for LJP but remains below supervised state-of-the-art results.
• Prompts were effective across languages (English, German, French, Italian) using multilingual LLMs such as mGPT, GPT-J-6B, and GPT-NeoX-20B.
• Long documents were truncated to 2048 tokens; output sequence length required tuning (best around 50 tokens) for best performance.
• There is variability in completions, with some outputs listing violated articles or providing explanations, but no single case matched all target explanations.
• Compared to supervised full models, zero-shot prompts show clear performance gaps but demonstrate domain transferability without additional fine-tuning or task-specific data.