[Paper Review] Legal Syllogism Prompting: Teaching Large Language Models for Legal Judgment Prediction

Cong Jiang, Xiaolei Yang|arXiv (Cornell University)|Jul 17, 2023

Artificial Intelligence in Law4 citations

TL;DR

This paper introduces Legal Syllogism Prompting (LoT), a zero-shot prompting method that teaches large language models (LLMs) to predict legal judgments by structuring reasoning as a legal syllogism: major premise (law), minor premise (facts), and conclusion (judgment). On the CAIL2018 Chinese criminal case dataset, LoT outperforms both baseline models and chain-of-thought prompting in zero-shot legal judgment prediction, enhancing model explainability through structured legal reasoning without fine-tuning or labeled examples.

ABSTRACT

Legal syllogism is a form of deductive reasoning commonly used by legal professionals to analyze cases. In this paper, we propose legal syllogism prompting (LoT), a simple prompting method to teach large language models (LLMs) for legal judgment prediction. LoT teaches only that in the legal syllogism the major premise is law, the minor premise is the fact, and the conclusion is judgment. Then the models can produce a syllogism reasoning of the case and give the judgment without any learning, fine-tuning, or examples. On CAIL2018, a Chinese criminal case dataset, we performed zero-shot judgment prediction experiments with GPT-3 models. Our results show that LLMs with LoT achieve better performance than the baseline and chain of thought prompting, the state-of-art prompting method on diverse reasoning tasks. LoT enables the model to concentrate on the key information relevant to the judgment and to correctly understand the legal meaning of acts, as compared to other methods. Our method enables LLMs to predict judgment along with law articles and justification, which significantly enhances the explainability of models.

Motivation & Objective

• To address the lack of explainability in existing legal judgment prediction (LJP) models based on deep learning and fine-tuning.
• To enable large language models (LLMs) to perform zero-shot legal judgment prediction without fine-tuning or labeled examples.
• To enhance model interpretability by structuring reasoning according to legal syllogism—law, facts, and judgment—mirroring judicial reasoning.
• To improve model selectivity and sensitivity in distinguishing relevant facts and appropriate charges.
• To demonstrate that LLMs can generate legally coherent, explainable judgments using only a prompting framework.

Proposed method

• LoT introduces a prompting template that explicitly structures reasoning into three components: major premise (legal rule), minor premise (case facts), and conclusion (judgment).
• The method relies on a simple instruction such as 'Let’s think in legal syllogism' to guide the LLM into generating structured deductive reasoning.
• No fine-tuning or labeled examples are used; the model generates the syllogism directly from the input case description and pre-trained knowledge.
• The approach leverages the LLM’s inherent reasoning capability by constraining the output format to align with legal logic, improving focus on relevant facts and legal rules.
• The model is evaluated on the CAIL2018 dataset using GPT-3 in a zero-shot setting, with performance measured via standard LJP metrics.
• Post-processing may be used in future to improve legal article accuracy, though this is not part of the core method.

Experimental results

Research questions

• RQ1 Can a simple prompting method like LoT enable zero-shot legal judgment prediction in LLMs without fine-tuning or labeled examples?
• RQ2 Does structuring reasoning as a legal syllogism improve model selectivity in focusing on relevant facts and legal rules?
• RQ3 How does LoT compare to chain-of-thought prompting in terms of accuracy and explainability for legal judgment prediction?
• RQ4 Can LLMs generate legally coherent judgments with just the syllogistic structure, without explicit training on legal data?
• RQ5 To what extent does LoT enhance sensitivity in distinguishing between multiple legal charges based on a single sequence of actions?

Key findings

• LoT significantly outperforms both the baseline and chain-of-thought prompting in zero-shot legal judgment prediction on the CAIL2018 dataset.
• The model using LoT demonstrates higher selectivity by focusing only on facts relevant to the legal rule, avoiding irrelevant details common in chain-of-thought outputs.
• LoT improves sensitivity by correctly grouping multiple actions into a single legal charge (e.g., fraud) rather than over-segregating into multiple charges.
• The method enhances explainability by generating a complete syllogism, including the applicable law article and justification, which aligns with judicial reasoning practices.
• LoT enables LLMs to produce accurate judgments with full legal reasoning in a zero-shot setting, without any fine-tuning or example demonstrations.
• The results suggest that legal syllogism prompting effectively unlocks the deductive reasoning capability of LLMs for legal tasks.