[Paper Review] Predicting the Law Area and Decisions of French Supreme Court Cases

Octavia-Maria Şulea, Marcos Zampieri+2 more|arXiv (Cornell University)|Jan 1, 2017

Artificial Intelligence in Law17 references23 citations

TL;DR

This paper proposes a text classification approach using linear SVM with bag-of-words features to predict the law area, ruling outcome, and temporal period of French Supreme Court cases. It achieves 96% F1 score in ruling prediction and 90% in law area classification, while introducing a novel masking strategy to simulate real-world test scenarios where target labels are hidden from case descriptions.

ABSTRACT

In this paper, we investigate the application of text classification methods to predict the law area and the decision of cases judged by the French Supreme Court. We also investigate the influence of the time period in which a ruling was made over the textual form of the case description and the extent to which it is necessary to mask the judge's motivation for a ruling to emulate a real-world test scenario. We report results of 96% f1 score in predicting a case ruling, 90% f1 score in predicting the law area of a case, and 75.9% f1 score in estimating the time span when a ruling has been issued using a linear Support Vector Machine (SVM) classifier trained on lexical features.

Motivation & Objective

• To investigate text classification methods for predicting legal outcomes and law areas in French Supreme Court rulings.
• To evaluate the impact of temporal variation on the linguistic form of case descriptions.
• To simulate realistic deployment scenarios by masking target prediction cues in case descriptions.
• To assess the feasibility of predicting case dates from textual features alone.
• To demonstrate the robustness of lexical features in legal text classification despite label masking.

Proposed method

• Trained a linear Support Vector Machine (SVM) classifier on bag-of-words (BOW) and bigram features extracted from case descriptions.
• Applied feature masking based on ranking salient words to hide mentions of law area, ruling, and time period, simulating real-world test conditions.
• Used TF-IDF weighting and information gain for feature selection to improve model generalization.
• Evaluated performance on three tasks: ruling prediction, law area classification, and temporal classification into 7- or 14-year intervals.
• Tested type-token ratio as a lexical richness feature to assess its contribution to temporal prediction.
• Compared results against random baseline and standard baselines to validate model effectiveness.

Experimental results

Research questions

• RQ1 Can text classification models accurately predict the ruling outcome of French Supreme Court cases using only the case description?
• RQ2 To what extent does the linguistic form of case descriptions vary across different time periods in French legal texts?
• RQ3 How effective is a linear SVM with BOW features in predicting the law area of a case?
• RQ4 Can temporal classification of rulings be achieved using only lexical features, and how does performance vary with class granularity?
• RQ5 Does masking explicit mentions of target labels (e.g., 'cassation', '2005') significantly increase the difficulty of the prediction task?

Key findings

• The linear SVM model achieved a 96% F1 score in predicting the ruling outcome of French Supreme Court cases using masked case descriptions.
• The model reached 90% F1 score in classifying the law area of a case, demonstrating strong performance on legal text categorization.
• Temporal classification into 14-year intervals achieved a 73.9% F1 score with bigram features, significantly outperforming the 19.1% baseline.
• The type-token ratio feature alone reached 43% F1, but did not improve performance when combined with BOW features.
• The masking strategy successfully simulated realistic test conditions, confirming that predictions rely on formulaic expressions rather than factual case-specific details.
• The results suggest that French Supreme Court rulings are highly predictable due to standardized language, which may contribute to the high performance observed.