[Paper Review] Exploring the Use of Text Classification in the Legal Domain

Octavia-Maria Şulea, Marcos Zampieri+4 more|arXiv (Cornell University)|Oct 25, 2017

Artificial Intelligence in Law14 references105 citations

TL;DR

The paper studies applying text classification to French Supreme Court rulings, using SVM ensembles on masked case descriptions to predict law area, ruling type, and ruling date, with strong performance gains over prior work.

ABSTRACT

In this paper, we investigate the application of text classification methods to support law professionals. We present several experiments applying machine learning techniques to predict with high accuracy the ruling of the French Supreme Court and the law area to which a case belongs to. We also investigate the influence of the time period in which a ruling was made on the form of the case description and the extent to which we need to mask information in a full case ruling to automatically obtain training and test data that resembles case descriptions. We developed a mean probability ensemble system combining the output of multiple SVM classifiers. We report results of 98% average F1 score in predicting a case ruling, 96% F1 score for predicting the law area of a case, and 87.07% F1 score on estimating the date of a ruling.

Motivation & Objective

• Motivate automation to support legal professionals in tasks like legal research and case preparation.
• Investigate whether text classification can predict law area, ruling, and ruling date from case descriptions.
• Assess how masking affects training data realism and model performance.
• Compare ensemble results to a previous baseline and analyze which aspects of the data influence predictions.

Proposed method

• Use a mean probability ensemble of SVM classifiers with word unigrams and bigrams as features.
• Train and evaluate with stratified 10-fold cross-validation to handle class imbalance.
• Mask target labels in training/test data to simulate realistic drafts available to lawyers.
• Preprocess data by removing give-away references and, for certain tasks, eliminating label words from texts.
• Compare ensemble results to Şulea et al. (2017) baselines using LIBLINEAR SVM on bag-of-words features.

Experimental results

Research questions

• RQ1 Can text classification predict the law area of a case from its description with masked targets?
• RQ2 Can an ensemble of SVM classifiers accurately predict court rulings from a case description under realistic masking?
• RQ3 Can temporal text classification estimate the decade or time span of a ruling from its description?
• RQ4 How does masking affect model performance compared to previous approaches?

Key findings

• Ensemble achieves 96.8% average precision/recall and 96.8% accuracy for law area prediction (8 classes).
• Ensemble achieves 98.6% F1/Precision and 98.6% accuracy for 6-class ruling prediction; for 8-class ruling prediction, F1 of 95.8% and accuracy 96.2%.
• Ensemble achieves 87.0% F1 and 87.0% accuracy for temporal prediction (7 classes).
• Across tasks, the mean probability ensemble outperforms the Şulea et al. (2017) baselines on the same dataset.
• The confusion analysis shows non-lieu and annulation are among the more challenging ruling classes due to fewer examples.