[Paper Review] Identification of Rhetorical Roles of Sentences in Indian Legal Judgments

Paheli Bhattacharya, Shounak Paul+3 more|arXiv (Cornell University)|Nov 13, 2019

Artificial Intelligence in Law37 citations

TL;DR

The paper studies labeling sentences with seven rhetorical roles in Indian Supreme Court judgments, collecting a gold-standard annotated corpus and showing deep neural models outperform handcrafted-feature CRF baselines for 7-class sequence labeling.

ABSTRACT

Automatically understanding the rhetorical roles of sentences in a legal case judgement is an important problem to solve, since it can help in several downstream tasks like summarization of legal judgments, legal search, and so on. The task is challenging since legal case documents are usually not well-structured, and these rhetorical roles may be subjective (as evident from variation of opinions between legal experts). In this paper, we address this task for judgments from the Supreme Court of India. We label sentences in 50 documents using multiple human annotators, and perform an extensive analysis of the human-assigned labels. We also attempt automatic identification of the rhetorical roles of sentences. While prior approaches towards this task used Conditional Random Fields over manually handcrafted features, we explore the use of deep neural models which do not require hand-crafting of features. Experiments show that neural models perform much better in this task than baseline methods which use handcrafted features.

Motivation & Objective

• Create a high-quality gold standard corpus annotating sentence-level rhetorical roles in Indian Supreme Court judgments.
• Investigate inter-annotator agreement and subjectivity among legal experts for seven rhetorical roles.
• Develop and evaluate neural models for automatic 7-class sentence labeling in legal texts.

Proposed method

• Annotate 50 Supreme Court judgments across five domains with seven rhetorical labels using three annotators.
• Analyze inter-annotator agreement with strict, lenient, and average F-scores using GATE annotations.
• Treat sentence labeling as a 7-class sequence labeling problem.
• Implement baseline CRF models with handcrafted features drawn from prior work.
• Develop two neural architectures: (i) Hierarchical BiLSTM classifier, (ii) Hierarchical BiLSTM-CRF classifier.
• Experiment with random and pretrained sentence embeddings (trained on a large set of legal documents).

Experimental results

Research questions

• RQ1 Can deep neural models automatically identify sentence-level rhetorical roles in Indian legal judgments without hand-crafted features?
• RQ2 What is the extent of agreement among human annotators on rhetorical role labeling, and how does this compare to model agreement?
• RQ3 Do pretrained domain-specific embeddings improve performance over random initializations for this task?
• RQ4 Does adding a CRF layer atop BiLSTM yield substantial gains given long legal document sequences?
• RQ5 How well do models generalize across different legal domains within Indian judiciary?

Key findings

• Neural models significantly outperform handcrafted-feature CRF baselines in macro F-score (e.g., up to ~0.82–0.82+ for Hier-BiLSTM-CRF with pretrained embeddings).
• Pretrained legal-domain embeddings markedly improve performance over random initializations for both neural models.
• Hier-BiLSTM-CRF offers only modest gains over Hier-BiLSTM in this setting due to long sequences and limited data.
• The best model maintains consistent performance across five domains (0.82–0.86 macro F-score range) except Labour & Industrial law, where IAA was lower.
• There is substantial human annotator subjectivity for certain label pairs (e.g., Ratio vs. PRE/Fac), mirrored by model confusion.
• Agreement analyses show model performance aligns with human disagreement patterns, indicating inherent subjectivity in several label pairs.