[논문 리뷰] Identification of Rhetorical Roles of Sentences in Indian Legal Judgments

Paheli Bhattacharya, Shounak Paul외 3명|arXiv (Cornell University)|2019. 11. 13.

Artificial Intelligence in Law인용 수 37

한 줄 요약

이 논문은 인도 대법원 판결에서 7가지 수사적 역할로 문장을 라벨링하는 것을 연구하고, 골드 표준 주석 코퍼스를 수집하며, 7-class 시퀀스 라벨링에 대해 깊은 신경망 모델이 수작업 특징 CRF 기반선보다 우수하다는 것을 보인다.

ABSTRACT

Automatically understanding the rhetorical roles of sentences in a legal case judgement is an important problem to solve, since it can help in several downstream tasks like summarization of legal judgments, legal search, and so on. The task is challenging since legal case documents are usually not well-structured, and these rhetorical roles may be subjective (as evident from variation of opinions between legal experts). In this paper, we address this task for judgments from the Supreme Court of India. We label sentences in 50 documents using multiple human annotators, and perform an extensive analysis of the human-assigned labels. We also attempt automatic identification of the rhetorical roles of sentences. While prior approaches towards this task used Conditional Random Fields over manually handcrafted features, we explore the use of deep neural models which do not require hand-crafting of features. Experiments show that neural models perform much better in this task than baseline methods which use handcrafted features.

연구 동기 및 목표

• Create a high-quality gold standard corpus annotating sentence-level rhetorical roles in Indian Supreme Court judgments.
• Investigate inter-annotator agreement and subjectivity among legal experts for seven rhetorical roles.
• Develop and evaluate neural models for automatic 7-class sentence labeling in legal texts.

제안 방법

• Annotate 50 Supreme Court judgments across five domains with seven rhetorical labels using three annotators.
• Analyze inter-annotator agreement with strict, lenient, and average F-scores using GATE annotations.
• Treat sentence labeling as a 7-class sequence labeling problem.
• Implement baseline CRF models with handcrafted features drawn from prior work.
• Develop two neural architectures: (i) Hierarchical BiLSTM classifier, (ii) Hierarchical BiLSTM-CRF classifier.
• Experiment with random and pretrained sentence embeddings (trained on a large set of legal documents).

실험 결과

연구 질문

• RQ1 Can deep neural models automatically identify sentence-level rhetorical roles in Indian legal judgments without hand-crafted features?
• RQ2 What is the extent of agreement among human annotators on rhetorical role labeling, and how does this compare to model agreement?
• RQ3 Do pretrained domain-specific embeddings improve performance over random initializations for this task?
• RQ4 Does adding a CRF layer atop BiLSTM yield substantial gains given long legal document sequences?
• RQ5 How well do models generalize across different legal domains within Indian judiciary?

주요 결과

• Neural models significantly outperform handcrafted-feature CRF baselines in macro F-score (e.g., up to ~0.82–0.82+ for Hier-BiLSTM-CRF with pretrained embeddings).
• Pretrained legal-domain embeddings markedly improve performance over random initializations for both neural models.
• Hier-BiLSTM-CRF offers only modest gains over Hier-BiLSTM in this setting due to long sequences and limited data.
• The best model maintains consistent performance across five domains (0.82–0.86 macro F-score range) except Labour & Industrial law, where IAA was lower.
• There is substantial human annotator subjectivity for certain label pairs (e.g., Ratio vs. PRE/Fac), mirrored by model confusion.
• Agreement analyses show model performance aligns with human disagreement patterns, indicating inherent subjectivity in several label pairs.