[Paper Review] Prototype-Based Interpretability for Legal Citation Prediction

Chu Fei Luo, Rohan Bhambhoria+2 more|arXiv (Cornell University)|May 25, 2023

Artificial Intelligence in Law4 citations

TL;DR

This paper introduces a prototype-based interpretability framework for legal citation prediction (LCP), modeling lawyer reasoning by integrating precedents and statutory provisions as inputs. It achieves strong performance through a modified prototype architecture with enhanced interpretability, validated through expert feedback and ablation studies, marking the first use of prototype-based methods in high-stakes legal NLP tasks.

ABSTRACT

Deep learning has made significant progress in the past decade, and demonstrates potential to solve problems with extensive social impact. In high-stakes decision making areas such as law, experts often require interpretability for automatic systems to be utilized in practical settings. In this work, we attempt to address these requirements applied to the important problem of legal citation prediction (LCP). We design the task with parallels to the thought-process of lawyers, i.e., with reference to both precedents and legislative provisions. After initial experimental results, we refine the target citation predictions with the feedback of legal experts. Additionally, we introduce a prototype architecture to add interpretability, achieving strong performance while adhering to decision parameters used by lawyers. Our study builds on and leverages the state-of-the-art language processing models for law, while addressing vital considerations for high-stakes tasks with practical societal impact.

Motivation & Objective

• Address the lack of interpretability in existing legal citation prediction models, which often rely solely on case facts and fail to reflect how lawyers reason using precedents and statutes.
• Reframe legal citation prediction as a task grounded in legal reasoning—drawing from both case law (precedents) and legislative provisions (statutes)—to better align with practitioner workflows.
• Develop a prototype-based deep learning architecture that enhances model interpretability by learning representative examples of citations, enabling users to trace decisions via similarity to known legal references.
• Integrate feedback from legal experts to refine the task definition and improve model relevance, demonstrating the value of multidisciplinary collaboration in high-stakes AI applications.
• Evaluate the practical utility of the model beyond accuracy, analyzing its behavior through input perturbations and latent space analysis to ensure alignment with legal reasoning patterns.

Proposed method

• Define a new task, Legal Citation Prediction (LCP), where inputs include both the legal problem (abstracted from facts) and reference materials (precedents and statutes), mimicking how lawyers formulate arguments.
• Implement a prototype-based neural architecture where each citation is represented as a prototype vector in embedding space, and predictions are made based on cosine similarity between input representation and prototypes.
• Introduce a modified loss function that encourages the model to align with legal reasoning by strengthening similarity to relevant precedents and statutory provisions, using a weighted combination of cross-entropy and prototype contrastive loss.
• Use k-means clustering to initialize prototypes, with fine-tuning to optimize for citation prediction while maintaining interpretability through prototype proximity.
• Apply input perturbations and latent space analysis to evaluate model robustness and interpretability, ensuring predictions are sensitive to changes in legal context.
• Train and validate the model using a curated dataset with expert-verified citations, selecting the best model based on validation macro F1 rather than loss to avoid collapse during training.

Experimental results

Research questions

• RQ1 How can legal citation prediction be redefined to reflect the actual reasoning process of lawyers, who rely on both case law and statutory provisions?
• RQ2 To what extent does incorporating legal precedents and statutory provisions as input improve model performance and interpretability compared to fact-only baselines?
• RQ3 Can a prototype-based architecture effectively model legal citation prediction while enabling human experts to interpret and validate model decisions?
• RQ4 How does expert feedback influence the refinement of the task definition and model performance in high-stakes legal NLP tasks?
• RQ5 What is the trade-off between performance and interpretability in legal citation prediction, and how does the prototype method balance these in practice?

Key findings

• The proposed LCP task, enhanced with legal expert feedback, significantly improves the relevance and practical utility of predicted citations compared to fact-only baselines.
• The prototype-based architecture achieves strong performance on legal citation prediction while enabling interpretability through similarity-based reasoning to known legal references.
• Input perturbation and latent space analysis confirm that the model's predictions are sensitive to changes in legal context, supporting its alignment with human reasoning patterns.
• The model's performance is higher on procedural (easier) citations and lower on non-procedural (harder) citations, indicating a long-tail challenge in legal citation prediction that remains unresolved.
• Model collapse during training was observed due to unstable k-means initialization and loss function dynamics, resolved by selecting the best model based on macro F1 rather than loss.
• The system is designed for use by legal professionals, not laypeople, and its interpretability allows experts to validate or reject predictions based on similarity to known legal authorities.