[Paper Review] Enhancing Pre-Trained Language Models with Sentence Position Embeddings for Rhetorical Roles Recognition in Legal Opinions

Anas Belfathi, Nicolás Hernández+1 more|arXiv (Cornell University)|Oct 8, 2023

Artificial Intelligence in LawSocial Sciences3 citations

TL;DR

This paper proposes enhancing BERT with sentence position embeddings—absolute, normalized, and k-quantile—for improved rhetorical role recognition in legal opinions. By integrating positional information into a single BERT encoder or a hierarchical model with a reduced context window, the approach achieves a 79% F1-score, outperforming prior methods and demonstrating efficiency with lower computational cost.

ABSTRACT

The legal domain is a vast and complex field that involves a considerable amount of text analysis, including laws, legal arguments, and legal opinions. Legal practitioners must analyze these texts to understand legal cases, research legal precedents, and prepare legal documents. The size of legal opinions continues to grow, making it increasingly challenging to develop a model that can accurately predict the rhetorical roles of legal opinions given their complexity and diversity. In this research paper, we propose a novel model architecture for automatically predicting rhetorical roles using pre-trained language models (PLMs) enhanced with knowledge of sentence position information within a document. Based on an annotated corpus from the LegalEval@SemEval2023 competition, we demonstrate that our approach requires fewer parameters, resulting in lower computational costs when compared to complex architectures employing a hierarchical model in a global-context, yet it achieves great performance. Moreover, we show that adding more attention to a hierarchical model based only on BERT in the local-context, along with incorporating sentence position information, enhances the results.

Motivation & Objective

• Address the challenge of accurately predicting rhetorical roles in long, unstructured legal opinions.
• Overcome limitations of standard pre-trained language models in handling long legal documents beyond 512 tokens.
• Improve performance on rhetorical role sequence labeling by incorporating sentence position information into BERT.
• Evaluate the impact of different positional encoding schemes (absolute, normalized, k-quantile) on model performance.
• Demonstrate that a smaller context window in hierarchical modeling enhances performance when combined with positional embeddings.

Proposed method

• Integrate absolute, normalized, and k-quantile sentence position embeddings into the input representation of BERT.
• Apply three types of positional encoding: absolute (sentence index), normalized (scaled by document length), and k-quantile (dividing document into k segments).
• Design a hierarchical model (HiBERT) with a BERT encoder for sentence-level encoding and a shallow Transformer encoder for local sentence context.
• Use a dynamic window size based on the average length of consecutive sentences with the same rhetorical role label.
• Fine-tune models on the LegalEval@SemEval2023 dataset using weighted F1 as the primary metric.
• Evaluate both single-encoder (BERT) and hierarchical (HiBERT) architectures with varying positional encoding types and window sizes.

Experimental results

Research questions

• RQ1 How does injecting sentence position information affect BERT’s performance in rhetorical role classification for legal texts?
• RQ2 Which type of sentence position encoding—absolute, normalized, or k-quantile—yields the best performance?
• RQ3 Can a hierarchical model with localized attention and sentence position embeddings outperform standard BERT in this task?
• RQ4 Does reducing the context window size in the hierarchical model improve performance and efficiency?
• RQ5 To what extent does combining positional embeddings with local context modeling enhance sequence labeling in long legal documents?

Key findings

• The proposed HiBERT model with absolute position embeddings and a window size of 4 sentences achieved the highest F1-score of 79%.
• Normalizing sentence positions improved performance over raw absolute positions, with a F1-score of 75% on BERT-NorPos.
• The k-quantile approach with 8 parts yielded the best results among quantile variants, though it underperformed normalized positions.
• Reducing the hierarchical model’s context window from full to half the average sentence cluster size improved performance, especially with absolute positions.
• The HiBERT-AbsPosHalf model achieved a 79% F1-score, outperforming the baseline BERT-HSLN model (79% F1), indicating that localized context with absolute position is optimal.
• The approach achieved high performance with fewer parameters and lower computational cost compared to complex hierarchical models, making it efficient for real-world deployment.