[Paper Review] ParaLaw Nets -- Cross-lingual Sentence-level Pretraining for Legal Text Processing

Ha-Thanh Nguyen, Vu Tran+7 more|arXiv (Cornell University)|Jun 25, 2021

Topic Modeling12 references8 citations

TL;DR

This paper proposes ParaLaw Nets, a cross-lingual sentence-level pretraining framework for legal text processing that leverages parallel legal texts in Japanese and English to reduce semantic ambiguity. By training on aligned sentence pairs with data augmentation via negation rules, the method achieves state-of-the-art performance on the COLIEE-2021 legal question-answering benchmark, with the NFSP Base model attaining 60.49% accuracy on the blind test set.

ABSTRACT

Ambiguity is a characteristic of natural language, which makes expression ideas flexible. However, in a domain that requires accurate statements, it becomes a barrier. Specifically, a single word can have many meanings and multiple words can have the same meaning. When translating a text into a foreign language, the translator needs to determine the exact meaning of each element in the original sentence to produce the correct translation sentence. From that observation, in this paper, we propose ParaLaw Nets, a pretrained model family using sentence-level cross-lingual information to reduce ambiguity and increase the performance in legal text processing. This approach achieved the best result in the Question Answering task of COLIEE-2021.

Motivation & Objective

• To address semantic ambiguity in legal text processing by leveraging cross-lingual sentence-level alignment.
• To improve zero-shot and few-shot generalization in legal NLP tasks using multilingual pretraining.
• To develop a robust pretraining strategy that handles complex negations in Japanese legal texts.
• To evaluate the effectiveness of cross-lingual sentence-level pretraining on legal benchmark datasets like COLIEE-2021.
• To demonstrate that sentence-level parallel data enhances model understanding beyond word-level or document-level alignment.

Proposed method

• The model is pre-trained on parallel sentences from the Japanese Civil Code and its English translation, using aligned sentence pairs to enforce semantic consistency.
• Data augmentation is applied via negation rules: for each lawful sentence, its negated version is generated with reversed label (unlawful), increasing training diversity.
• A curriculum learning strategy is used for Japanese data: initial training on the first three negation rules before full dataset training to stabilize convergence.
• The approach is evaluated on a lawfulness classification task using six models: BERT Multilingual, XLM-RoBERTa, NFSP, and NMSP variants (base and distilled).
• Cross-lingual sentence-level pretraining is implemented by jointly encoding source and target sentences to learn shared semantic representations.
• Fine-tuning is performed on the COLIEE-2021 legal question-answering dataset, with model performance measured on blind test set accuracy.

Experimental results

Research questions

• RQ1 Can sentence-level cross-lingual pretraining reduce ambiguity in legal text understanding compared to monolingual or word-level alignment methods?
• RQ2 How does data augmentation via negation rules improve model robustness in legal text classification tasks?
• RQ3 Why do distilled models and XLM-RoBERTa fail to converge on the Japanese legal data, while NFSP and NMSP models succeed?
• RQ4 Does pretraining on parallel legal sentences lead to better zero-shot or few-shot performance on downstream legal NLP tasks?
• RQ5 To what extent does the model’s performance generalize across language pairs with differing syntactic and morphological complexity, such as English and Japanese?

Key findings

• The NFSP Base model achieved 60.49% accuracy on the COLIEE-2021 blind test set, outperforming all other submitted systems.
• The NMSP Base model achieved 55.56% accuracy, ranking second, while BERT Multilingual scored 46.91%, indicating the superiority of the proposed cross-lingual approach.
• The distilled variants of NFSP and NMSP failed to converge, with accuracy below 52%, suggesting that knowledge distillation may not be effective for this specific cross-lingual setup.
• XLM-RoBERTa and distilled models exhibited unstable training loss fluctuating around 0.7, indicating poor learning dynamics on the Japanese legal data.
• The BERT Multilingual model showed better convergence than XLM-RoBERTa but still underperformed compared to NFSP and NMSP, highlighting the benefit of sentence-level cross-lingual pretraining.
• The curriculum learning strategy was essential for training on Japanese data, as full-data training with all negation rules caused non-convergence, suggesting higher complexity in Japanese negation patterns.