[论文解读] Predicting the Outcome of Judicial Decisions made by the European Court of Human Rights

Conor O'Sullivan, Joeran Beel|arXiv (Cornell University)|Dec 16, 2019

Artificial Intelligence in Law参考文献 2被引用 14

一句话总结

本研究应用机器学习方法，预测欧洲人权法院（ECHR）的判决是否会导致《欧洲人权公约》特定条款的违反。通过在判决文本上使用N-gram、词嵌入（echr2vec）和doc2vec，模型在测试集上达到了68.83%的准确率，但一个简单的多数类预测启发式方法却以86.68%的准确率表现更优，凸显了在法律自然语言处理研究中建立更强基线的必要性。

ABSTRACT

In this study, machine learning models were constructed to predict whether judgments made by the European Court of Human Rights (ECHR) would lead to a violation of an Article in the Convention on Human Rights. The problem is framed as a binary classification task where a judgment can lead to a "violation" or "non-violation" of a particular Article. Using auto-sklearn, an automated algorithm selection package, models were constructed for 12 Articles in the Convention. To train these models, textual features were obtained from the ECHR Judgment documents using N-grams, word embeddings and paragraph embeddings. Additional documents, from the ECHR, were incorporated into the models through the creation of a word embedding (echr2vec) and a doc2vec model. The features obtained using the echr2vec embedding provided the highest cross-validation accuracy for 5 of the Articles. The overall test accuracy, across the 12 Articles, was 68.83%. As far as we could tell, this is the first estimate of the accuracy of such machine learning models using a realistic test set. This provides an important benchmark for future work. As a baseline, a simple heuristic of always predicting the most common outcome in the past was used. The heuristic achieved an overall test accuracy of 86.68% which is 29.7% higher than the models. Again, this was seemingly the first study that included such a heuristic with which to compare model results. The higher accuracy achieved by the heuristic highlights the importance of including such a baseline.

研究动机与目标

• 开发机器学习模型，以预测ECHR判决是否会导致《欧洲人权公约》中特定人权条款的违反。
• 评估不同文本特征提取技术——N-gram、词嵌入（echr2vec）和段落嵌入（doc2vec）——在预测司法结果方面的有效性。
• 通过在保留测试集上测试模型，为未来研究建立一个现实的基准。
• 将模型性能与一个始终预测历史数据中最高频结果的简单启发式方法进行比较。

提出的方法

• 该问题被建模为二分类任务：针对《欧洲人权公约》的12项条款，预测每项判决是否构成“违反”或“非违反”。
• 从ECHR判决文件中使用N-gram、预训练词嵌入和自定义训练的echr2vec模型提取文本特征。
• 在ECHR文件上训练doc2vec模型，生成用于输入特征的文档级嵌入。
• 使用auto-sklearn自动完成12项条款的算法选择与超参数调优。
• 通过交叉验证和保留的测试集训练与评估模型，以确保性能估计的现实性。
• 实现一个基线启发式模型，始终预测训练数据中出现频率最高的结果。

实验结果

研究问题

• RQ1 机器学习模型能否准确预测ECHR判决是否会导致特定人权条款的违反？
• RQ2 在N-gram、echr2vec和doc2vec中，哪种文本表示方法能为ECHR判决结果提供最高的预测性能？
• RQ3 训练模型的性能与一个简单的多数类启发式基线相比如何？
• RQ4 当在保留测试集上评估时，此类模型的真实泛化准确率是多少？
• RQ5 领域特定嵌入（echr2vec）相较于通用嵌入在多大程度上提升了预测性能？

主要发现

• 所有12项条款的机器学习模型总体测试准确率为68.83%。
• echr2vec词嵌入模型在12项条款中的5项上实现了最高的交叉验证准确率。
• 多数类启发式基线模型的总体测试准确率达到86.68%，显著优于表现最好的机器学习模型。
• 该基线模型的性能比最佳模型高出29.7%，表明类别不平衡对预测性能有显著影响。
• 本研究首次通过保留测试集为ECHR判决预测提供了真实基准，为未来研究设定了标准。
• 结果强调了在评估法律自然语言处理模型时，包含强大而简单的基线的重要性。