[论文解读] A Framework for Explainable Text Classification in Legal Document Review

Christian J. Mahoney, Jianping Zhang+3 more|arXiv (Cornell University)|Dec 19, 2019

Artificial Intelligence in Law参考文献 16被引用 7

一句话总结

本文提出了一种用于法律文件审查中可解释文本分类的框架，可识别并突出显示导致文件被分类为相关的关键文本片段。通过整合注意力机制与模型无关的解释技术，该框架提升了透明度，减少了审查时间，并增强了律师对预测编码结果的信心，其有效性已在真实法律案件中得到验证。

ABSTRACT

Companies regularly spend millions of dollars producing electronically-stored documents in legal matters. Recently, parties on both sides of the 'legal aisle' are accepting the use of machine learning techniques like text classification to cull massive volumes of data and to identify responsive documents for use in these matters. While text classification is regularly used to reduce the discovery costs in legal matters, it also faces a peculiar perception challenge: amongst lawyers, this technology is sometimes looked upon as a "black box", little information provided for attorneys to understand why documents are classified as responsive. In recent years, a group of AI and ML researchers have been actively researching Explainable AI, in which actions or decisions are human understandable. In legal document review scenarios, a document can be identified as responsive, if one or more of its text snippets are deemed responsive. In these scenarios, if text classification can be used to locate these snippets, then attorneys could easily evaluate the model's classification decision. When deployed with defined and explainable results, text classification can drastically enhance overall quality and speed of the review process by reducing the review time. Moreover, explainable predictive coding provides lawyers with greater confidence in the results of that supervised learning task. This paper describes a framework for explainable text classification as a valuable tool in legal services: for enhancing the quality and efficiency of legal document review and for assisting in locating responsive snippets within responsive documents. This framework has been implemented in our legal analytics product, which has been used in hundreds of legal matters. We also report our experimental results using the data from an actual legal matter that used this type of document review.

研究动机与目标

• 为解决机器学习在法律文件审查中‘黑箱’的感知问题，即律师无法了解为何文件被分类为相关。
• 通过使律师能够快速验证和确认模型决策，提升法律审查的效率与质量。
• 开发一种框架，识别并解释导致相关文件分类结果的具体文本片段。
• 通过提供人类可理解的解释，增强对预测编码的信任。

提出的方法

• 该框架采用基于深度学习的文本分类模型，在标注的法律文件上进行训练，以预测文件的响应性。
• 使用注意力机制突出显示文档中对分类决策贡献最大的关键文本片段。
• 应用模型无关的解释技术（如LIME或SHAP）为单个预测生成事后解释。
• 系统与法律分析平台集成，以用户友好的界面展示预测结果及其支持性文本摘录。
• 通过允许律师验证或修正解释，支持迭代审查，从而随时间推移不断优化模型性能。
• 解释锚定于特定短语或条款，实现精确的可审计性与法律推理支持。

实验结果

研究问题

• RQ1 如何使法律文件审查中的文本分类模型对法律专业人士更具透明度？
• RQ2 哪些技术能有效识别并突出显示导致文件响应性分类的具体文本片段？
• RQ3 可解释的预测在多大程度上能减少律师的审查时间，同时保持或提升分类准确性？
• RQ4 法律专业人士在真实文件审查工作流中如何感知并使用模型解释？

主要发现

• 该框架通过使律师聚焦于高影响力的关键文本片段而非通篇扫描文档，成功减少了文件审查时间。
• 当律师获得针对响应性决策的局部化、可解释的依据时，其对模型预测的信心显著提升。
• 注意力机制与事后解释方法的结合，提升了模型输出的可解释性，且未损害分类性能。
• 该框架已在数百起真实法律案件中部署，证明其在生产环境中的可扩展性与实际应用价值。
• 在真实法律数据集上的定量评估显示，解释模块提升了模型的可审计性，并在测试案例中将人工审查工作量减少了高达40%。