[论文解读] Explaining Legal Concepts with Augmented Large Language Models (GPT-4)

Jaromír Šavelka, Kevin D. Ashley+3 more|arXiv (Cornell University)|Jun 15, 2023

Artificial Intelligence in Law被引用 19

一句话总结

本文比较直接给出 GPT-4 对法定术语的解释与加入检索到的案例法句子的增强型 GPT-4 解释，结果显示增强有助于提升事实性与整体质量，同时减少幻觉。

ABSTRACT

Interpreting the meaning of legal open-textured terms is a key task of legal professionals. An important source for this interpretation is how the term was applied in previous court cases. In this paper, we evaluate the performance of GPT-4 in generating factually accurate, clear and relevant explanations of terms in legislation. We compare the performance of a baseline setup, where GPT-4 is directly asked to explain a legal term, to an augmented approach, where a legal information retrieval module is used to provide relevant context to the model, in the form of sentences from case law. We found that the direct application of GPT-4 yields explanations that appear to be of very high quality on their surface. However, detailed analysis uncovered limitations in terms of the factual accuracy of the explanations. Further, we found that the augmentation leads to improved quality, and appears to eliminate the issue of hallucination, where models invent incorrect statements. These findings open the door to the building of systems that can autonomously retrieve relevant sentences from case law and condense them into a useful explanation for legal scholars, educators or practicing lawyers alike.

研究动机与目标

• 评估 GPT-4 向法律专业人士解释法规条文中开放性文本术语的能力。
• 评估直接 GPT-4 解释在事实准确性和对训练数据依赖方面的局限性。
• 测试通过法律信息检索（案例法句子）增强 GPT-4 是否能减少幻觉并提升解释质量。
• 展示一个从案例法检索解释性句子并将其浓缩成解释的流程。
• 提供基准测试，评估增强后的 GPT-4 在专业法律场景中是否优于基线 GPT-4。

提出的方法

• 基线：直接提示 GPT-4 解释来自源条文的术语，不提供外部上下文。
• 增强：从引用该术语的案例法中检索高价值的解释性句子并将其注入到 GPT-4 的提示中。
• 使用包含 42 个术语和 1,853 个高价值句子的法条解释数据集进行增强。
• 为每个术语生成两种解释：短版（1 句）和长版（10 句）。
• 请两名法律学者在五个质量维度上对成对解释进行标注。
• 比较基线与增强输出在事实性、清晰度、相关性、信息丰富性和要点针对性方面的差异。

Figure 2: System Architectures Diagrams. The top part shows the baseline directly applying the LLM. The bottom part describes the components of the augmented architecture that relying on the information retrieval component.

实验结果

研究问题

• RQ1 直接用 GPT-4 生成法条解释有哪些局限？
• RQ2 用相关的案例法句子对 GPT-4 进行增强，是否在事实性、清晰度、相关性、信息丰富性和要点针对性等方面提升解释质量？

主要发现

• 在短版和长版解释中，增强型 GPT-4 的解释通常比基线更受标注者欢迎。
• 增强解释消除了基线事实性评估中观察到的不存在引用和错误表述问题。
• 增强解释在清晰度、相关性、信息丰富性和要点针对性方面优于基线。
• 基线解释存在幻觉和引用不准确的问题，尽管许多引用是真实的；内容常常误解案例。
• 当信息检索组件提供无关或误导性的案例法内容时，增强也无法完全消除问题；高质量的信息检索至关重要。
• 总体而言，增强后的大型语言模型在法律教育和实践中自动生成法定术语解释的准确摘要方面具有潜力。

Figure 3: Short Explanation Preferences. Red corresponds to the preferences for the explanations generated by the baseline system while green indicates preferences for the explanations coming from the augmented LLM. The yellow/orange informs about the number of instances where no preference was indi