[论文解读] Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence

John J. Nay, David Karamardian+7 more|arXiv (Cornell University)|Jun 12, 2023

Artificial Intelligence in Law被引用 13

一句话总结

这篇论文研究大型语言模型如何获取并提升在税法中的法律推理能力，展示在不同模型版本中的涌现能力，以及情境信息和提示对性能的影响。

ABSTRACT

Better understanding of Large Language Models' (LLMs) legal analysis abilities can contribute to improving the efficiency of legal services, governing artificial intelligence, and leveraging LLMs to identify inconsistencies in law. This paper explores LLM capabilities in applying tax law. We choose this area of law because it has a structure that allows us to set up automated validation pipelines across thousands of examples, requires logical reasoning and maths skills, and enables us to test LLM capabilities in a manner relevant to real-world economic lives of citizens and companies. Our experiments demonstrate emerging legal understanding capabilities, with improved performance in each subsequent OpenAI model release. We experiment with retrieving and utilising the relevant legal authority to assess the impact of providing additional legal context to LLMs. Few-shot prompting, presenting examples of question-answer pairs, is also found to significantly enhance the performance of the most advanced model, GPT-4. The findings indicate that LLMs, particularly when combined with prompting enhancements and the correct legal texts, can perform at high levels of accuracy but not yet at expert tax lawyer levels. As LLMs continue to advance, their ability to reason about law autonomously could have significant implications for the legal profession and AI governance.

研究动机与目标

• 理解LLMs如何进行税法分析并识别在模型迭代中出现的能力。
• 评估提供法律权威和情境信息对LLM性能的影响。
• 评估少量示例提示对最强模型的影响。
• 确定当前的LLMs是否能够达到专家级的税法推理水平。

提出的方法

• 建立跨数千个税法示例的自动化验证流程，以测试LLM推理。
• 检索并纳入相关法律权威，为模型提供适当的法律语境。
• 比较OpenAI模型版本之间的性能（例如较早版本与最新版本），以识别涌现能力。
• 通过向模型提供问答对来评估少量示例提示。
• 分析更多法律文本和情境对税法问题求解的影响。

实验结果

研究问题

• RQ1 当模型随着版本推进时，LLMs在税法中是否展现涌现的法律理解能力？
• RQ2 提供法律权威和情境信息如何影响LLM在税法任务中的表现？
• RQ3 少量示例提示是否显著提升最先进模型在税法推理中的准确性？
• RQ4 目前的LLMs是否能够达到专家级税法律师的准确性和一致性水平？

主要发现

• LLMs在连续的模型版本中展现出日益提高的法律理解。
• 提供法律权威和情境信息提升了性能。
• 少量示例提示显著改善最有能力模型的结果。
• LLMs在税法任务中可以达到高准确性，但尚未达到专家级税务律师的水平。
• 进一步提升LLM能力可能对法律职业和AI治理产生深远影响。