[论文解读] GPT Takes the Bar Exam

Michael James Bommarito, Daniel Katz|arXiv (Cornell University)|Dec 29, 2022

Artificial Intelligence in Law被引用 55

一句话总结

GPT-3.5 (text-davinci-003) 能以零-shot 提示接近律师资格考试题目，在完整的 NCBE MBE 练习考试中达到 50.3%，并超过随机猜测；它通过某些类别，但并非整体考试。

ABSTRACT

Nearly all jurisdictions in the United States require a professional license exam, commonly referred to as "the Bar Exam," as a precondition for law practice. To even sit for the exam, most jurisdictions require that an applicant completes at least seven years of post-secondary education, including three years at an accredited law school. In addition, most test-takers also undergo weeks to months of further, exam-specific preparation. Despite this significant investment of time and capital, approximately one in five test-takers still score under the rate required to pass the exam on their first try. In the face of a complex task that requires such depth of knowledge, what, then, should we expect of the state of the art in "AI?" In this research, we document our experimental evaluation of the performance of OpenAI's `text-davinci-003` model, often-referred to as GPT-3.5, on the multistate multiple choice (MBE) section of the exam. While we find no benefit in fine-tuning over GPT-3.5's zero-shot performance at the scale of our training data, we do find that hyperparameter optimization and prompt engineering positively impacted GPT-3.5's zero-shot performance. For best prompt and parameters, GPT-3.5 achieves a headline correct rate of 50.3% on a complete NCBE MBE practice exam, significantly in excess of the 25% baseline guessing rate, and performs at a passing rate for both Evidence and Torts. GPT-3.5's ranking of responses is also highly-correlated with correctness; its top two and top three choices are correct 71% and 88% of the time, respectively, indicating very strong non-entailment performance. While our ability to interpret these results is limited by nascent scientific understanding of LLMs and the proprietary nature of GPT, we believe that these results strongly suggest that an LLM will pass the MBE component of the Bar Exam in the near future.

研究动机与目标

• 评估前沿大型语言模型在不进行微调的情况下能否处理律师考试 MBE 问题。
• 评估提示设计与超参数对零-shot 性能的影响。
• 将 GPT-3.5 的表现与 NCBE 的各类别平均值及人类考生进行比较。
• 探索微调是否相比零-shot 在法律多项选择题上具有优势。

提出的方法

• 使用 text-davinci-003 的零-shot 提示回答 NCBE MBE 练习题。
• 试验多种提示风格，包括对前选项进行排序。
• 调整超参数（temperature、top_p、best_of、max_tokens）并尝试微调。
• 分析正确性、答案排序与正确性的相关性，以及按类别的表现。
• 将模型输出与 NCBE 报告的平均值及人类表现进行比较。

实验结果

研究问题

• RQ1 GPT-3.5 能否在不进行任务特定微调的情况下通过任何 MBE 类别？
• RQ2 提示设计和超参数如何影响法律多项选择题的零-shot 性能？
• RQ3 模型的前 k 名答案排序是否与跨类别的正确性相关？
• RQ4 微调是否相对于零-shot 的 GPT-3.5 在 MBE 问题上提高了性能？

主要发现

• GPT-3.5 在各类别的平均值为 50%，超过 25% 的随机猜测基线。
• GPT-3.5 通过两个类别（Evidence 和 Torts），在 Evidence 问题上与人类持平。
• GPT-3.5 的前两名和前三名答案选择与正确性高度相关（分别为 71% 与 88%）。
• 排序提示（前三名选择）在所用提示中表现最佳。
• 本研究中对 GPT-3.5 进行微调并未优于零-shot 基线。