[Paper Review] GPT Takes the Bar Exam

Michael James Bommarito, Daniel Katz|arXiv (Cornell University)|Dec 29, 2022

Artificial Intelligence in Law55 citations

TL;DR

GPT-3.5 (text-davinci-003) can approach bar-exam questions with zero-shot prompts, achieving 50.3% on a full NCBE MBE practice exam and surpassing random guessing; it passes some categories but not the overall exam.

ABSTRACT

Nearly all jurisdictions in the United States require a professional license exam, commonly referred to as "the Bar Exam," as a precondition for law practice. To even sit for the exam, most jurisdictions require that an applicant completes at least seven years of post-secondary education, including three years at an accredited law school. In addition, most test-takers also undergo weeks to months of further, exam-specific preparation. Despite this significant investment of time and capital, approximately one in five test-takers still score under the rate required to pass the exam on their first try. In the face of a complex task that requires such depth of knowledge, what, then, should we expect of the state of the art in "AI?" In this research, we document our experimental evaluation of the performance of OpenAI's `text-davinci-003` model, often-referred to as GPT-3.5, on the multistate multiple choice (MBE) section of the exam. While we find no benefit in fine-tuning over GPT-3.5's zero-shot performance at the scale of our training data, we do find that hyperparameter optimization and prompt engineering positively impacted GPT-3.5's zero-shot performance. For best prompt and parameters, GPT-3.5 achieves a headline correct rate of 50.3% on a complete NCBE MBE practice exam, significantly in excess of the 25% baseline guessing rate, and performs at a passing rate for both Evidence and Torts. GPT-3.5's ranking of responses is also highly-correlated with correctness; its top two and top three choices are correct 71% and 88% of the time, respectively, indicating very strong non-entailment performance. While our ability to interpret these results is limited by nascent scientific understanding of LLMs and the proprietary nature of GPT, we believe that these results strongly suggest that an LLM will pass the MBE component of the Bar Exam in the near future.

Motivation & Objective

• Assess whether state-of-the-art LLMs can tackle Bar Exam MBE questions without fine-tuning.
• Evaluate the impact of prompt engineering and hyperparameters on zero-shot performance.
• Compare GPT-3.5 performance to NCBE averages and human test-takers across categories.
• Investigate whether fine-tuning offers advantages over zero-shot performance on legal multiple-choice items.

Proposed method

• Use zero-shot prompts with text-davinci-003 to answer NCBE MBE practice questions.
• Experiment with multiple prompt styles including rank-ordering top choices.
• Vary hyperparameters (temperature, top_p, best_of, max_tokens) and attempt fine-tuning.
• Analyze correctness, correlation of answer rankings with correctness, and category-wise performance.
• Compare model outputs to NCBE-reported averages and human performance.

Experimental results

Research questions

• RQ1 Can GPT-3.5 pass any MBE category without task-specific fine-tuning?
• RQ2 How do prompt design and hyperparameters affect zero-shot performance on legal MCQs?
• RQ3 Is the model’s top-k answer ranking correlated with correctness across categories?
• RQ4 Does fine-tuning improve performance relative to zero-shot GPT-3.5 on MBE questions?

Key findings

• GPT-3.5 achieves an average of 50% across categories, exceeding the 25% random-guess baseline.
• GPT-3.5 passes two categories (Evidence and Torts) and is at parity with humans for Evidence questions.
• GPT-3.5’s top-2 and top-3 answer selections correlate highly with correctness (71% and 88% respectively).
• Rank-order prompting (top three choices) yielded the best performance among prompts used.
• Fine-tuning GPT-3.5 did not improve performance over the zero-shot baseline in this study.