[Paper Review] Blind Judgement: Agent-Based Supreme Court Modelling With GPT

Sil Hamilton|arXiv (Cornell University)|Jan 12, 2023

Artificial Intelligence in LawSocial Sciences12 citations

TL;DR

The paper trains a nine-agent GPT-2 based system to simulate Roberts IV Supreme Court decisions and opinions, achieving better-than-random accuracy on 96 test cases and revealing a link between model accuracy and judicial alignment to precedent.

ABSTRACT

We present a novel Transformer-based multi-agent system for simulating the judicial rulings of the 2010-2016 Supreme Court of the United States. We train nine separate models with the respective authored opinions of each supreme justice active ca. 2015 and test the resulting system on 96 real-world cases. We find our system predicts the decisions of the real-world Supreme Court with better-than-random accuracy. We further find a correlation between model accuracy with respect to individual justices and their alignment between legal conservatism & liberalism. Our methods and results hold significance for researchers interested in using language models to simulate politically-charged discourse between multiple agents.

Motivation & Objective

• Demonstrate a multi-agent, language-model-based approach to simulating Supreme Court decision-making.
• Evaluate whether distributing the task across nine justice-specific models improves predictive performance.
• Investigate the relationship between a justice's alignment to precedent and the model's predictive accuracy.

Proposed method

• Construct nine separate GPT-2 based models trained on each active justice's authored opinions from 2003-2016.
• Use a two-step training process starting from a unanimous-decision base model and then fine-tuning per-justice models.
• Train with prompts structured as discrete Python-like dictionaries containing issue, topic, opinion, and decision fields.
• Evaluate on 96 held-out cases using ROC accuracy and Cohen’s kappa to account for class distribution bias.
• Quantify ideological alignment by correlating per-justice accuracy with the frequency of voting against precedent (Pearson r).
• Compare multi-agent performance to a single-agent baseline fine-tuned on majority opinions.

Experimental results

Research questions

• RQ1 Can a multi-agent GPT-2 based system predict real-world Supreme Court decisions better than chance?
• RQ2 Does distributing knowledge across individual justice models improve predictive accuracy compared to a single-agent baseline?
• RQ3 Is there a measurable relationship between a justice’s alignment with precedent and the model’s prediction accuracy?

Key findings

• The aggregated multi-agent system achieves 60% accuracy (kappa ≈ 0.18) on 96 test cases.
• Per-justice accuracy ranges from 50% to 65% across the nine models (Alito 65%, Ginsburg 62%, Thomas 59%, Breyer 58%, Roberts 57%, Kagan 56%, Kennedy 54%, Sotomayor 51%, Scalia 50%).
• A moderate correlation (r ≈ 0.56) exists between higher model accuracy and justices who less frequently overturn precedent.
• A single-agent model trained on majority opinions achieves 54% accuracy (kappa ≈ 0.08) and lower population overlap with the 96-case test set (92.4% vs 68.5%).
• The model exhibits precedent hallucination, citing non-existent or incorrect precedent in generated opinions.
• Conservative justices are more predictable in this setup, suggesting ideological alignment influences predictability.