[Paper Review] AI and Legal Argumentation: Aligning the Autonomous Levels of AI Legal Reasoning

Lance B. Eliot|arXiv (Cornell University)|Sep 11, 2020

Artificial Intelligence in Law4 citations

TL;DR

This paper introduces a novel meta-framework that applies the Levels of Autonomy (LoA) model to AI-driven legal argumentation, proposing a structured approach to measure and align progress toward human-equivalent legal reasoning. By mapping AI legal reasoning to the CARE framework—Crafting, Assessing, Refining, and Engaging—the study provides a scalable, hierarchical model for evaluating AI's maturity in legal reasoning tasks, offering a benchmark for future development in autonomous legal AI systems.

ABSTRACT

Legal argumentation is a vital cornerstone of justice, underpinning an adversarial form of law, and extensive research has attempted to augment or undertake legal argumentation via the use of computer-based automation including Artificial Intelligence (AI). AI advances in Natural Language Processing (NLP) and Machine Learning (ML) have especially furthered the capabilities of leveraging AI for aiding legal professionals, doing so in ways that are modeled here as CARE, namely Crafting, Assessing, Refining, and Engaging in legal argumentation. In addition to AI-enabled legal argumentation serving to augment human-based lawyering, an aspirational goal of this multi-disciplinary field consists of ultimately achieving autonomously effected human-equivalent legal argumentation. As such, an innovative meta-approach is proposed to apply the Levels of Autonomy (LoA) of AI Legal Reasoning (AILR) to the maturation of AI and Legal Argumentation (AILA), proffering a new means of gauging progress in this ever-evolving and rigorously sought domain.

Motivation & Objective

• To address the lack of standardized metrics for evaluating the autonomy of AI in legal reasoning.
• To bridge the gap between current AI capabilities in legal argumentation and the aspirational goal of human-equivalent legal reasoning.
• To provide a structured, scalable framework for assessing the maturity of AI systems in legal domains.
• To align AI's functional capabilities in legal argumentation with established autonomy levels, enabling clearer progress tracking.
• To establish a common language and evaluation standard for interdisciplinary research in AI and law.

Proposed method

• Adapting the Levels of Autonomy (LoA) model from autonomous vehicles to AI legal reasoning, creating a tiered scale of autonomy from manual to fully autonomous.
• Mapping AI legal reasoning to the CARE framework: Crafting, Assessing, Refining, and Engaging in legal argumentation.
• Using the CARE model as a functional taxonomy to define what AI systems must achieve at each autonomy level.
• Applying the LoA model to categorize existing AI systems based on their ability to perform legal reasoning tasks independently.
• Proposing a meta-approach that enables researchers and developers to assess where their AI systems stand in the autonomy spectrum.
• Using visual and conceptual models (9 figures) to illustrate the progression of AI capabilities across the autonomy levels.

Experimental results

Research questions

• RQ1 How can the Levels of Autonomy (LoA) model be adapted to measure the maturity of AI systems in legal argumentation?
• RQ2 What functional capabilities define each level of autonomy in AI legal reasoning?
• RQ3 How does the CARE framework (Crafting, Assessing, Refining, Engaging) align with the autonomy levels in legal AI?
• RQ4 What benchmarks or criteria can be used to evaluate whether an AI system achieves human-equivalent legal reasoning?
• RQ5 How can the proposed framework guide the development and evaluation of future autonomous legal AI systems?

Key findings

• The proposed LoA framework provides a scalable and interpretable model for evaluating the autonomy of AI in legal reasoning, enabling clearer progress tracking.
• The CARE framework effectively structures the functional components of legal argumentation across autonomy levels, offering a practical taxonomy.
• The integration of LoA with legal reasoning allows for the classification of existing AI systems based on their level of independence in legal tasks.
• The framework enables researchers to identify gaps in current AI systems, particularly in higher autonomy levels such as autonomous refinement and engagement.
• The model supports interdisciplinary collaboration by offering a shared vocabulary and evaluation standard for AI and legal scholars.
• The paper establishes a foundation for future research by defining measurable milestones toward human-equivalent legal reasoning in AI.