[Paper Review] Using Large Language Models to Support Thematic Analysis in Empirical Legal Studies

Jakub Drápal, Hannes Westermann+1 more|arXiv (Cornell University)|Oct 28, 2023

Law in Society and Culture5 citations

TL;DR

This paper proposes a novel LLM-powered framework that supports legal experts in thematic analysis by automating initial coding, theme discovery, and zero-shot classification of criminal court opinions on theft. Using GPT-4 on 785 factual descriptions, the model generated high-quality initial codes, improved with expert feedback, and achieved strong zero-shot performance (R@1: 0.66 overall) in predicting expert-identified themes, demonstrating strong potential for accelerating inductive coding in empirical legal studies.

ABSTRACT

Thematic analysis and other variants of inductive coding are widely used qualitative analytic methods within empirical legal studies (ELS). We propose a novel framework facilitating effective collaboration of a legal expert with a large language model (LLM) for generating initial codes (phase 2 of thematic analysis), searching for themes (phase 3), and classifying the data in terms of the themes (to kick-start phase 4). We employed the framework for an analysis of a dataset (n=785) of facts descriptions from criminal court opinions regarding thefts. The goal of the analysis was to discover classes of typical thefts. Our results show that the LLM, namely OpenAI's GPT-4, generated reasonable initial codes, and it was capable of improving the quality of the codes based on expert feedback. They also suggest that the model performed well in zero-shot classification of facts descriptions in terms of the themes. Finally, the themes autonomously discovered by the LLM appear to map fairly well to the themes arrived at by legal experts. These findings can be leveraged by legal researchers to guide their decisions in integrating LLMs into their thematic analyses, as well as other inductive coding projects.

Motivation & Objective

• To address the time-intensive nature of inductive coding in empirical legal studies by integrating large language models (LLMs) into thematic analysis workflows.
• To evaluate whether LLMs can generate high-quality initial codes from raw factual descriptions of theft cases.
• To assess the effectiveness of expert feedback in refining LLM-generated codes and themes.
• To investigate the LLM’s ability to autonomously discover and classify themes in a zero-shot setting without prior labeling.
• To validate the alignment between LLM-discovered themes and those identified by legal experts in a real-world criminal law context.

Proposed method

• The framework integrates GPT-4 into phases 2 and 3 of thematic analysis: initial coding and theme discovery.
• Initial codes were generated via prompt engineering, with subsequent refinement using natural language feedback from legal experts.
• Themes were discovered through iterative LLM prompting, using clustering-like reasoning on generated codes.
• Zero-shot classification of facts into expert-defined themes was evaluated using recall at 1 and 3 (R@1, R@3).
• The LLM’s end-to-end performance was assessed by comparing its autonomously discovered themes against manually identified themes from legal experts.
• Expert supervision was applied at key stages to improve theme coherence and alignment with research goals.

Figure 3: The diagram describes the proposed framework supporting phases 2 and 3 of thematic analysis. The data points are batched (1) to fit the LLM’s prompt. Initial codes for each batch are predicted (2) in an iterative fashion utilizing the interim results (3 and 4). Expert feedback on the initi

Experimental results

Research questions

• RQ1 RQ1: How successfully can the LLM perform initial coding of the data?
• RQ2 RQ2: To what degree can a subject matter expert improve the quality of the initial codes via natural language feedback?
• RQ3 RQ3: How successfully can the LLM predict themes for the analyzed data points?
• RQ4 RQ4: How successfully can the LLM autonomously discover themes and associate them with the analyzed data?

Key findings

• The LLM generated reasonable initial codes for 785 factual descriptions of theft cases, with quality significantly improved through expert feedback.
• The zero-shot classification of facts into expert-defined themes achieved an overall R@1 of 0.66 and R@3 of 0.82, indicating strong performance across most categories.
• Performance varied by theme: GPT-4 achieved perfect zero-shot recall (R@1 = 1.0) for 'theft of energy' and high performance for 'theft in a shop' (R@1 = 0.95).
• The LLM discovered 8 themes autonomously, which mapped reasonably well to the 14 themes identified by legal experts, with some LLM themes subsuming multiple expert themes (e.g., 'theft in commercial settings' covering 'theft at work' and 'breaking into another object').
• The LLM identified some behaviors missed by experts, such as 'theft from a gym', suggesting potential for uncovering novel patterns.
• Despite strong performance, some themes (e.g., 'robbery of cellars' and 'theft from an open-access place') showed low recall, indicating limitations in capturing nuanced or less common theft types.

Figure 5: The graph shows mapping between themes discovered by subject matter expert (left) and the themes discovered by the proposed framework (right).