[Paper Review] Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?

Jaromír Šavelka, Kevin D. Ashley+3 more|arXiv (Cornell University)|Jun 24, 2023

Topic ModelingComputer Science38 references11 citations

TL;DR

GPT-4, when prompted with annotation guidelines, performs on par with well-trained law student annotators for legal sentence analysis and can batch-predict at lower cost, though chain-of-thought prompts offer limited benefit and model brittleness is a concern.

ABSTRACT

We evaluated the capability of generative pre-trained transformers~(GPT-4) in analysis of textual data in tasks that require highly specialized domain expertise. Specifically, we focused on the task of analyzing court opinions to interpret legal concepts. We found that GPT-4, prompted with annotation guidelines, performs on par with well-trained law student annotators. We observed that, with a relatively minor decrease in performance, GPT-4 can perform batch predictions leading to significant cost reductions. However, employing chain-of-thought prompting did not lead to noticeably improved performance on this task. Further, we demonstrated how to analyze GPT-4's predictions to identify and mitigate deficiencies in annotation guidelines, and subsequently improve the performance of the model. Finally, we observed that the model is quite brittle, as small formatting related changes in the prompt had a high impact on the predictions. These findings can be leveraged by researchers and practitioners who engage in semantic/pragmatic annotations of texts in the context of the tasks requiring highly specialized domain expertise.

Motivation & Objective

• Assess GPT-4's capability to analyze court opinions for legal concept interpretation against human annotators.
• Explore batch prediction as a cost-effective alternative to single-sentence evaluation.
• Evaluate whether chain-of-thought prompting improves accuracy in this specialized task.
• Investigate how annotation guideline changes affect GPT-4 performance and model robustness.

Proposed method

• Use GPT-4 with system prompts derived from annotation guidelines to perform sentence-level and batch labeling.
• Compare GPT-4 outputs to gold labels from a published dataset of court opinions and statutory interpretations.
• Experiment with single-label predictions versus batch predictions to assess cost and performance trade-offs.
• Test chain-of-thought prompting to see if explanations improve or degrade accuracy.
• Modify annotation guidelines to identify improvements in GPT-4 performance and assess robustness to prompt changes.

Figure 1: A mock-up interface with an example statutory provision on the left (1). The user indicated that they are interested in the meaning of the “common business purpose“ phrase as used in the provision (2). The system responds with a list of sentences that are deemed useful for explaining the m

Experimental results

Research questions

• RQ1 RQ1: How does GPT-4's annotation performance compare to human annotators on this specialized legal task?
• RQ2 RQ2: Can GPT-4 effectively perform batch predictions without significant loss in accuracy?
• RQ3 RQ3: Does requiring explanations (CoT prompting) improve GPT-4's predictions on this task?
• RQ4 RQ4: How do modifications to annotation guidelines impact GPT-4's performance?
• RQ5 RQ5: How robust are GPT-4 predictions to small prompt changes across experiments?

Key findings

• GPT-4's overall F1 is 0.53 when predicting single labels with original instructions, comparable to human law students on this dataset.
• Batch predictions yield an F1 of 0.52 with slightly lower performance than single-point predictions but at a substantially reduced cost.
• Prompting with explanations (CoT) does not improve performance under original guidelines and can degrade results, especially in batch settings.
• Refining annotation guidelines (prompt) improves single-label F1 to 0.57, and reduces certain mislabeling issues observed previously.
• Batch predictions with updated guidelines and explanations show mixed results, with some configurations preserving performance while others degrade it; robustness to prompt changes remains an issue.
• GPT-4's predictions are brittle: small prompt formatting changes can significantly affect outcomes.

Figure 8: The annotator agreement (Krippendorff’s $\alpha$ ) of the manually created annotations and GPT-4 predictions computed against the consensus (gold) labels. GPT-4 performs comparably to human annotators (law students).