[Paper Review] GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models

Tyna Eloundou, Sam Manning+2 more|arXiv (Cornell University)|Mar 17, 2023

Artificial Intelligence in Healthcare and Education537 citations

TL;DR

The paper uses a new exposure rubric applied to O*NET task data, with human and GPT-4 classifications, to estimate how LLMs could affect U.S. work tasks and occupations. It finds widespread exposure and substantial gains when using LLM-powered software.

ABSTRACT

We investigate the potential implications of large language models (LLMs), such as Generative Pre-trained Transformers (GPTs), on the U.S. labor market, focusing on the increased capabilities arising from LLM-powered software compared to LLMs on their own. Using a new rubric, we assess occupations based on their alignment with LLM capabilities, integrating both human expertise and GPT-4 classifications. Our findings reveal that around 80% of the U.S. workforce could have at least 10% of their work tasks affected by the introduction of LLMs, while approximately 19% of workers may see at least 50% of their tasks impacted. We do not make predictions about the development or adoption timeline of such LLMs. The projected effects span all wage levels, with higher-income jobs potentially facing greater exposure to LLM capabilities and LLM-powered software. Significantly, these impacts are not restricted to industries with higher recent productivity growth. Our analysis suggests that, with access to an LLM, about 15% of all worker tasks in the US could be completed significantly faster at the same level of quality. When incorporating software and tooling built on top of LLMs, this share increases to between 47 and 56% of all tasks. This finding implies that LLM-powered software will have a substantial effect on scaling the economic impacts of the underlying models. We conclude that LLMs such as GPTs exhibit traits of general-purpose technologies, indicating that they could have considerable economic, social, and policy implications.

Motivation & Objective

• Motivate understanding of how LLMs (GPTs) could affect labor markets beyond model capabilities alone.
• Develop and apply a rubric to measure task exposure to LLMs using human and GPT-4 classifications.
• Aggregate task-level exposure to occupation- and industry-level insights.
• Highlight the role of complementary technologies and LLM-powered software in scaling economic impact.

Proposed method

• Construct a task- and occupation-level exposure measure using O*NET DWAs and tasks (19,265 tasks; 2,087 DWAs).
• Annotate exposure with human judgments and GPT-4 classifications under a 50% time-reduction rubric.
• Define three exposure metrics: alpha (E1), beta (E1 + 0.5*E2), and zeta (E1 + E2).
• Aggregate task exposures to occupations with core tasks weighted double relative to supplemental tasks.
• Compare human and GPT-4 annotations and assess agreement and correlations across occupation levels.

Figure 1: Taken directly from GPT-4 Technical Report ( OpenAI, 2023b, ) . To get a sense of how quickly model capabilities are progressing – consider the jump in exam performance between GPT-3.5 and GPT-4 ( OpenAI, 2023b, ) .

Experimental results

Research questions

• RQ1 What share of U.S. occupations have exposure to LLMs at the task level?
• RQ2 How do exposure estimates differ between human raters and GPT-4 classifications?
• RQ3 What is the impact of LLMs alone versus LLM-powered software on task completion time?
• RQ4 How do exposure patterns vary by occupation, wage level, and industry?

Key findings

• Approximately 80% of workers belong to an occupation with at least 10% of tasks exposed to LLMs (beta measure).
• Around 19% of workers are in occupations with at least 50% of tasks exposed to LLMs (beta measure).
• On average, about 15% of all worker tasks could be completed significantly faster with an LLM alone; with LLM-powered software this rises to 47–56%.
• Average occupation-level alpha values are around 0.14–0.15; beta around 0.30 (human) and 0.34 (GPT-4); zeta higher, indicating substantial potential exposure across occupations.
• Exposure tends to be higher for higher-wage jobs and for information-processing industries; exposure correlates with programming and writing skills, and negatively with science/critical thinking skills.
• There is substantial agreement between human and GPT-4 annotations at the occupation level regarding exposure to LLM systems.

Figure 2: Human raters (x-axis) and GPT-4 ratings (y-axis) show a high degree of agreement about LLM exposure by occupation. We compute occupation-level exposure in these figures by averaging the task-level exposures under the $\beta$ method. O*NET designates some tasks as "core" and others "supplem