[Paper Review] A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models

Junjie Ye, Xuanting Chen+13 more|arXiv (Cornell University)|Mar 18, 2023

Topic Modeling186 citations

TL;DR

The paper analyzes GPT-3 and GPT-3.5 series across nine NLU tasks over 21 datasets, comparing zero-shot and few-shot performance, and finds RLHF improves generative quality but can harm some tasks.

ABSTRACT

GPT series models, such as GPT-3, CodeX, InstructGPT, ChatGPT, and so on, have gained considerable attention due to their exceptional natural language processing capabilities. However, despite the abundance of research on the difference in capabilities between GPT series models and fine-tuned models, there has been limited attention given to the evolution of GPT series models' capabilities over time. To conduct a comprehensive analysis of the capabilities of GPT series models, we select six representative models, comprising two GPT-3 series models (i.e., davinci and text-davinci-001) and four GPT-3.5 series models (i.e., code-davinci-002, text-davinci-002, text-davinci-003, and gpt-3.5-turbo). We evaluate their performance on nine natural language understanding (NLU) tasks using 21 datasets. In particular, we compare the performance and robustness of different models for each task under zero-shot and few-shot scenarios. Our extensive experiments reveal that the overall ability of GPT series models on NLU tasks does not increase gradually as the models evolve, especially with the introduction of the RLHF training strategy. While this strategy enhances the models' ability to generate human-like responses, it also compromises their ability to solve some tasks. Furthermore, our findings indicate that there is still room for improvement in areas such as model robustness.

Motivation & Objective

• Understand how GPT-3 and GPT-3.5 series capabilities evolve over time.
• Compare GPT-3 and GPT-3.5 models across multiple NLU tasks and datasets.
• Assess zero-shot and few-shot performance for each model-task pair.
• Evaluate robustness and identify where improvements are needed.
• Analyze the impact of reinforcement learning from human feedback (RLHF) on capabilities.

Proposed method

• Select six representative models from the GPT-3 and GPT-3.5 series (davinci, text-davinci-001, code-davinci-002, text-davinci-002, text-davinci-003, gpt-3.5-turbo).
• Evaluate model performance on nine natural language understanding tasks using 21 datasets.
• Compare zero-shot and few-shot settings for each task and model.
• Assess robustness of models across tasks and settings.
• Analyze how RLHF training affects task performance versus generation quality.
• Provide a cross-model synthesis of capability evolution.

Experimental results

Research questions

• RQ1 Do GPT-3 and GPT-3.5 models show gradual improvements across the evaluated NLU tasks?
• RQ2 How does RLHF influence the balance between task-solving ability and generation quality across models?
• RQ3 What are the robustness characteristics of GPT-3 vs GPT-3.5 models on these tasks?
• RQ4 Are there specific tasks where newer models underperform relative to earlier ones?

Key findings

• Overall ability of GPT series on NLU tasks does not increase gradually with model evolution.
• RLHF training enhances generation to be more human-like but can compromise performance on some tasks.
• There remains substantial room for improvement in model robustness across tasks.
• Differences in performance between GPT-3 and GPT-3.5 series are task- and setting-dependent.
• The study highlights trade-offs between generation quality and task-solving capabilities introduced by RLHF.