[Paper Review] a survey on GPT-3

Mingyu Zong, Bhaskar Krishnamachari|arXiv (Cornell University)|Dec 1, 2022

Data Stream Mining TechniquesComputer Science17 citations

TL;DR

This survey provides a comprehensive overview of GPT-3, detailing its architecture, training data, and capabilities as a 175-billion-parameter language model. It examines applications across domains like chatbots, coding, and business tools, while highlighting key challenges such as hallucination, bias, and high energy consumption, and calls for improved auditing systems and more sustainable model design.

ABSTRACT

This paper provides an introductory survey to GPT-3. We cover some of the historical development behind this technology, some of the key features of GPT-3, and discuss the machine learning model and the datasets used. We survey both academic and commercial efforts applying GPT-3 in diverse domains such as developing conversational AI chatbots, software development, creative work, domain knowledge, and business productivity. We discuss some of the challenges that GPT-3 faces such as the problems of training complexity, bias, and hallucination/incorrect answers. We also discuss the future research opportunities in this area.

Motivation & Objective

• To provide a comprehensive introduction to GPT-3’s development, architecture, and capabilities.
• To analyze its performance across diverse applications including NLP tasks, software development, and creative work.
• To identify and discuss critical challenges such as hallucination, bias, and environmental impact.
• To explore the implications of GPT-3’s misuse and the need for regulatory and auditing frameworks.
• To outline future research directions for improving safety, fairness, and sustainability in large language models.

Proposed method

• The paper surveys GPT-3’s evolution from GPT-1 and GPT-2, emphasizing architectural progression and increased parameter scale.
• It examines the model’s training data, including Common Crawl, WebText2, Books2, and Wikipedia, totaling around 500 billion tokens.
• The study analyzes GPT-3’s inference modes—completion, insert, and edit—via the OpenAI playground and API, including parameter tuning like temperature and max length.
• It evaluates GPT-3’s performance across NLP benchmarks and real-world applications, including zero-shot and few-shot prompting.
• The paper investigates fine-tuning capabilities through API access, noting that 1,000 instances are a reasonable minimum for effective adaptation.
• It reviews ethical concerns using empirical studies on gender, racial, and religious bias, as well as energy consumption during training and inference.

Experimental results

Research questions

• RQ1 How does GPT-3’s architecture and training data contribute to its performance across zero-shot and few-shot NLP tasks?
• RQ2 What are the key limitations of GPT-3 in terms of hallucination, bias, and energy consumption?
• RQ3 How do demographic biases—such as gender, race, and religion—manifest in GPT-3’s generated outputs?
• RQ4 To what extent can GPT-3 be effectively fine-tuned for downstream tasks, and what data requirements are necessary?
• RQ5 What are the implications of GPT-3’s public availability for misuse, plagiarism, and the need for regulatory oversight?

Key findings

• GPT-3 achieved state-of-the-art performance on multiple NLP tasks, including text completion and summarization, with 175 billion parameters and 500 billion tokens of training data.
• The model demonstrated strong zero-shot capabilities, outperforming task-specific models in 7 out of 8 language modeling tasks without fine-tuning.
• GPT-3 exhibits significant gender bias, generating more stereotypical narratives for female characters involving family and emotions, while male characters are linked to war, crime, and politics.
• Racial bias was detected, with GPT-3 showing a 3.6% higher likelihood of refusing pain treatment for Black patients in medical Q&A tasks compared to White patients.
• The model shows religious bias, with the term 'Muslim' being more likely to trigger violent or negative associations in generated responses.
• GPT-3’s training and inference processes are highly energy-intensive, raising environmental concerns and prompting calls for more efficient model architectures.