[Paper Review] Exploring Transformers in Natural Language Generation: GPT, BERT, and XLNet

M. Onat Topal, Anil Bas+1 more|arXiv (Cornell University)|Feb 16, 2021

Topic ModelingComputer Science57 citations

TL;DR

The paper surveys three major Transformer-based models (GPT, BERT, XLNet) and their impact on natural language generation, highlighting advantages over RNN/LSTM architectures and potential applications from poetry to summarization.

ABSTRACT

Recent years have seen a proliferation of attention mechanisms and the rise of Transformers in Natural Language Generation (NLG). Previously, state-of-the-art NLG architectures such as RNN and LSTM ran into vanishing gradient problems; as sentences grew larger, distance between positions remained linear, and sequential computation hindered parallelization since sentences were processed word by word. Transformers usher in a new era. In this paper, we explore three major Transformer-based models, namely GPT, BERT, and XLNet, that carry significant implications for the field. NLG is a burgeoning area that is now bolstered with rapid developments in attention mechanisms. From poetry generation to summarization, text generation derives benefit as Transformer-based language models achieve groundbreaking results.

Motivation & Objective

• Explain why Transformers overcome vanishing gradients and sequential processing limits in NLG.
• Compare GPT, BERT, and XLNet and their implications for text generation tasks.
• Highlight how attention mechanisms enable advances in poetry generation, summarization, and related NLG tasks.

Proposed method

• Discuss the evolution of attention mechanisms and the rise of Transformer-based NLG models.
• Describe how GPT, BERT, and XLNet influence text generation through their architectures.
• Argue the implications of Transformer-based models for parallelization and long-range dependencies.

Experimental results

Research questions

• RQ1 What are the key characteristics of GPT, BERT, and XLNet that impact natural language generation?
• RQ2 How do Transformer-based models address limitations of RNNs/LSTMs in NLG tasks?
• RQ3 What are the practical implications of these models for applications like poetry generation and summarization?

Key findings

• Transformer-based models bring significant implications for natural language generation.
• Attention mechanisms enable improved parallelization and handling of long-range dependencies.
• There are broad applications in poetry generation, summarization, and other NLG tasks discussed in the context of these models.