[Paper Review] An Overview on Language Models: Recent Developments and Outlook

Chengwei Wei, Yun-Cheng Wang+2 more|arXiv (Cornell University)|Mar 10, 2023

Topic Modeling210 references19 citations

TL;DR

This paper surveys conventional and pre-trained language models across linguistic units, architectures, training methods, evaluation, and applications, and discusses future directions in the pre-trained era.

ABSTRACT

Language modeling studies the probability distributions over strings of texts. It is one of the most fundamental tasks in natural language processing (NLP). It has been widely used in text generation, speech recognition, machine translation, etc. Conventional language models (CLMs) aim to predict the probability of linguistic sequences in a causal manner, while pre-trained language models (PLMs) cover broader concepts and can be used in both causal sequential modeling and fine-tuning for downstream applications. PLMs have their own training paradigms (usually self-supervised) and serve as foundation models in modern NLP systems. This overview paper provides an introduction to both CLMs and PLMs from five aspects, i.e., linguistic units, architectures, training methods, evaluation methods, and applications. Furthermore, we discuss the relationship between CLMs and PLMs and shed light on the future directions of language modeling in the pre-trained era.

Motivation & Objective

• Introduce CLMs and PLMs and clarify their relationship and distinctions.
• Survey linguistic units, architectures, training methods, evaluation methods, and applications of language models.
• Discuss future research directions, including large and efficient language models.

Proposed method

• Define CLMs as auto-regressive models with causal prediction.
• Survey alternative LM families (structural, bidirectional, permutation) and tokenization schemes (characters, words/subwords, phrases, sentences).
• Review architectures (N-gram, maximum entropy, neural nets, RNNs, Transformers) and their evolution.
• Outline pre-training objectives (token prediction, MLM, next-sentence, text infilling, etc.) and fine-tuning/prompts strategies (fine-tuning, adapter tuning, prompt tuning).
• Discuss evaluation methods (intrinsic perplexity, PLL/PPPL for bidirectional LMs, extrinsic benchmarks like GLUE/SuperGLUE) and evaluation beyond single metrics (efficiency, bias, robustness).

Experimental results

Research questions

• RQ1 How do CLMs and PLMs differ in training, causality, and token representation?
• RQ2 What are the key LM architectures and tokenization choices, and how do they impact performance?
• RQ3 What pre-training objectives and fine-tuning/prompting strategies enable effective downstream transfer?
• RQ4 How are language models evaluated intrinsically and extrinsically, and what is their relationship?
• RQ5 What are the future directions and challenges for language models in the pre-trained era?

Key findings

• PLMs dominate modern NLP as foundation models learned from large unlabeled corpora and fine-tuned for tasks.
• Bidirectional and permutation-based LMs offer alternatives to strict causal prediction and enable flexible representations.
• Subword tokenization (BPE, WordPiece) mitigates OOV issues and balances vocabulary size with expressivity.
• Transformers enable long-range dependencies and parallel training, with encoder, decoder, and encoder-decoder variants suited to different tasks.
• Pre-training objectives, fine-tuning strategies, and prompting methods shape downstream performance and task adaptability.
• Beyond traditional metrics, efficiency, bias, and robustness are critical considerations in LM evaluation.