[Paper Review] Recent Trends in Deep Learning Based Natural Language Processing

Tom Young, Devamanyu Hazarika+2 more|arXiv (Cornell University)|Aug 9, 2017

Topic ModelingComputer Science156 references1,254 citations

TL;DR

This survey reviews major deep learning models and methods for NLP, including word embeddings, CNNs, RNNs, contextualized embeddings, attention, memory, reinforcement learning, and unsupervised approaches, outlining their evolution and future directions.

ABSTRACT

Deep learning methods employ multiple processing layers to learn hierarchical representations of data and have produced state-of-the-art results in many domains. Recently, a variety of model designs and methods have blossomed in the context of natural language processing (NLP). In this paper, we review significant deep learning related models and methods that have been employed for numerous NLP tasks and provide a walk-through of their evolution. We also summarize, compare and contrast the various models and put forward a detailed understanding of the past, present and future of deep learning in NLP.

Motivation & Objective

• Motivate the shift from shallow models to deep learning in NLP by highlighting limitations of hand-crafted features and high-dimensional sparse representations.
• Summarize distributed representations and their impact on NLP performance across tasks.
• Review core DL architectures (CNNs, RNNs, recursive nets) and their NLP applications, including memory and attention mechanisms.
• Discuss recent trends in unsupervised pre-training, contextualized embeddings, and transfer learning for NLP tasks.
• Provide a forward-looking perspective on how these trends shape the future of NLP research and practice.

Proposed method

• Describe the distributed representation paradigm and foundations of word embeddings.
• Explain Word2Vec (CBOW and Skip-gram) and their impact on semantic similarity and compositionality.
• Discuss character embeddings and their benefits for morphologically rich languages and OOV handling.
• Introduce contextualized embeddings (ELMo) and the shift from global to context-sensitive word representations.
• Summarize pre-trained language models and transfer learning (OpenAI-GPT, BERT) and their downstream impact.
• Review CNN, RNN, and recursive architectures for various NLP tasks and their evolution, including attention and memory-augmented approaches.

Experimental results

Research questions

• RQ1 What are the major deep learning models and architectures that have shaped NLP tasks?
• RQ2 How have contextualized embeddings and pre-trained language models influenced performance and transfer learning in NLP?
• RQ3 What are the benefits and limitations of CNNs, RNNs, and recursive networks for different NLP tasks?
• RQ4 What future directions and open challenges are suggested by current trends in DL for NLP?

Key findings

• Word embeddings (Word2Vec, GloVe) established distributed representations that underpin many NLP tasks.
• Contextualized embeddings (ELMo, later BERT/transformers) provide sentence- and task-specific representations that improve performance.
• CNNs pioneered effective n-gram feature extraction for sentence modeling and various NLP tasks.
• RNNs and variants (LSTM, GRU) enable modeling of sequential dependencies in language data.
• Pre-trained language models and transfer learning (OpenAI-GPT, BERT) enable unsupervised pre-training and fine-tuning for diverse NLP tasks.
• The survey highlights memory-augmented models, attention mechanisms, reinforcement learning, and unsupervised sentence representation learning as important trends.