[Paper Review] TextNAS: A Neural Architecture Search Space tailored for Text Representation

Yujing Wang, Yaming Yang+8 more|arXiv (Cornell University)|Dec 23, 2019

Topic Modeling40 references4 citations

TL;DR

TextNAS proposes a customized neural architecture search space for text representation that integrates convolutional, recurrent, pooling, and self-attention layers in a multi-path ensemble structure. Using the ENAS algorithm, it discovers architectures that outperform state-of-the-art models on text classification and natural language inference tasks, achieving SOTA performance with strong transferability.

ABSTRACT

Learning text representation is crucial for text classification and other language related tasks. There are a diverse set of text representation networks in the literature, and how to find the optimal one is a non-trivial problem. Recently, the emerging Neural Architecture Search (NAS) techniques have demonstrated good potential to solve the problem. Nevertheless, most of the existing works of NAS focus on the search algorithms and pay little attention to the search space. In this paper, we argue that the search space is also an important human prior to the success of NAS in different applications. Thus, we propose a novel search space tailored for text representation. Through automatic search, the discovered network architecture outperforms state-of-the-art models on various public datasets on text classification and natural language inference tasks. Furthermore, some of the design principles found in the automatic network agree well with human intuition.

Motivation & Objective

• To address the challenge of designing optimal text representation networks, which remain non-trivial despite diverse architectures in the literature.
• To demonstrate that the search space is a critical human prior in NAS, not just the search algorithm.
• To design a macro-level search space tailored for text that supports layer mixture and multi-path ensembles.
• To enable automatic discovery of high-performing, transferable architectures for text classification and NLI tasks.
• To validate that automatically discovered architectures align with human intuition and outperform manually designed baselines.

Proposed method

• Design a macro search space based on a directed acyclic graph (DAG) that supports convolutional, recurrent, pooling, and self-attention layers.
• Integrate multi-path ensembling to allow different types of layers to act as distinct feature extractors.
• Use the ENAS reinforcement learning algorithm with validation accuracy as the reward signal to search for optimal architectures.
• Perform grid search on the best architecture to tune hyperparameters on the validation set.
• Train the final architecture from scratch using combined training and validation data with the best hyperparameters.
• Transfer the discovered architecture to other text tasks without retraining the search process.

Experimental results

Research questions

• RQ1 Can a customized search space significantly improve NAS performance in text representation compared to generic or micro-level search spaces?
• RQ2 Does a macro search space with multi-path ensembling outperform micro search spaces in text classification tasks?
• RQ3 Can automatically discovered architectures via TextNAS achieve state-of-the-art performance on downstream text classification and NLI tasks?
• RQ4 To what extent do the automatically discovered architectures reflect human-designed design principles?
• RQ5 How transferable are the discovered architectures across different text datasets and tasks?

Key findings

• TextNAS achieved state-of-the-art test accuracy on all text classification datasets except Sogou, with a 66.56% accuracy on the YAHOO dataset, outperforming prior methods.
• On the SNLI and MNLI NLI datasets, TextNAS achieved 87.4% accuracy on SNLI and 74.9% on MNLI-matched, significantly outperforming the 24-layer Transformer (85.2% and 70.4% respectively).
• The macro search space outperformed the micro search space on all evaluated text classification tasks, with gains of up to 4.55% on the AMZ dataset.
• The automatically discovered architecture demonstrated strong transferability, achieving SOTA results on multiple downstream tasks without retraining the search process.
• The architecture design principles discovered by TextNAS align well with human intuition, such as combining CNN for local features, RNN for sequential modeling, and self-attention for long-range dependencies.
• The search process was efficient and scalable, enabling discovery on smaller datasets followed by transfer to larger ones with consistent performance gains.