[Paper Review] Evolving Character-level Convolutional Neural Networks for Text Classification

Trevor Londt, Xiaoying Gao+2 more|arXiv (Cornell University)|Dec 3, 2020

Evolutionary Algorithms and Applications32 references4 citations

TL;DR

This paper proposes a surrogate-based evolutionary deep learning algorithm using genetic programming to automatically evolve optimal character-level convolutional neural network (char-CNN) architectures for text classification. The method outperforms five manually designed CNNs and one LSTM model in accuracy while reducing training time via surrogate models, revealing that architectural width and early-stage feature richness enhance performance.

ABSTRACT

Character-level convolutional neural networks (char-CNN) require no knowledge of the semantic or syntactic structure of the language they classify. This property simplifies its implementation but reduces its classification accuracy. Increasing the depth of char-CNN architectures does not result in breakthrough accuracy improvements. Research has not established which char-CNN architectures are optimal for text classification tasks. Manually designing and training char-CNNs is an iterative and time-consuming process that requires expert domain knowledge. Evolutionary deep learning (EDL) techniques, including surrogate-based versions, have demonstrated success in automatically searching for performant CNN architectures for image analysis tasks. Researchers have not applied EDL techniques to search the architecture space of char-CNNs for text classification tasks. This article demonstrates the first work in evolving char-CNN architectures using a novel EDL algorithm based on genetic programming, an indirect encoding and surrogate models, to search for performant char-CNN architectures automatically. The algorithm is evaluated on eight text classification datasets and benchmarked against five manually designed CNN architecture and one long short-term memory (LSTM) architecture. Experiment results indicate that the algorithm can evolve architectures that outperform the LSTM in terms of classification accuracy and five of the manually designed CNN architectures in terms of classification accuracy and parameter count.

Motivation & Objective

• To address the lack of optimal char-CNN architectures for text classification, which are typically manually designed and time-consuming to tune.
• To overcome the limitations of fixed-depth, shallow char-CNNs that fail to improve accuracy with increased depth.
• To automate the search for performant char-CNN architectures using evolutionary computation, reducing reliance on expert knowledge and costly trial-and-error.
• To evaluate the effectiveness of surrogate models and indirect encoding in accelerating the evolution of valid, trainable char-CNNs.
• To investigate architectural properties—such as width, depth, and path density—that influence char-CNN performance.

Proposed method

• A novel genetic programming-based algorithm, SurDG-EC, is used to evolve char-CNN architectures through indirect encoding, ensuring all generated networks are structurally valid and trainable.
• An indirect encoding scheme represents architectures via a compact genotype, enabling efficient exploration of the vast architecture space.
• Surrogate models are employed to predict validation accuracy of candidate networks without full training, drastically reducing computational cost.
• The evolutionary process uses fitness evaluation via backpropagation on a subset of data, with selection, crossover, and mutation to evolve better-performing phenotypes.
• Phenotype evaluation includes analysis of path density, parameter count, depth, and activation maps to understand architectural impact on performance.
• The algorithm is benchmarked across eight text classification datasets against five manually designed CNNs and one LSTM architecture.

Experimental results

Research questions

• RQ1 Can an evolutionary deep learning approach automatically discover high-performing char-CNN architectures without manual design or domain expertise?
• RQ2 What architectural characteristics—such as depth, width, or path density—contribute most to improved classification accuracy in char-CNNs?
• RQ3 How does the use of surrogate models affect the efficiency and effectiveness of evolving char-CNNs compared to full training?
• RQ4 To what extent does architectural width, especially in early layers, improve the performance of char-CNNs?
• RQ5 Can genealogical analysis of evolved networks reveal patterns or design principles that explain improved performance?

Key findings

• The SurDG-EC algorithm evolved char-CNN architectures that outperformed a state-of-the-art VDCNN-29-Kmax model in terms of classification accuracy on multiple datasets.
• The evolved models achieved comparable or better accuracy than five manually designed CNNs while using fewer parameters, indicating improved parameter efficiency.
• There is strong evidence that increased width in early layers enhances performance, suggesting architectural breadth is a key factor in improving generalization.
• Phenotypes with fewer than 25 distinct paths between input and output layers showed higher accuracy, indicating that excessive branching reduces performance.
• The depth of the network positively correlates with accuracy up to approximately ten cells (20 convolutional layers), after which performance degrades, suggesting a performance ceiling for depth without skip connections.
• Activation visualization revealed sparse, selective neuron responses in high-performing models, suggesting that efficient, sparse representations may be a hallmark of effective char-CNNs.