[Paper Review] Text Understanding from Scratch

Xiang Zhang, Yann LeCun|arXiv (Cornell University)|Feb 5, 2015

Topic Modeling42 references421 citations

TL;DR

This paper proposes a character-level convolutional neural network (ConvNet) for text understanding that bypasses traditional linguistic structures like words and phrases, learning hierarchical representations directly from raw characters. It achieves state-of-the-art performance on multiple NLP tasks—including sentiment analysis, text categorization, and ontology classification—without any prior linguistic knowledge, demonstrating strong generalization across languages like English and Chinese.

ABSTRACT

This article demontrates that we can apply deep learning to text understanding from character-level inputs all the way up to abstract text concepts, using temporal convolutional networks (ConvNets). We apply ConvNets to various large-scale datasets, including ontology classification, sentiment analysis, and text categorization. We show that temporal ConvNets can achieve astonishing performance without the knowledge of words, phrases, sentences and any other syntactic or semantic structures with regards to a human language. Evidence shows that our models can work for both English and Chinese.

Motivation & Objective

• To demonstrate that deep learning models can perform text understanding from raw character inputs without relying on predefined linguistic structures such as words, phrases, or syntax.
• To investigate whether temporal ConvNets can learn hierarchical representations of text directly from characters, enabling end-to-end learning of abstract semantic concepts.
• To evaluate the model's performance on diverse NLP tasks, including sentiment analysis, text categorization, and ontology classification, across multiple languages.
• To show that the model generalizes across languages, including morphologically complex ones like Chinese, without language-specific engineering.

Proposed method

• The model uses 1D temporal convolutional networks (ConvNets) that process sequences of quantized characters as raw input, applying convolutions with learnable filters to extract local patterns.
• Each convolutional layer applies a sliding kernel over the input sequence with a specified stride, learning hierarchical features through stacked layers.
• Max-pooling layers are applied after convolutional layers to reduce sequence length and retain the most salient features, enabling deeper architectures.
• The final representation is pooled globally and passed through a fully connected layer for classification, with end-to-end training via backpropagation.
• The model is trained on large-scale datasets using stochastic gradient descent, with no reliance on word embeddings or linguistic preprocessing.
• For Chinese text, characters are converted to Pinyin (romanized form) with tone markers, and the resulting sequence is used as input to maintain character-level processing.

Experimental results

Research questions

• RQ1 Can a deep learning model achieve high performance in text understanding tasks without any prior knowledge of words, phrases, or syntactic structures?
• RQ2 To what extent can a character-level ConvNet learn meaningful semantic representations directly from raw character sequences?
• RQ3 Does the model generalize across different languages, including morphologically complex languages like Chinese, without language-specific feature engineering?
• RQ4 How does the performance of a character-level model compare to word-level models on standard NLP benchmarks?
• RQ5 Can the model achieve strong results on tasks like sentiment analysis and text categorization without relying on word embeddings or linguistic parsing?

Key findings

• The Large ConvNet achieved 95.12% test accuracy on the Sogou News dataset, outperforming both the Small ConvNet (91.35%) and a bag-of-words baseline (92.78%).
• On the IMDB movie review dataset, the model reached 87.0% test accuracy, demonstrating strong performance on sentiment analysis without word-level features.
• On the AG News text categorization dataset, the model achieved 92.0% test accuracy, showing effectiveness in multi-class text classification.
• The model achieved 99.14% training accuracy on Sogou News, indicating effective learning from raw character inputs without overfitting.
• The results on Chinese text using Pinyin input showed consistent performance, confirming the model’s ability to generalize across languages without linguistic preprocessing.
• The model’s success suggests that hierarchical feature learning from raw characters can replace traditional linguistic engineering in NLP tasks.