[Paper Review] Modelling, Visualising and Summarising Documents with a Single Convolutional Neural Network

Misha Denil, Alban Demiraj+3 more|arXiv (Cornell University)|Jun 15, 2014

Topic ModelingComputer Science28 references91 citations

TL;DR

This paper proposes a single, unified convolutional neural network (CNN) model that jointly learns word, sentence, and document-level representations by applying hierarchical convolutional operations. The model preserves word and sentence order, enables end-to-end document classification, and supports interpretable visualizations that directly yield high-quality automatic text summarization without additional training or feature engineering.

ABSTRACT

Capturing the compositional process which maps the meaning of words to that of documents is a central challenge for researchers in Natural Language Processing and Information Retrieval. We introduce a model that is able to represent the meaning of documents by embedding them in a low dimensional vector space, while preserving distinctions of word and sentence order crucial for capturing nuanced semantics. Our model is based on an extended Dynamic Convolution Neural Network, which learns convolution filters at both the sentence and document level, hierarchically learning to capture and compose low level lexical features into high level semantic concepts. We demonstrate the effectiveness of this model on a range of document modelling tasks, achieving strong results with no feature engineering and with a more compact model. Inspired by recent advances in visualising deep convolution networks for computer vision, we present a novel visualisation technique for our document networks which not only provides insight into their learning process, but also can be interpreted to produce a compelling automatic summarisation system for texts.

Motivation & Objective

• To develop a unified deep learning model that captures compositional semantics in documents by hierarchically combining word and sentence representations.
• To preserve word and sentence order in document representations, avoiding information loss inherent in bag-of-words or n-gram models.
• To enable automatic text summarization through novel visualization techniques applied to learned convolutional filters.
• To demonstrate that a single model trained for classification can be reused for summarization and interpretability without retraining.
• To explore the potential of unsupervised pretraining for improving document representation learning.

Proposed method

• The model uses a two-level hierarchical CNN: first applying convolutional operations on word embeddings within sentences, then on sentence embeddings across the document.
• Each level employs a modified Dynamic Convolutional Neural Network (DCNN) with cascaded convolution, max-pooling, and tanh nonlinearities to learn hierarchical features.
• Tied weights across sentences ensure consistent sentence-level representation learning, promoting generalization and parameter efficiency.
• The model is trained end-to-end via backpropagation using a softmax classifier on the final document embedding.
• Visualizations of filter activations (inspired by Simonyan et al.) are used to identify salient words and sentences, forming the basis for automatic summarization.
• Summaries are generated by selecting up to 20% of the most activated sentences based on visualization maps, without additional fine-tuning.

Experimental results

Research questions

• RQ1 Can a single, unified CNN model effectively learn hierarchical representations of documents while preserving word and sentence order?
• RQ2 To what extent can visualization of internal convolutional filters in a document-level CNN provide interpretable insights into model reasoning?
• RQ3 Can the same model used for document classification be directly repurposed for automatic text summarization using activation-based attention?
• RQ4 How does the performance of this model compare to heuristic baselines (e.g., first-sentence summarization) in summarization tasks?
• RQ5 What is the potential of unsupervised pretraining for improving the quality of learned document representations in this framework?

Key findings

• The model achieves strong performance on document classification tasks without any handcrafted features, demonstrating the effectiveness of end-to-end learning with hierarchical convolutions.
• Visualizations of filter activations successfully identify semantically important words and sentences, providing interpretability into the model’s internal reasoning.
• The visualization-driven summarization method outperforms the heuristic approach of selecting only the first sentence, as shown in qualitative examples (Figure 3).
• The model generates coherent and relevant summaries by selecting up to 20% of the most salient sentences based on activation patterns.
• The same model, once trained for classification, can be used directly for summarization and visualization without retraining or architectural changes.
• The hierarchical architecture enables joint learning of word, sentence, and document representations, supporting multiple downstream tasks from a single parameterized system.