[Paper Review] Recurrent Convolutional Neural Networks for Discourse Compositionality

Nal Kalchbrenner, Phil Blunsom|arXiv (Cornell University)|Jun 15, 2013

Topic ModelingComputer Science20 references188 citations

TL;DR

This paper proposes a novel Recurrent Convolutional Neural Network (RCNN) for discourse compositionality, combining a hierarchical convolutional neural network (HCNN) for sentence-level meaning composition with a speaker-conditioned RNN for modeling discourse-level interactions. The model achieves state-of-the-art performance (73.9% accuracy) on dialogue act classification without feature engineering or pretraining, using only greedy decoding and end-to-end training.

ABSTRACT

The compositionality of meaning extends beyond the single sentence. Just as words combine to form the meaning of sentences, so do sentences combine to form the meaning of paragraphs, dialogues and general discourse. We introduce both a sentence model and a discourse model corresponding to the two levels of compositionality. The sentence model adopts convolution as the central operation for composing semantic vectors and is based on a novel hierarchical convolutional neural network. The discourse model extends the sentence model and is based on a recurrent neural network that is conditioned in a novel way both on the current sentence and on the current speaker. The discourse model is able to capture both the sequentiality of sentences and the interaction between different speakers. Without feature engineering or pretraining and with simple greedy decoding, the discourse model coupled to the sentence model obtains state of the art performance on a dialogue act classification experiment.

Motivation & Objective

• To model the compositionality of meaning beyond single sentences, extending from word-level to discourse-level meaning formation.
• To address the lack of principled, neural network-based approaches for discourse compositionality that capture both sequential structure and speaker interaction.
• To develop a unified framework that combines sentence-level semantic composition with discourse-level modeling using end-to-end trainable neural networks.
• To evaluate the model on dialogue act classification, a key task in discourse understanding, without relying on pretraining or handcrafted features.

Proposed method

• The sentence model uses a hierarchical convolutional neural network (HCNN) that applies feature-wise convolutions across word vectors, with increasing kernel sizes to capture long-range dependencies in a depth of roughly √(2l) for a sentence of length l.
• Each convolution layer applies different weights per feature but shares weights across words, preserving word order without explicit syntactic parsing.
• The discourse model extends the sentence model using a recurrent neural network (RNN) whose hidden state is conditioned on both the current sentence vector and the current speaker's identity.
• The RNN is trained to predict dialogue acts using cross-entropy loss with L2 regularization, and predictions are made via greedy decoding over the output distribution.
• Word vectors are randomly initialized (25-dimensional) without pretraining, and the model learns semantic and pragmatic representations end-to-end from dialogue act labels.
• Discourse-level representations are extracted from the final hidden state of the RCNN, which are shown to capture pragmatic similarity across dialogues.

Experimental results

Research questions

• RQ1 Can a hierarchical convolutional architecture effectively model sentence-level meaning composition without relying on syntactic structure or pretraining?
• RQ2 Can a speaker-conditioned recurrent network capture both sequential discourse structure and inter-speaker interactions in dialogue?
• RQ3 Does an end-to-end trainable RCNN model achieve state-of-the-art performance on dialogue act classification without feature engineering or pretraining?
• RQ4 To what extent do the learned discourse vector representations reflect pragmatic similarity rather than just semantic similarity?

Key findings

• The RCNN model achieved 73.9% accuracy on dialogue act classification in the Switchboard Dialogue Act Corpus, outperforming the best prior method (LM-HMM trigram at 71.0%).
• The model surpassed all baseline models, including unigram, bigram, and trigram language model HMMs, as well as a majority baseline (31.5%) and random baseline (2.4%).
• The discourse vector representations generated by the RCNN captured pragmatic similarity, as evidenced by nearest neighbor examples where semantically different utterances were grouped due to similar dialogue act roles.
• The model achieved SOTA results without any pretraining or feature engineering, relying solely on end-to-end training with cross-entropy loss and greedy decoding.
• The ablation study showed that increasing the RNN depth beyond d=2 did not improve performance, indicating that short-term context is sufficient for this task.
• The learned representations reflect pragmatic roles: words like 'repair your own car' and 'manage the money' were grouped not by semantics but by their functional role in dialogue.