[Paper Review] Predicting Audience's Laughter Using Convolutional Neural Network

Lei Chen, Chong MIn Lee|arXiv (Cornell University)|Feb 8, 2017

Humor Studies and Applications15 references19 citations

TL;DR

This paper proposes a convolutional neural network (CNN)-based approach to predict audience laughter in TED Talk transcripts, leveraging automatically learned textual features to outperform traditional handcrafted linguistic feature methods. The CNN model achieved 86.1% accuracy on a pun dataset and 58.9% on a larger, diverse TED talk corpus, demonstrating superior performance and reduced need for manual feature engineering.

ABSTRACT

For the purpose of automatically evaluating speakers' humor usage, we build a presentation corpus containing humorous utterances based on TED talks. Compared to previous data resources supporting humor recognition research, ours has several advantages, including (a) both positive and negative instances coming from a homogeneous data set, (b) containing a large number of speakers, and (c) being open. Focusing on using lexical cues for humor recognition, we systematically compare a newly emerging text classification method based on Convolutional Neural Networks (CNNs) with a well-established conventional method using linguistic knowledge. The advantages of the CNN method are both getting higher detection accuracies and being able to learn essential features automatically.

Motivation & Objective

• To develop a speaker-independent humor recognition system for real-world presentations using audience laughter as a proxy for humor.
• To address the lack of open, authentic, and diverse corpora for studying humor in spoken presentations, as existing datasets are limited to scripted jokes or TV comedy.
• To compare the performance of a deep learning-based CNN model with a conventional machine learning model using handcrafted linguistic features.
• To evaluate whether CNNs can learn effective humor-relevant representations automatically, reducing reliance on expert-designed features.
• To establish a benchmark for humor recognition in natural, non-scripted spoken discourse using a large-scale, open corpus of TED Talk transcripts.

Proposed method

• Constructed a new corpus of 1,192 TED Talk transcripts, labeling utterances as humorous if followed by '(Laughter)' markup.
• Split the corpus into training (CV set) and development (Dev set) sets while ensuring all utterances from a single talk remained in the same partition to enforce speaker independence.
• Implemented a CNN model using word embeddings with multiple filter sizes (5, 6, 7), ReLU activation, and dropout regularization (0.7 and 0.35) for regularization.
• Used the Adam optimizer with early stopping based on a 10% validation split to prevent overfitting during training.
• Compared the CNN model against a conventional Random Forest classifier trained on manually engineered linguistic features, including incongruity, ambiguity, and phonetic style.
• Optimized hyperparameters using Tree Parzen Estimation (TPE) over 200 iterations to maximize model performance on the development set.

Experimental results

Research questions

• RQ1 Can a CNN-based model outperform traditional machine learning models using handcrafted linguistic features in detecting humor from spoken presentations?
• RQ2 How does the performance of a CNN model trained on TED Talk transcripts compare to that of a conventional model in terms of accuracy, precision, and recall?
• RQ3 To what extent does the CNN model’s ability to learn representations automatically reduce the need for manual feature engineering compared to conventional approaches?
• RQ4 How does the model generalize across diverse speakers and naturalistic speech patterns in authentic presentations?
• RQ5 What is the impact of architectural choices such as filter size, dropout rate, and optimization strategy on the CNN’s humor detection performance?

Key findings

• The CNN model achieved 86.1% accuracy on the Pun dataset, significantly outperforming the conventional method’s 78.3% accuracy.
• On the TED Talk dataset, the CNN model reached 58.9% accuracy, a notable improvement over the conventional method’s 52.0% accuracy.
• The CNN model improved precision from 0.515 (conventional) to 0.582 (CNN) on the TED dataset, indicating better discrimination of true humorous instances.
• The CNN model also achieved higher F1-scores (0.606 vs. 0.595) and recall (0.632 vs. 0.705), demonstrating more balanced performance across metrics.
• The CNN model required significantly less manual feature engineering, as it automatically learned relevant representations from raw text inputs.
• Despite improvements, performance on the TED dataset remains relatively low (58.9% accuracy), indicating that humor in authentic presentations remains a challenging task for current models.