[Paper Review] Humor Detection: A Transformer Gets the Last Laugh

Orion Weller, Kevin Seppi|arXiv (Cornell University)|Aug 31, 2019

Humor Studies and Applications18 references4 citations

TL;DR

This paper proposes a Transformer-based model for humor detection that leverages Reddit upvote scores as weak supervision to classify jokes as humorous or not. Trained on ~16,000 Reddit-labeled jokes, the model achieves state-of-the-art performance with 98.6% F1 on the Short Jokes dataset and 93.1% on the Puns dataset, outperforming prior CNN-based methods and approaching human-level agreement on Reddit-specific humor.

ABSTRACT

Much previous work has been done in attempting to identify humor in text. In this paper we extend that capability by proposing a new task: assessing whether or not a joke is humorous. We present a novel way of approaching this problem by building a model that learns to identify humorous jokes based on ratings gleaned from Reddit pages, consisting of almost 16,000 labeled instances. Using these ratings to determine the level of humor, we then employ a Transformer architecture for its advantages in learning from sentence context. We demonstrate the effectiveness of this approach and show results that are comparable to human performance. We further demonstrate our model's increased capabilities on humor identification problems, such as the previously created datasets for short jokes and puns. These experiments show that this method outperforms all previous work done on these tasks, with an F-measure of 93.1% for the Puns dataset and 98.6% on the Short Jokes dataset.

Motivation & Objective

• To develop a machine learning model that identifies humorous jokes using real-world user-uploaded ratings from Reddit.
• To investigate whether a Transformer architecture can outperform previous neural network models in humor detection tasks.
• To evaluate the model’s generalization across diverse joke types, including puns and short one-liners.
• To compare model predictions with human consensus on humor, particularly distinguishing between Reddit users and general audiences.
• To demonstrate the effectiveness of self-attention mechanisms in capturing subtle linguistic cues for humor.

Proposed method

• The model uses a pre-trained BERT-based Transformer encoder fine-tuned on a dataset of 15,984 Reddit jokes labeled by upvote count.
• Jokes with upvotes ≥ 200 are classified as humorous; those with < 200 are labeled not humorous, creating a binary classification task.
• The model is trained and evaluated on three variants: joke body only, punchline only, and full joke (body + punchline).
• Transfer learning is applied using three benchmark datasets: Short Jokes (231k jokes), Pun of the Day (16k puns), and Reddit’s own joke dataset.
• Human evaluation was conducted via Amazon Mechanical Turk with 199 participants rating 30 jokes each, using majority vote to determine ground truth.
• The model’s performance is compared against a CNN with Highway layers and human performance on the same tasks.

Experimental results

Research questions

• RQ1 Can a Transformer-based model effectively learn to detect humor using weak supervision from Reddit upvote scores?
• RQ2 How does the model’s performance on humor detection compare to prior state-of-the-art models, particularly CNN-based approaches?
• RQ3 Does the model generalize well to diverse joke types such as puns and short one-liners?
• RQ4 How do predictions from the model compare to human consensus, especially when distinguishing between Reddit users and general audiences?
• RQ5 Which part of a joke—body or punchline—carries more weight in determining perceived humor?

Key findings

• The Transformer model achieved a 98.6% F1 score on the Short Jokes dataset, representing an 8-percentage-point improvement over the previous best CNN-based method.
• On the Pun of the Day dataset, the model achieved 93.1% accuracy, a 4-percentage-point improvement over the best prior CNN model.
• The model achieved 72.4% accuracy on the Reddit test set, outperforming the CNN baseline (high 60s) and approaching the 66.3% agreement rate of general human raters.
• All models, including humans, relied more heavily on the punchline than the body for humor classification, indicating the punchline carries greater predictive weight.
• The model’s performance on the Reddit dataset suggests it learns the specific humor preferences of the r/Jokes community, indicating audience-specific humor detection is feasible.
• The results demonstrate that self-attention mechanisms in Transformers are highly effective at capturing the linguistic features that define humor, especially in short, context-dependent jokes.