[Paper Review] Duluth at SemEval-2017 Task 6: Language Models in Humor Detection

Xinru Yan, Ted Pedersen|arXiv (Cornell University)|Apr 27, 2017

Humor Studies and Applications5 references3 citations

TL;DR

This paper presents a language model-based approach for humor detection in social media, using N-gram models trained on both tweet and news corpora to rank humor likelihood. The system achieved strong performance, particularly with trigram models on news data in post-evaluation runs, outperforming tweet-trained models and demonstrating that corpus choice significantly impacts humor detection accuracy.

ABSTRACT

This paper describes the Duluth system that participated in SemEval-2017 Task 6 #HashtagWars: Learning a Sense of Humor. The system participated in Subtasks A and B using N-gram language models, ranking highly in the task evaluation. This paper discusses the results of our system in the development and evaluation stages and from two post-evaluation runs.

Motivation & Objective

• To develop a system that learns a sense of humor from social media data using statistical language models.
• To investigate whether language models trained on diverse corpora can effectively detect humorous content in short-form text.
• To compare the performance of N-gram language models (bigram, trigram) trained on tweet data versus general news data for humor detection.
• To evaluate the impact of preprocessing choices such as filtering, tokenization, and case sensitivity on model performance.
• To understand why certain hashtags yielded significantly better results than others, particularly #BreakUpIn5Words.

Proposed method

• Trained bigram and trigram language models using KenLM on two corpora: humorous tweets from #HashtagWars and general news data from the News Commentary and News Crawl corpora.
• Applied modified Kneser-Ney smoothing with back-off and no pruning to improve out-of-vocabulary word handling.
• Preprocessed tweets by filtering URLs, @mentions, and #hashtags, and tokenized by whitespace and punctuation.
• Computed log probability scores for each tweet using both tweet-trained and news-trained language models to assess deviation from 'common language'.
• Used lower probability scores from the news language model as a proxy for humor, based on the hypothesis that humorous content diverges from standard language patterns.
• Ranked tweets for Subtask B and compared pairs for Subtask A based on their log probability scores from the language models.

Experimental results

Research questions

• RQ1 How effective are N-gram language models in detecting humor in short social media texts?
• RQ2 Does training language models on general news corpora outperform training on humorous tweet corpora for humor detection?
• RQ3 How do different preprocessing strategies—such as filtering @mentions and #hashtags, tokenization, and case sensitivity—affect model performance?
• RQ4 Why did the system achieve significantly better results on the hashtag #BreakUpIn5Words compared to other hashtags?
• RQ5 Does the choice between bigram and trigram models significantly impact humor detection performance?

Key findings

• In the development phase, trigram models trained on tweet data achieved the highest Subtask B F1 score of 0.887, while bigram models performed slightly better on Subtask A with an accuracy of 0.548.
• During the official evaluation, models trained on news data outperformed those trained on tweet data, particularly in post-evaluation runs where news-based trigram models achieved the best results.
• Contrary to development results, bigram language models performed better than trigram models in the evaluation and post-evaluation stages, suggesting that simpler models may generalize better on unseen data.
• The system achieved a Subtask A accuracy of 0.913 and a Subtask B distance score of 0.636 on the hashtag #BreakUpIn5Words, outperforming all other systems on this specific hashtag.
• The performance gap between tweet- and news-trained models suggests that the size and diversity of the training corpus significantly influence model effectiveness, with larger news corpora yielding better results despite lower domain similarity.
• The study indicates that corpus choice and model complexity (bigram vs. trigram) are more influential than model type, and that future work should explore unigram and character-level models.