[Paper Review] Heuristic Feature Selection for Clickbait Detection

Matti Wiegmann, Michael Völske+3 more|arXiv (Cornell University)|Feb 4, 2018

Sentiment Analysis and Opinion Mining3 references3 citations

TL;DR

This paper proposes a heuristic wrapper method called leave-many-out feature selection to optimize feature subsets for clickbait detection in a regression task. By iteratively removing feature groups and measuring performance impact via mean squared error, the approach improves the baseline model’s performance by 20%, achieving second place in the Clickbait Challenge 2017 despite using a traditional classifier instead of deep learning.

ABSTRACT

We study feature selection as a means to optimize the baseline clickbait detector employed at the Clickbait Challenge 2017. The challenge's task is to score the "clickbaitiness" of a given Twitter tweet on a scale from 0 (no clickbait) to 1 (strong clickbait). Unlike most other approaches submitted to the challenge, the baseline approach is based on manual feature engineering and does not compete out of the box with many of the deep learning-based approaches. We show that scaling up feature selection efforts to heuristically identify better-performing feature subsets catapults the performance of the baseline classifier to second rank overall, beating 12 other competing approaches and improving over the baseline performance by 20%. This demonstrates that traditional classification approaches can still keep up with deep learning on this task.

Motivation & Objective

• To improve the performance of a baseline clickbait regression model that underperforms relative to deep learning approaches.
• To address the challenge of sparse features in social media data, where traditional filter and embedding methods fail.
• To develop a heuristic wrapper method effective for high-dimensional, sparse clickbait feature spaces.
• To identify high-impact features and potential feature interactions for future engineered feature design.
• To demonstrate that traditional machine learning models can remain competitive with deep learning in clickbait detection through intelligent feature selection.

Proposed method

• The method employs a leave-many-out strategy, iteratively removing groups of features and measuring the resulting change in mean squared error (MSE).
• Performance is evaluated using ridge regression as the base model, with MSE serving as the fitness function for feature subset selection.
• Feature categories include character n-grams, word n-grams, word list features, and engineered features such as word count and punctuation frequency.
• The approach systematically ranks features by their impact on prediction error, identifying discriminative, confusing, and redundant features.
• High-impact features are analyzed to suggest new engineered features, such as counts of words ending in -ly or -ing, based on their discriminatory power.
• The method is applied to the Clickbait Challenge 2017 dataset, which contains 19,538 labeled tweets for training and 18,979 for testing.

Experimental results

Research questions

• RQ1 Can heuristic wrapper feature selection outperform standard filter methods like χ² in a sparse, regression-based clickbait detection task?
• RQ2 Which feature categories contribute most significantly to predicting clickbaitiness scores?
• RQ3 Can feature interactions and redundant features be identified through iterative removal of feature groups?
• RQ4 To what extent can engineered features improve model performance compared to raw feature sets?
• RQ5 Can traditional machine learning models achieve competitive results with deep learning in clickbait detection through intelligent feature selection?

Key findings

• The leave-many-out feature selection method improved the baseline model’s performance by 20% in terms of mean squared error, achieving second place in the Clickbait Challenge 2017.
• The engineered features, particularly 'part of day' and 'number of dots', showed the highest impact, with scores of 4.74 and 7.18 respectively in the leave-many-out analysis.
• The 'number of words ending in -ly or -ing' was identified as a promising candidate for new engineered features due to high discriminatory power of individual n-grams.
• Character-level features such as '@' and '.' were highly discriminatory, with scores of 0.73 and 0.53 respectively, while '!' was also highly impactful at 0.82.
• The word list feature 'Easy Words' had a strong positive impact (score: 2.60), suggesting lexical simplicity is a key indicator of clickbait.
• The method successfully identified that some features, like '@' in character n-grams, were highly discriminatory even when the corresponding word-level feature ('number of mentions') was not.