[Paper Review] Saved You A Click: Automatically Answering Clickbait Titles

Oliver Johnson, Beicheng Lou+2 more|arXiv (Cornell University)|Dec 15, 2022

Misinformation and Its Impacts4 citations

TL;DR

This paper proposes a system that automatically answers clickbait titles by fine-tuning both extractive (RoBERTa) and abstractive (T5) question-answering models on user-generated answers from Reddit's 'SavedYouAClick' and Facebook's 'StopClickbait' pages. After fine-tuning, the extractive model outperforms on ROUGE scores, while the abstractive model achieves higher BERTscores and generates more fluent, grammatically correct responses.

ABSTRACT

Often clickbait articles have a title that is phrased as a question or vague teaser that entices the user to click on the link and read the article to find the explanation. We developed a system that will automatically find the answer or explanation of the clickbait hook from the website text so that the user does not need to read through the text themselves. We fine-tune an extractive question and answering model (RoBERTa) and an abstractive one (T5), using data scraped from the 'StopClickbait' Facebook pages and Reddit's 'SavedYouAClick' subforum. We find that both extractive and abstractive models improve significantly after finetuning. We find that the extractive model performs slightly better according to ROUGE scores, while the abstractive one has a slight edge in terms of BERTscores.

Motivation & Objective

• To address the problem of time-wasting clickbait headlines by automatically generating concise answers to clickbait titles.
• To explore whether clickbait spoilers can be modeled as a closed-domain question-answering task using extractive and abstractive QA approaches.
• To evaluate the performance of fine-tuned extractive and abstractive models on a newly scraped dataset of user-generated clickbait answers.
• To compare the strengths and weaknesses of extractive versus abstractive models in generating accurate, fluent, and factually correct answers to clickbait hooks.

Proposed method

• Scraped 2,538 Reddit and 1,287 Facebook posts from 'SavedYouAClick' and 'StopClickbait' pages, forming a (context, question, answer) dataset.
• Fine-tuned an extractive QA model (RoBERTa) pre-trained on SQuAD2.0 and NewsQA using the scraped data, with no manually labeled answer spans.
• Fine-tuned an abstractive QA model (T5) on the same dataset, enabling it to generate new, paraphrased answers not necessarily present in the text.
• Used ROUGE and BERTscore metrics to evaluate both models quantitatively, with BERTscore measuring semantic similarity to reference answers.
• Applied self-supervised span detection techniques (e.g., spanBERT) to approximate answer spans for the extractive model despite missing gold spans.
• Conducted qualitative analysis on test examples to assess fluency, factual correctness, and paraphrasing accuracy of model outputs.

Experimental results

Research questions

• RQ1 Can extractive and abstractive QA models effectively answer clickbait titles using only user-generated answers from social media forums?
• RQ2 How do ROUGE and BERTscore metrics compare in evaluating the quality of answers generated by extractive versus abstractive models?
• RQ3 Does fine-tuning on user-generated clickbait spoilers significantly improve model performance on unseen clickbait titles?
• RQ4 In what scenarios do extractive models outperform abstractive models, and vice versa, particularly regarding factual consistency and fluency?
• RQ5 What are the main failure modes of both model types when the clickbait title is vague, list-based, or semantically ambiguous?

Key findings

• Both the extractive (RoBERTa) and abstractive (T5) models showed significant performance improvements after fine-tuning on the scraped Reddit and Facebook data.
• The extractive model achieved higher ROUGE-F1 scores (47.20 on Reddit fine-tuned RoBERTa-SQuAD) compared to the abstractive model, indicating better span-matching accuracy.
• The abstractive model outperformed the extractive model on BERTscore (e.g., 90.26 vs. 88.54 for RoBERTa-SQuAD Reddit), suggesting better semantic alignment with reference answers.
• The abstractive model generated more fluent, grammatically correct, and paraphrased responses, while the extractive model was more consistent in avoiding factual errors from paraphrasing.
• Both models struggled with list-based clickbait titles (e.g., '10 reasons why...') and overly vague headlines, where answers span multiple non-contiguous text segments.
• The presence of noisy or superfluous details in user-generated answers introduced data quality variance, affecting model generalization and evaluation reliability.