[Paper Review] Quantifying Controversy in Social Media

Kiran Garimella, Gianmarco De Francisci Morales+2 more|arXiv (Cornell University)|Jul 18, 2015

Social Media and Politics36 references4 citations

TL;DR

This paper proposes a domain-agnostic, graph-based framework to quantify controversy in social media by constructing conversation graphs from user interactions (e.g., retweets) and applying a three-stage pipeline: graph construction, partitioning to identify opposing viewpoints, and measuring controversy using random-walk-based metrics. The key contribution is the introduction of the RWC (Random Walk Controversy) measure, which outperforms existing methods in distinguishing controversial from non-controversial topics, particularly on Twitter, with content features proving largely ineffective.

ABSTRACT

Which topics spark the most heated debates on social media? Identifying those topics is not only interesting from a societal point of view, but also allows the filtering and aggregation of social media content for disseminating news stories. In this paper, we perform a systematic methodological study of controversy detection by using the content and the network structure of social media. Unlike previous work, rather than study controversy in a single hand-picked topic and use domain specific knowledge, we take a general approach to study topics in any domain. Our approach to quantifying controversy is based on a graph-based three-stage pipeline, which involves (i) building a conversation graph about a topic; (ii) partitioning the conversation graph to identify potential sides of the controversy; and (iii) measuring the amount of controversy from characteristics of the graph. We perform an extensive comparison of controversy measures, different graph-building approaches, and data sources. We use both controversial and non-controversial topics on Twitter, as well as other external datasets. We find that our new random-walk-based measure outperforms existing ones in capturing the intuitive notion of controversy, and show that content features are vastly less helpful in this task.

Motivation & Objective

• To develop a general-purpose, domain-agnostic method for detecting and quantifying controversy in social media without relying on domain-specific knowledge or curated datasets.
• To systematically evaluate a wide range of controversy measures, graph construction methods, and data sources across diverse topics, including both controversial and non-controversial ones.
• To identify which structural, sentiment, and content-based features are most predictive of controversy in social media conversations.
• To establish a reliable, scalable framework for measuring controversy that can be applied to real-world systems and used to mitigate echo chambers and filter bubbles.

Proposed method

• Construct a conversation graph where nodes represent users and edges represent interactions such as retweets, mentions, or replies related to a topic defined by a text query (e.g., a hashtag on Twitter).
• Apply graph partitioning algorithms—specifically METIS and correlation clustering with negative edges—to identify clusters of users with opposing viewpoints.
• Develop and evaluate multiple controversy measures, including conductance, modularity, assortativity, and a novel random-walk-based measure (RWC), to quantify the degree of polarization.
• Use the RWC measure, which computes the ratio of expected hitting times between random walks in the graph, to assess controversy based on the separation between opposing clusters.
• Evaluate all measures using a large-scale dataset of Twitter conversations and external datasets, with human-annotated labels for controversy as a gold standard.
• Compare the performance of content-based features (e.g., keywords, sentiment) against structural features, finding that structural features—especially retweet-based endorsements—yield significantly better results.

Experimental results

Research questions

• RQ1 Can controversy in social media be reliably quantified using a general, domain-agnostic approach without relying on hand-curated datasets or domain-specific knowledge?
• RQ2 Which graph-based features—structural, sentiment, or content-based—are most effective in distinguishing controversial from non-controversial topics?
• RQ3 How do different controversy measures, including conductance, modularity, and RWC, perform in capturing the intuitive notion of controversy across diverse topics?
• RQ4 To what extent do content-based features such as keywords and sentiment contribute to controversy detection, and can they be reliably used in isolation?
• RQ5 Does the proposed RWC measure generalize across different data sources and domains, and does it outperform existing state-of-the-art measures?

Key findings

• The random-walk-based controversy measure (RWC) significantly outperforms all other evaluated measures in distinguishing controversial from non-controversial topics, demonstrating superior separation in empirical evaluation.
• Content-based features such as keywords and sentiment are vastly less effective in detecting controversy, with sentiment analysis showing only modest promise, while structural features like retweets are far more predictive.
• Graph partitioning using METIS with retweet edges yields the most reliable identification of opposing viewpoints, outperforming methods based on mentions or random partitions.
• Traditional measures like modularity and assortativity fail to correlate with human-annotated controversy scores, indicating they are poor indicators of polarization in this context.
• The RWC measure generalizes well across different datasets and domains, including Twitter and external social media data, confirming its robustness and scalability.
• Simple structural measures such as cut size or conductance do not generalize across graphs of varying sizes and densities, limiting their utility in large-scale systems.