[Paper Review] Seeing Things from a Different Angle: Discovering Diverse Perspectives about Claims

Sihao Chen, Daniel Khashabi+3 more|arXiv (Cornell University)|Jun 8, 2019

Topic Modeling48 references7 citations

TL;DR

This paper introduces the task of substantiated perspective discovery, proposing a new NLP challenge to identify diverse, evidence-supported viewpoints on controversial claims. The authors create Perspective, a high-quality dataset of 907 claims, 11,164 perspectives, and 8,092 evidence paragraphs using online debates, search augmentation, and crowdsourcing, showing that human performance far exceeds state-of-the-art models, highlighting a key challenge for future NLP research.

ABSTRACT

One key consequence of the information revolution is a significant increase and a contamination of our information supply. The practice of fact checking won't suffice to eliminate the biases in text data we observe, as the degree of factuality alone does not determine whether biases exist in the spectrum of opinions visible to us. To better understand controversial issues, one needs to view them from a diverse yet comprehensive set of perspectives. For example, there are many ways to respond to a claim such as "animals should have lawful rights", and these responses form a spectrum of perspectives, each with a stance relative to this claim and, ideally, with evidence supporting it. Inherently, this is a natural language understanding task, and we propose to address it as such. Specifically, we propose the task of substantiated perspective discovery where, given a claim, a system is expected to discover a diverse set of well-corroborated perspectives that take a stance with respect to the claim. Each perspective should be substantiated by evidence paragraphs which summarize pertinent results and facts. We construct PERSPECTRUM, a dataset of claims, perspectives and evidence, making use of online debate websites to create the initial data collection, and augmenting it using search engines in order to expand and diversify our dataset. We use crowd-sourcing to filter out noise and ensure high-quality data. Our dataset contains 1k claims, accompanied with pools of 10k and 8k perspective sentences and evidence paragraphs, respectively. We provide a thorough analysis of the dataset to highlight key underlying language understanding challenges, and show that human baselines across multiple subtasks far outperform ma-chine baselines built upon state-of-the-art NLP techniques. This poses a challenge and opportunity for the NLP community to address.

Motivation & Objective

• To address selection bias in information retrieval by enabling systems to surface diverse, well-supported perspectives on controversial claims.
• To formalize the task of substantiated perspective discovery as a core NLP challenge involving stance detection, perspective relevance, and evidence grounding.
• To create a high-quality, scalable dataset that captures nuanced perspectives and their supporting evidence for controversial claims.
• To identify key language understanding challenges in perspective diversity, stance alignment, and evidence relevance that current NLP models struggle to resolve.

Proposed method

• Construct the Perspective dataset using online debate forums as seed data, then expand with search engine results and paraphrasing to increase diversity and coverage.
• Use crowdsourcing to filter noise, validate perspective-stance alignment, and verify evidence relevance, ensuring high data quality.
• Design five subtasks: perspective relevance, stance classification, perspective equivalence, evidence extraction, and end-to-end perspective discovery.
• Train supervised baselines using retrieval (IR) and BERT-based models for each subtask, comparing performance against human-annotated gold standards.
• Apply strict quality control to avoid filtering based on content intent (e.g., offensive language), preserving diversity while ensuring reliability.
• Focus on semantic relations between claims, perspectives, and evidence, abstracting away from source credibility and factual truth for methodological clarity.

Experimental results

Research questions

• RQ1 How can we systematically identify a diverse yet comprehensive set of perspectives on a given controversial claim, each supported by relevant evidence?
• RQ2 What are the core NLP challenges in determining the stance of a perspective relative to a claim, and in identifying whether two perspectives express equivalent ideas?
• RQ3 To what extent can current NLP models, including BERT, match human-level performance in recognizing relevant perspectives and supporting evidence for a claim?
• RQ4 How do retrieval-based and neural-based methods compare in identifying the most relevant perspectives and evidence paragraphs for a given claim?
• RQ5 What are the key bottlenecks in current NLP systems when it comes to understanding nuanced, context-dependent perspectives and their grounding in factual evidence?

Key findings

• Human performance on the perspective relevance task reached 72.5% F1, significantly outperforming the best machine baseline (50.8% F1) using IR + BERT.
• On stance classification, humans achieved 90.9% F1, while the best model (BERT) reached only 70.8%, indicating a large performance gap.
• For perspective equivalence, the human F1 score was 83.7%, compared to 63.7% for the BERT-based baseline, showing strong limitations in semantic equivalence detection.
• Evidence extraction performance was highest for humans (60.7% F1), with IR + BERT achieving 55.7%, and IR alone at 46.8%, highlighting the difficulty of long-context evidence matching.
• The overall system performance (end-to-end perspective discovery) reached 40.0% F1 with human annotation, compared to only 17.5% with IR + BERT, underscoring the gap in holistic reasoning.
• Despite using state-of-the-art models, all machine baselines significantly underperform human annotators across all subtasks, revealing a critical challenge for NLP in understanding complex, diverse perspectives.