[Paper Review] Accuracy and Political Bias of News Source Credibility Ratings by Large Language Models

Kai‐Cheng Yang, Filippo Menczer|arXiv (Cornell University)|Apr 1, 2023

Misinformation and Its Impacts41 citations

TL;DR

ChatGPT can rate news outlet credibility and its ratings moderately align with human experts (rho = 0.54); it achieves AUC = 0.89 for binary classification and can handle multilingual and satirical domains at low cost (~$3 for 7,523 domains).

ABSTRACT

Search engines increasingly leverage large language models (LLMs) to generate direct answers, and AI chatbots now access the Internet for fresh data. As information curators for billions of users, LLMs must assess the accuracy and reliability of different sources. This paper audits nine widely used LLMs from three leading providers -- OpenAI, Google, and Meta -- to evaluate their ability to discern credible and high-quality information sources from low-credibility ones. We find that while LLMs can rate most tested news outlets, larger models more frequently refuse to provide ratings due to insufficient information, whereas smaller models are more prone to making errors in their ratings. For sources where ratings are provided, LLMs exhibit a high level of agreement among themselves (average Spearman's $ρ= 0.79$), but their ratings align only moderately with human expert evaluations (average $ρ= 0.50$). Analyzing news sources with different political leanings in the US, we observe a liberal bias in credibility ratings yielded by all LLMs in default configurations. Additionally, assigning partisan roles to LLMs consistently induces strong politically congruent bias in their ratings. These findings have important implications for the use of LLMs in curating news and political information.

Motivation & Objective

• Evaluate whether ChatGPT can rate credibility of a large set of news outlets.
• Quantify alignment between ChatGPT ratings and human expert judgments (Lin et al., MBFC, NewsGuard).
• Assess performance across non-English and satirical domains.
• Discuss implications for using LLMs in misinformation research and media literacy.

Proposed method

• Compile a subset of 7,523 news domains from the Tranco popular list and prompt ChatGPT to rate credibility on a 0–1 scale in a zero-shot setting.
• Use the OpenAI API gpt-3.5-turbo-0301 with temperature 0 and extra JSON-format instructions to obtain domain ratings.
• Process ~7,523 domains via five concurrent processes, taking ~2 hours at a cost of ~$3.
• Rescale human expert ratings (Lin et al. aggregate, MBFC, NewsGuard) to 0–1 for comparison.
• Evaluate correlations with Spearman ρ and assess binary classification performance using AUC and F1 scores.

Experimental results

Research questions

• RQ1 Can ChatGPT rate the credibility of a wide set of news outlets in a zero-shot setting?
• RQ2 How well do ChatGPT ratings correlate with human expert judgments across languages and satirical sources?
• RQ3 Can ChatGPT ratings function as an effective classifier to identify low-credibility outlets?
• RQ4 How does performance differ between English and non-English domains, including satirical sites?

Key findings

• ChatGPT rated 7,282 of 7,523 domains; 241 domains yielded errors due to lack of information.
• ChatGPT ratings show a moderate correlation with human expert ratings (Spearman ρ = 0.54, p < 0.001).
• When compared to binary labels from NewsGuard and MBFC, ChatGPT achieves an AUC of 0.89.
• Using a threshold near 0.5 yields best F1 scores (approximately 0.73 with NewsGuard, 0.63 with MBFC).
• Correlations vary by language; English outlets show notable correlations with NewsGuard (e.g., ρ ≈ 0.51) and MBFC (ρ ≈ 0.60 total), while non-English outlets also show significant correlations (e.g., MBFC non-English ρ ≈ 0.65; Italian ρ ≈ 0.38).
• ChatGPT demonstrates some ability to identify satire websites (77.4% recognition in the MBFC satire list) and can ground responses with contextual justifications.