[Paper Review] Fine-Grained Analysis of Propaganda in News Articles

Giovanni Da San Martino, Seunghak Yu+3 more|arXiv (Cornell University)|Oct 6, 2019

Topic Modeling23 references95 citations

TL;DR

This paper introduces a fine-grained fragment-level approach to propaganda detection, annotating 18 techniques, creating a large annotated corpus, proposing an evaluation metric, and presenting a multi-granularity neural network that outperforms strong BERT baselines.

ABSTRACT

Propaganda aims at influencing people's mindset with the purpose of advancing a specific agenda. Previous work has addressed propaganda detection at the document level, typically labelling all articles from a propagandistic news outlet as propaganda. Such noisy gold labels inevitably affect the quality of any learning system trained on them. A further issue with most existing systems is the lack of explainability. To overcome these limitations, we propose a novel task: performing fine-grained analysis of texts by detecting all fragments that contain propaganda techniques as well as their type. In particular, we create a corpus of news articles manually annotated at the fragment level with eighteen propaganda techniques and we propose a suitable evaluation measure. We further design a novel multi-granularity neural network, and we show that it outperforms several strong BERT-based baselines.

Motivation & Objective

• Motivate the need for fragment-level propaganda analysis rather than document-level labeling.
• Create a high-quality corpus with expert annotations of 18 propaganda techniques at the fragment level.
• Propose an evaluation measure that accommodates partial overlaps and varying fragment lengths.
• Develop a multi-granularity neural network that leverages lower-granularity signals to improve higher-granularity predictions.
• Demonstrate that the proposed model outperforms strong BERT-based baselines on fragment- and sentence-level tasks.

Proposed method

• Define 18 journalistic propaganda techniques suitable for fragment-level annotation.
• Assemble and annotate a corpus of 451 news articles (350k tokens) with fragment-level labels for techniques.
• Propose a partial-overlap aware evaluation metric inspired by plagiarism detection and NER-based ideas.
• Develop a multi-granularity network that uses lower-granularity signals (sentence-level) to inform higher-granularity predictions (fragment-level).
• Fine-tune BERT-based baselines (BERT, BERT-Joint, BERT-Grain) and compare with the proposed multi-granularity network.
• Evaluate on two tasks: SLC (sentence-level classification) and FLC (fragment-level classification) with a tailored loss and gating mechanism.

Experimental results

Research questions

• RQ1 Can we reliably detect and label propaganda fragments at a fine-grained level in news articles?
• RQ2 Does a multi-granularity architecture that leverages sentence-level signals improve fragment-level propaganda detection over standard BERT baselines?
• RQ3 How effective is the proposed evaluation measure at rewarding partial overlaps and varying fragment lengths?

Key findings

• The corpus contains 7,485 propaganda technique instances across 21,230 sentences (35.2%).
• The most frequent techniques are loaded language (2,547 instances) and name calling/labeling (1,294 instances).
• The proposed multi-granularity network (MGN) outperforms BERT-based baselines on fragment-level detection, especially when using a gating mechanism (Sigmoid or ReLU).
• For fragment-level detection, MGN with Sigmoid achieves P=24.42, R=21.05, F1=22.58 on full-task evaluation using spans+labels, with higher precision than baselines.
• For sentence-level detection, MGN yields substantial gains over BERT baselines, raising recall by 8.42% and F1 by 3.24 points in the all-propaganda setting.
• The study demonstrates that incorporating lower-granularity signals can meaningfully improve higher-granularity tasks, while negative gating reduces noisy negative samples.