[Paper Review] A Paradigm Shift: Detecting Human Rights Violations Through Web Images

Grigorios Kalliatakis, Shoaib Ehsan+1 more|arXiv (Cornell University)|Mar 30, 2017

Global Security and Public Health8 references3 citations

TL;DR

This paper proposes a paradigm shift in human rights monitoring by leveraging web images and deep learning to detect violations without relying on costly manual analysis. It demonstrates that while real-world images from search engines like Google and Bing are abundant, only a small fraction (e.g., 3.64% for 'refugees') are relevant, highlighting the critical need for task-specific visual concept collection and advanced computer vision techniques such as CNNs to improve detection accuracy.

ABSTRACT

The growing presence of devices carrying digital cameras, such as mobile phones and tablets, combined with ever improving internet networks have enabled ordinary citizens, victims of human rights abuse, and participants in armed conflicts, protests, and disaster situations to capture and share via social media networks images and videos of specific events. This paper discusses the potential of images in human rights context including the opportunities and challenges they present. This study demonstrates that real-world images have the capacity to contribute complementary data to operational human rights monitoring efforts when combined with novel computer vision approaches. The analysis is concluded by arguing that if images are to be used effectively to detect and identify human rights violations by rights advocates, greater attention to gathering task-specific visual concepts from large-scale web images is required.

Motivation & Objective

• To reduce reliance on expensive, trained imagery analysts in human rights monitoring by automating detection using digital images.
• To investigate the feasibility of using real-world web images—particularly from search engines—for identifying human rights violations.
• To address the challenge of low relevance in image search results for human rights keywords, which hinders dataset creation.
• To establish a foundation for developing large-scale, annotated datasets of human rights violation images using web-based sources.
• To explore the potential of deep learning, especially CNNs, in extracting high-level semantic representations for violation detection.

Proposed method

• Using Google and Bing APIs to collect images based on human rights keywords such as 'child labour', 'refugees', and 'police violence'.
• Evaluating the relevance of retrieved images by comparing them to expert-defined concepts of human rights violations.
• Applying a comparative analysis across multiple queries to assess retrieval quality and relevance ratios.
• Identifying systematic issues in keyword-based image retrieval, such as misclassification (e.g., 'armed conflict' returning military parades).
• Leveraging deep learning models, particularly Convolutional Neural Networks (CNNs), to learn high-level visual representations from web images.
• Proposing a future research direction focused on representation learning to improve detection of human rights violations in real-world images.

Experimental results

Research questions

• RQ1 Can web images from public sources like Google and Bing be effectively used to detect human rights violations?
• RQ2 What is the actual relevance rate of images retrieved using human rights-related keywords compared to general object categories?
• RQ3 Why do standard image search engines fail to return high-quality, task-specific images for human rights monitoring?
• RQ4 How can computer vision techniques, especially CNNs, be adapted to detect subtle or context-dependent human rights violations in real-world images?
• RQ5 What role can large-scale, web-sourced image collections play in reducing the cost and increasing the scalability of human rights monitoring?

Key findings

• Only a small fraction of images retrieved via search engines for human rights keywords are relevant—e.g., just 3.64% for 'refugees' on Bing and 18% on Google.
• For common object categories like 'car' or 'dog', relevance rates exceed 80%, highlighting a stark contrast with human rights terms.
• The low relevance of human rights-related image searches indicates a major challenge in data collection for training detection models.
• Misleading results are common, such as 'armed conflict' returning images of military parades rather than actual conflict events.
• Despite the challenges, real-world images hold significant potential as complementary data when combined with advanced computer vision techniques.
• The study concludes that future efforts must prioritize gathering task-specific visual concepts from large-scale web images to enable effective, scalable human rights monitoring.