[Paper Review] REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets

Angelina Wang, Alexander Liu+7 more|arXiv (Cornell University)|Apr 16, 2020

Video Surveillance and Tracking Methods4 citations

TL;DR

REVISE is an automated tool that detects and mitigates bias in visual datasets across three dimensions: object-based, person-based, and geography-based. By analyzing object frequency, scale, context, demographic representation, and geographic distribution, it surfaces actionable insights to guide dataset curation and model training, reducing downstream bias in computer vision systems.

ABSTRACT

Machine learning models are known to perpetuate and even amplify the biases present in the data. However, these data biases frequently do not become apparent until after the models are deployed. Our work tackles this issue and enables the preemptive analysis of large-scale datasets. REVISE (REvealing VIsual biaSEs) is a tool that assists in the investigation of a visual dataset, surfacing potential biases along three dimensions: (1) object-based, (2) person-based, and (3) geography-based. Object-based biases relate to the size, context, or diversity of the depicted objects. Person-based metrics focus on analyzing the portrayal of people within the dataset. Geography-based analyses consider the representation of different geographic locations. These three dimensions are deeply intertwined in how they interact to bias a dataset, and REVISE sheds light on this; the responsibility then lies with the user to consider the cultural and historical context, and to determine which of the revealed biases may be problematic. The tool further assists the user by suggesting actionable steps that may be taken to mitigate the revealed biases. Overall, the key aim of our work is to tackle the machine learning bias problem early in the pipeline. REVISE is available at https://github.com/princetonvisualai/revise-tool

Motivation & Objective

• Address the lack of systematic bias detection in large-scale visual datasets before model deployment.
• Identify under- and misrepresentations in visual data that may lead to biased model behavior.
• Provide dataset creators and users with actionable recommendations to mitigate detected biases.
• Enable proactive bias mitigation by analyzing object, person, and geographic representation in datasets.
• Support transparency and fairness in computer vision by surfacing hidden biases that manual review may miss.

Proposed method

• Analyzes visual datasets using three bias dimensions: object-based (frequency, scale, context, diversity), person-based (demographic representation, gender/skin tone annotations), and geography-based (regional representation).
• Uses object co-occurrence patterns and scene context to infer potential query-based data collection strategies.
• Generates query recommendations (e.g., 'airplane and sky' vs. 'airplane and ground') to recover images with desired object scales.
• Employs proxy metrics based on image co-occurrence to estimate original search queries when actual queries are unavailable.
• Validates query effectiveness using Flickr image search results to confirm control over object size and context.
• Integrates with real-world datasets (COCO, OpenImages, YFCC100M, BDD100K) to demonstrate practical utility.

Experimental results

Research questions

• RQ1 How can biases in visual datasets be systematically detected across object, person, and geographic dimensions?
• RQ2 To what extent do object size and context in images reflect underlying data collection biases?
• RQ3 How do demographic annotations (e.g., gender, skin tone) correlate with image scale and context, and what risks do they pose?
• RQ4 What role does geographic provenance play in shaping dataset representation, especially in user-generated content?
• RQ5 Can proxy-based query recommendations effectively recover images with desired object scales when original queries are unknown?

Key findings

• In COCO, objects like airplanes, beds, and pizzas were frequently depicted as large, likely due to underrepresentation of aerial or distant views, suggesting a need for targeted data collection.
• In OpenImages, 69% of small, low-resolution people were labeled as male by annotators, indicating a risk of propagating gender bias in downstream models.
• In YFCC100M, 47% of countries—especially in developing regions—were predominantly represented by visitor-generated photos, leading to stereotypical geographic portrayals.
• Query-based recommendations (e.g., 'pizza and classroom' vs. 'pizza and broccoli') successfully controlled object size in search results, demonstrating feasibility of bias mitigation through targeted queries.
• Object co-occurrence patterns serve as a reliable proxy for original search queries when actual queries are unavailable, enabling effective bias analysis.
• The tool enables detection of previously unnoticed biases without requiring prior knowledge of specific bias types, enhancing proactive fairness efforts.