[Paper Review] Hollywood Identity Bias Dataset: A Context Oriented Bias Analysis of Movie Dialogues

Sandhya Singh, Prapti Roy+9 more|arXiv (Cornell University)|May 31, 2022

Authorship Attribution and Profiling4 citations

TL;DR

This paper introduces the Hollywood Identity Bias Dataset (HIBD), a novel, context-aware, dialogue-level dataset of 35 Hollywood movie scripts annotated for seven identity bias categories—gender, race/ethnicity, religion, age, occupation, LGBTQ, and other—along with sensitivity, sentiment, emotion, and rationale. The dataset contains 49,117 sentences with 1,181 biased instances (2.5% skew), enabling high-precision bias detection via expert-validated, context-rich annotations, setting a foundation for AI-driven bias mitigation in scriptwriting.

ABSTRACT

Movies reflect society and also hold power to transform opinions. Social biases and stereotypes present in movies can cause extensive damage due to their reach. These biases are not always found to be the need of storyline but can creep in as the author's bias. Movie production houses would prefer to ascertain that the bias present in a script is the story's demand. Today, when deep learning models can give human-level accuracy in multiple tasks, having an AI solution to identify the biases present in the script at the writing stage can help them avoid the inconvenience of stalled release, lawsuits, etc. Since AI solutions are data intensive and there exists no domain specific data to address the problem of biases in scripts, we introduce a new dataset of movie scripts that are annotated for identity bias. The dataset contains dialogue turns annotated for (i) bias labels for seven categories, viz., gender, race/ethnicity, religion, age, occupation, LGBTQ, and other, which contains biases like body shaming, personality bias, etc. (ii) labels for sensitivity, stereotype, sentiment, emotion, emotion intensity, (iii) all labels annotated with context awareness, (iv) target groups and reason for bias labels and (v) expert-driven group-validation process for high quality annotations. We also report various baseline performances for bias identification and category detection on our dataset.

Motivation & Objective

• To address the lack of domain-specific, context-aware datasets for identifying social biases in movie scripts, particularly at the dialogue level.
• To mitigate the risk of controversial or harmful content in films by enabling early detection of biases during script development.
• To support the development of AI models that can flag identity-based biases (e.g., gender, race, LGBTQ) before production, reducing legal and reputational risks.
• To provide a high-quality, expert-validated annotation framework that captures not only bias presence but also its nature (explicit/implicit), target group, and rationale.
• To establish a benchmark for bias detection and category classification in cinematic dialogue using a diverse, real-world dataset.

Proposed method

• The dataset was constructed from 35 Hollywood movie scripts, with dialogue turns annotated at the sentence level for seven identity bias categories: gender, race/ethnicity, religion, age, occupation, LGBTQ, and other.
• Each bias was labeled as explicit or implicit, with the target group and rationale for the bias annotation recorded to ensure context awareness and interpretability.
• Additional annotations were applied for sensitivity, sentiment (positive/negative), emotion, and emotion intensity to enrich the contextual understanding of each utterance.
• A multi-stage expert-driven validation process was employed, involving domain experts and NLP specialists, to ensure high annotation quality and consistency.
• Baseline models were trained on the dataset for bias identification and category detection using state-of-the-art NLP architectures, including BERT and RoBERTa, fine-tuned on the annotated data.
• The dataset was curated with attention to class imbalance, acknowledging the 2.5% bias prevalence (1,181 biased sentences out of 49,117), to support robust evaluation of bias detection models.

Experimental results

Research questions

• RQ1 How prevalent are identity-based biases in Hollywood movie dialogues, and what are their most common forms and categories?
• RQ2 To what extent can context-aware annotations improve the accuracy and interpretability of bias detection in cinematic dialogue?
• RQ3 How do explicit and implicit biases differ in linguistic markers and target group representation across identity categories?
• RQ4 Can fine-tuned deep learning models achieve human-level performance in detecting identity biases in movie scripts using this new dataset?
• RQ5 What is the impact of multi-label, context-rich annotations (including sentiment, emotion, and rationale) on bias detection performance?

Key findings

• The Hollywood Identity Bias Dataset (HIBD) contains 49,117 dialogue sentences from 35 Hollywood movie scripts, with 1,181 instances (2.5%) annotated as biased, highlighting the challenge of data skew in bias detection.
• The dataset captures biases across seven identity categories—gender, race/ethnicity, religion, age, occupation, LGBTQ, and other—encompassing forms such as body shaming and personality bias.
• Each bias annotation includes explicit/implicit labeling, target group identification, and a rationale, enabling context-aware and interpretable analysis.
• The dataset supports multi-task learning with annotations for sentiment, emotion, and emotion intensity, enhancing the richness of contextual understanding.
• Baseline models fine-tuned on HIBD achieve competitive performance in bias detection and category classification, demonstrating the dataset’s utility for training and evaluating NLP systems.
• The expert-validated annotation pipeline ensures high-quality, reliable data, making HIBD a reliable benchmark for future research on bias detection in cinematic text.