[Paper Review] Ah, that's the great puzzle: On the Quest of a Holistic Understanding of the Harms of Recommender Systems on Children

Robin Ungruh, Maria Soledad Pera|arXiv (Cornell University)|May 3, 2024

Ethics and Legal Issues in Pediatric HealthcareMedicine3 citations

TL;DR

This position paper argues for a holistic, multidisciplinary approach to understanding and mitigating the harms of recommender systems (RS) on children, emphasizing that RS—designed primarily for adults—often expose children to developmentally inappropriate or harmful content due to lack of child-specific design, maturity ratings, and algorithmic transparency. The key contribution is a call to action for researchers, practitioners, and policymakers to collaboratively develop child-centered RS frameworks that prioritize well-being through nuanced, context-aware harm detection beyond traditional maturity ratings.

ABSTRACT

Children come across various media items online, many of which are selected by recommender systems (RS) primarily designed for adults. The specific nature of the content selected by RS to display on online platforms used by children - although not necessarily targeting them as a user base - remains largely unknown. This raises questions about whether such content is appropriate given children's vulnerable stages of development and the potential risks to their well-being. In this position paper, we reflect on the relationship between RS and children, emphasizing the possible adverse effects of the content this user group might be exposed to online. As a step towards fostering safer interactions for children in online environments, we advocate for researchers, practitioners, and policymakers to undertake a more comprehensive examination of the impact of RS on children - one focused on harms. This would result in a more holistic understanding that could inform the design and deployment of strategies that would better suit children's needs and preferences while actively mitigating the potential harm posed by RS; acknowledging that identifying and addressing these harms is complex and multifaceted.

Motivation & Objective

• To examine the risks posed by adult-designed recommender systems (RS) to children’s digital well-being due to mismatched content curation.
• To highlight the limitations of current maturity rating systems in capturing subtle or non-explicit harms such as stereotypes, misinformation, or developmental inappropriateness.
• To call for a multidisciplinary, systems-level approach to identifying and mitigating RS-related harms across diverse domains (e.g., video, music, social media).
• To advocate for stakeholder collaboration among researchers, developers, designers, and policymakers in creating RS that prioritize children’s developmental needs and safety.
• To position the need for a comprehensive framework that accounts for children’s cognitive development, content domain, and algorithmic behavior in RS decision-making.

Proposed method

• Conducts a critical reflection on existing RS architectures and their deployment in platforms popular with children, such as YouTube Kids, Netflix, and TikTok.
• Analyzes the role of RS in shaping children’s content discovery, especially in platforms with limited search functionality for minors.
• Evaluates the limitations of human-curated maturity ratings (e.g., MPA, TVPG) in capturing non-explicit harms like stereotyping, misinformation, or psychological impact.
• Explores the challenges of automated harm detection in user-generated content platforms due to volume, diversity, and context sensitivity.
• Proposes a multi-dimensional framework for assessing RS harms by integrating child development stages, content domain, and algorithmic behavior.
• Advocates for collaborative, cross-domain research involving technical, psychological, and policy perspectives to build responsible RS for children.

Experimental results

Research questions

• RQ1 How do adult-oriented recommender systems expose children to developmentally inappropriate or harmful content, even when not explicitly targeting them?
• RQ2 To what extent do current maturity rating systems fail to capture subtle or non-explicit harms such as stereotypes, misinformation, or psychological effects in media content?
• RQ3 How does the domain of content (e.g., video games vs. movies vs. social media) influence the perceived and actual harm of recommended items to children?
• RQ4 What are the key gaps in algorithmic transparency and child-specific design that prevent effective harm mitigation in RS for children?
• RQ5 How can researchers, practitioners, and policymakers collaboratively develop a holistic, multidisciplinary framework for assessing and mitigating RS-related harms on children?

Key findings

• Recommender systems frequently expose children to content not intended for their developmental stage, even when platforms claim to offer child-safe environments.
• Maturity ratings, while widely used, are insufficient for capturing non-explicit harms such as gender stereotypes, misinformation, or psychological manipulation in media.
• The lack of standardized, automated harm detection mechanisms for user-generated content (e.g., on YouTube or TikTok) severely limits scalable safety measures for children.
• Children’s cognitive and emotional development varies significantly, making age-based ratings alone inadequate for assessing content appropriateness or harm potential.
• The domain of content—such as interactive video games versus passive video consumption—can significantly alter the perceived and actual harm of recommended media.
• There is a critical need for a multidisciplinary, systems-level approach that integrates developmental psychology, algorithmic transparency, and policy design to create safer RS for children.