[Paper Review] From human-centered to social-centered artificial intelligence: Assessing ChatGPT's impact through disruptive events

Skyler Wang, Ned Cooper+1 more|arXiv (Cornell University)|May 31, 2023

Artificial Intelligence in Healthcare and EducationMedicine3 citations

TL;DR

This paper challenges the dominant human-centered framework in AI evaluation by proposing a social-centered approach to assess ChatGPT’s impact, arguing that its effects on social groups, institutions, and norms are as critical as individual-level concerns like bias or hallucination. Through analysis of disruptive events, the authors demonstrate that LLMs like ChatGPT reshape collective practices and social structures, urging technologists to adopt longitudinal, ethnographic, and participatory evaluation methods beyond individual-centric metrics.

ABSTRACT

Large language models (LLMs) and dialogue agents represent a significant shift in artificial intelligence (AI) research, particularly with the recent release of the GPT family of models. ChatGPT's generative capabilities and versatility across technical and creative domains led to its widespread adoption, marking a departure from more limited deployments of previous AI systems. While society grapples with the emerging cultural impacts of this new societal-scale technology, critiques of ChatGPT's impact within machine learning research communities have coalesced around its performance or other conventional safety evaluations relating to bias, toxicity, and "hallucination." We argue that these critiques draw heavily on a particular conceptualization of the "human-centered" framework, which tends to cast atomized individuals as the key recipients of technology's benefits and detriments. In this article, we direct attention to another dimension of LLMs and dialogue agents' impact: their effects on social groups, institutions, and accompanying norms and practices. By analyzing ChatGPT's social impact through a social-centered framework, we challenge individualistic approaches in AI development and contribute to ongoing debates around the ethical and responsible deployment of AI systems. We hope this effort will call attention to more comprehensive and longitudinal evaluation tools (e.g., including more ethnographic analyses and participatory approaches) and compel technologists to complement human-centered thinking with social-centered approaches.

Motivation & Objective

• To critique the limitations of human-centered AI evaluation frameworks that prioritize individual users over collective social effects.
• To examine how ChatGPT’s deployment disrupts social groups, institutions, and established norms, moving beyond individual-level concerns like bias or hallucination.
• To advocate for a paradigm shift in AI development and evaluation toward social-centered approaches that account for systemic and long-term social transformations.
• To call for the integration of ethnographic and participatory methods in AI assessment to capture broader societal impacts.
• To contribute to ethical AI discourse by highlighting the need for comprehensive evaluation tools that reflect real-world social dynamics.

Proposed method

• The authors employ a social-centered theoretical framework to analyze ChatGPT’s impact, shifting focus from individual users to social groups and institutions.
• They use qualitative analysis of disruptive events—such as rapid institutional adoption or societal controversies—to trace how ChatGPT alters social norms and practices.
• The study draws on existing literature and real-world case studies to illustrate shifts in educational, professional, and civic domains following ChatGPT’s release.
• The method emphasizes interpretive and critical analysis over quantitative metrics, prioritizing contextual understanding of social change.
• The authors advocate for longitudinal, ethnographic, and participatory evaluation methods as complements to conventional safety and performance assessments.
• The framework is applied to reframe common critiques of LLMs, such as hallucination and bias, as systemic social phenomena rather than isolated technical flaws.

Experimental results

Research questions

• RQ1 How does ChatGPT’s deployment affect social groups and institutions beyond individual users?
• RQ2 In what ways do disruptive events reveal the social-centered impacts of large language models like ChatGPT?
• RQ3 Why are conventional human-centered evaluation metrics insufficient for capturing the broader societal implications of generative AI?
• RQ4 How can social-centered frameworks improve the ethical and responsible development of AI systems?
• RQ5 What alternative evaluation methods are needed to assess the long-term social consequences of LLMs?

Key findings

• ChatGPT’s impact extends beyond individual users, significantly altering institutional practices in education, work, and civic engagement.
• Disruptive events such as widespread academic use or policy responses highlight systemic shifts in social norms and organizational behavior.
• The paper demonstrates that issues like hallucination and bias are not just technical flaws but reflect deeper social and institutional vulnerabilities.
• Conventional human-centered evaluations fail to capture the collective, structural transformations driven by LLMs, such as changes in knowledge production and decision-making processes.
• The authors conclude that longitudinal, ethnographic, and participatory evaluation methods are essential for understanding and governing the social impacts of AI.
• A shift toward social-centered AI evaluation is necessary to ensure ethical and responsible deployment that accounts for societal-level consequences.