[Paper Review] Can AI Serve as a Substitute for Human Subjects in Software Engineering Research?

Marco Aurélio Gerosa, Bianca Trinkenreich+2 more|arXiv (Cornell University)|Nov 18, 2023

Ethics and Social Impacts of AI4 citations

TL;DR

This vision paper proposes using large language models (LLMs) like ChatGPT to generate synthetic qualitative data as a scalable alternative to human subjects in software engineering research. By employing persona-based prompting, multi-persona dialogues, and mega-persona surveys, LLMs can simulate diverse user perspectives, enabling efficient data collection for interviews, focus groups, and surveys—offering a promising augmentation to traditional methods, though not a full replacement due to the irreplaceable depth of human empathy and nuance.

ABSTRACT

Research within sociotechnical domains, such as Software Engineering, fundamentally requires a thorough consideration of the human perspective. However, traditional qualitative data collection methods suffer from challenges related to scale, labor intensity, and the increasing difficulty of participant recruitment. This vision paper proposes a novel approach to qualitative data collection in software engineering research by harnessing the capabilities of artificial intelligence (AI), especially large language models (LLMs) like ChatGPT. We explore the potential of AI-generated synthetic text as an alternative source of qualitative data, by discussing how LLMs can replicate human responses and behaviors in research settings. We examine the application of AI in automating data collection across various methodologies, including persona-based prompting for interviews, multi-persona dialogue for focus groups, and mega-persona responses for surveys. Additionally, we discuss the prospective development of new foundation models aimed at emulating human behavior in observational studies and user evaluations. By simulating human interaction and feedback, these AI models could offer scalable and efficient means of data generation, while providing insights into human attitudes, experiences, and performance. We discuss several open problems and research opportunities to implement this vision and conclude that while AI could augment aspects of data gathering in software engineering research, it cannot replace the nuanced, empathetic understanding inherent in human subjects in some cases, and an integrated approach where both AI and human-generated data coexist will likely yield the most effective outcomes.

Motivation & Objective

• To address the growing challenges in recruiting human participants for qualitative software engineering research, especially from underrepresented groups.
• To explore whether AI-generated synthetic text can serve as a viable alternative or supplement to human-sourced qualitative data.
• To propose practical methods for using foundation models to simulate human behaviors and responses across common qualitative research methodologies.
• To identify open problems and research opportunities in integrating AI-generated data with human-generated data in sociotechnical software engineering research.
• To advocate for an integrated approach where AI and human data coexist to enhance research scalability and validity.

Proposed method

• Using persona-based prompting to guide LLMs into generating responses that reflect specific demographic or psychographic profiles, such as non-technical users or experienced developers.
• Employing multi-persona dialogue techniques to simulate focus groups by generating responses from multiple virtual participants with distinct viewpoints.
• Creating mega-persona responses to emulate survey responses from aggregated user profiles, enabling large-scale data generation for survey-based research.
• Leveraging the few-shot and few-shot few-shot capabilities of LLMs to fine-tune responses for consistency, style, and behavioral authenticity.
• Designing prompts that systematically vary to elicit responses reflecting different attitudes, motivations, and experiences, such as OSS contributor motivations.
• Using statistical distributions from real contributor surveys (e.g., from 23.1% with <3 years of experience) to inform and ground synthetic response distributions.

Figure 1: Using prompt engineering in a large language model to interview a specific persona. The conversation was generated using GPT-4.

Experimental results

Research questions

• RQ1 Can LLMs generate synthetic qualitative data that accurately reflects the perspectives, motivations, and behaviors of real human subjects in software engineering research?
• RQ2 How can persona-based prompting be used to simulate diverse user profiles in virtual interviews and focus groups?
• RQ3 To what extent can AI-generated responses replicate the variability and nuance of real human responses in surveys and observational studies?
• RQ4 What are the limitations of AI-generated data in capturing empathetic, context-sensitive, and culturally nuanced human experiences?
• RQ5 How can AI-generated and human-sourced qualitative data be meaningfully integrated to improve research scalability and validity?

Key findings

• LLMs can generate synthetic responses that closely mimic human perspectives, such as estimating that 40% of long-term OSS contributors might strongly agree that they contribute because proprietary software cannot solve certain problems.
• The method of persona-based prompting enables consistent simulation of user types, such as estimating that 50% of contributors with less than 3 years of experience might strongly agree that they contribute to improve their skills.
• Multi-persona dialogues can simulate group dynamics in focus groups, with responses reflecting diverse viewpoints across experience levels and motivations.
• Mega-persona responses can generate survey-like data with statistically plausible distributions, such as estimating 30% of contributors might somewhat agree with a given statement and 15% remain neutral.
• The approach shows promise for scaling qualitative data collection while maintaining relevance to real-world user behaviors and motivations.
• Despite these capabilities, the study concludes that AI cannot fully replace human subjects due to the lack of genuine empathy and contextual depth in synthetic responses.

Figure 2: Tweaking the persona to interviewing a woman contributor. The conversation was generated using GPT-4.