[Paper Review] Conversational AI-Powered Design: ChatGPT as Designer, User, and Product

A. Baki Kocaballı|arXiv (Cornell University)|Feb 15, 2023

Artificial Intelligence in Healthcare and Education30 citations

TL;DR

The paper investigates ChatGPT's capabilities in a human-centered design process by simulating designer, user, and product roles across a hypothetical design project, reporting mostly appropriate responses with some drawbacks.

ABSTRACT

The recent advancements in Large Language Models (LLMs), particularly conversational LLMs like ChatGPT, have prompted changes in a range of fields, including design. This study aims to examine the capabilities of ChatGPT in a human-centered design process. To this end, a hypothetical design project was conducted, where ChatGPT was utilized to generate personas, simulate interviews with fictional users, create new design ideas, simulate usage scenarios and conversations between an imaginary prototype and fictional users, and lastly evaluate user experience. The results show that ChatGPT effectively performed the tasks assigned to it as a designer, user, or product, providing mostly appropriate responses. The study does, however, highlight some drawbacks such as forgotten information, partial responses, and a lack of output diversity. The paper explains the potential benefits and limitations of using conversational LLMs in design, discusses its implications, and suggests directions for future research in this rapidly evolving area.

Motivation & Objective

• Motivate the study by examining how conversational LLMs influence design practice.
• Evaluate ChatGPT’s ability to generate personas, conduct user interviews, brainstorm ideas, simulate usage scenarios, and evaluate user experience.
• Identify benefits and limitations of using conversational LLMs for design tasks in terms of response quality and consistency.

Proposed method

• Conduct a hypothetical design project using ChatGPT in multiple design roles (designer, user, product).
• Have ChatGPT generate personas and simulate interviews with fictional users.
• Have ChatGPT create new design ideas and simulate usage scenarios and conversations between a prototype and fictional users.
• Have ChatGPT evaluate user experience for the imagined product.
• Analyze the quality and limitations of ChatGPT’s outputs across tasks.

Experimental results

Research questions

• RQ1 Can ChatGPT effectively perform design tasks across designer, user, and product roles in a human-centered process?
• RQ2 What are the strengths and limitations of using conversational LLMs for generating personas, conducting interviews, ideating, simulating interactions, and evaluating UX?
• RQ3 To what extent do ChatGPT outputs align with appropriate responses and useful design insights in a hypothetical project?

Key findings

• ChatGPT effectively performed assigned tasks as designer, user, or product, providing mostly appropriate responses.
• Drawbacks include forgotten information, partial responses, and a lack of output diversity.
• The study discusses potential benefits and limitations of conversational LLMs for design and outlines implications and future research directions.