[Paper Review] ChatGPT and Other Large Language Models as Evolutionary Engines for Online Interactive Collaborative Game Design

Pier Luca Lanzi, Daniele Loiacono|arXiv (Cornell University)|Feb 9, 2023

Artificial Intelligence in GamesComputer Science47 references11 citations

TL;DR

The paper presents an online framework that uses interactive evolution driven by large language models (LLMs) to collaboratively design game concepts as free-form text, evaluated by human participants. It evaluates the approach via three tasks with about 80 designers using Telegram for feedback.

ABSTRACT

Large language models (LLMs) have taken the scientific world by storm, changing the landscape of natural language processing and human-computer interaction. These powerful tools can answer complex questions and, surprisingly, perform challenging creative tasks (e.g., generate code and applications to solve problems, write stories, pieces of music, etc.). In this paper, we present a collaborative game design framework that combines interactive evolution and large language models to simulate the typical human design process. We use the former to exploit users' feedback for selecting the most promising ideas and large language models for a very complex creative task - the recombination and variation of ideas. In our framework, the process starts with a brief and a set of candidate designs, either generated using a language model or proposed by the users. Next, users collaborate on the design process by providing feedback to an interactive genetic algorithm that selects, recombines, and mutates the most promising designs. We evaluated our framework on three game design tasks with human designers who collaborated remotely.

Motivation & Objective

• Demonstrate that LLMs can implement evolutionary operators (random initialization, crossover, mutation) for free-form textual game ideas.
• Enable real-time, online collaboration between humans and LLMs to evolve design concepts.
• Assess the feasibility and user experience of a Telegram-based interface for collecting evaluations.
• Evaluate the framework across multiple design tasks and participant groups to validate generality and practicality.

Proposed method

• Represent game concepts as free-form text and manage them in an online population and database.
• Use an interactive evolutionary algorithm with tournament selection, crossover, and mutation implemented via prompts to an LLM.
• Publish candidate designs to users via Telegram or web interfaces and collect qualitative three-valued feedback (positive/neutral/negative).
• Employ LLMs (ChatGPT or DaVinci GPT-3) to implement genetic operators for textual ideas.
• Run multiple experiments with fixed durations to mimic structured design sessions and a Global Game Jam scenario.
• Analyze evaluations, idea lengths, and emergent mechanics to assess creativity and coherence.

Experimental results

Research questions

• RQ1 Can LLMs effectively implement genetic operators (random initialization, crossover, mutation) for free-form text ideas in an interactive evolutionary setting?
• RQ2 Does a Telegram-based online feedback loop support collaborative game design and yield novel emergent mechanics?
• RQ3 What are the qualitative strengths and limitations (e.g., coherence, novelty) of LLM-driven design iterations in a real-world setting?
• RQ4 How does the framework perform across different design tasks (board game, video game) and event formats (workshop vs. Global Game Jam)?

Key findings

• The framework supported around 80 designers across three design tasks with a four-day workflow for two tasks and a shorter Global Game Jam session.
• The board game task produced longer concepts and a shift toward ecosystem-maintenance mechanics over iterations, indicating emergent novelty.
• The video game task yielded novel mechanics such as color manipulation and light/reflection interactions, showing emergent experimentation.
• Participants reported emergent novelty and diverse narratives as positives, while coherence and redundancy due to small population size were noted as negatives.
• Evaluations totaled 799 for board game concepts and 1025 for video game concepts across the two main experiments; the Global Game Jam session was shorter with fewer evaluations.
• There was no observed performance difference between ChatGPT and DaVinci GPT-3 in this study; both served as viable LLM options for the operators.