[Paper Review] Improvised Comedy as a Turing Test

Kory W. Mathewson, Piotr Mirowski|arXiv (Cornell University)|Nov 23, 2017

Artificial Intelligence in Games7 references17 citations

TL;DR

This paper proposes using improvised comedy as a real-world Turing test to evaluate human-like AI performance in live theater. An AI chatbot (A.L.Ex) with speech recognition and synthesis, embodied in a humanoid robot, performed alongside human actors in 30 shows; audiences frequently failed to distinguish AI from human improvisers, demonstrating strong suspension of disbelief and emotional engagement.

ABSTRACT

The best improvisational theatre actors can make any scene partner, of any skill level or ability, appear talented and proficient in the art form, and thus "make them shine". To challenge this improvisational paradigm, we built an artificial intelligence (AI) trained to perform live shows alongside human actors for human audiences. Over the course of 30 performances to a combined audience of almost 3000 people, we have refined theatrical games which involve combinations of human and (at times, adversarial) AI actors. We have developed specific scene structures to include audience participants in interesting ways. Finally, we developed a complete show structure that submitted the audience to a Turing test and observed their suspension of disbelief, which we believe is key for human/non-human theatre co-creation.

Motivation & Objective

• To investigate whether audiences can reliably distinguish AI-generated improvisational dialogue from human performance in live theater.
• To explore how emotional engagement and narrative immersion are affected when audiences believe an AI is a co-creator in a theatrical performance.
• To evaluate the role of physical embodiment (e.g., humanoid robots) in enhancing audience suspension of disbelief in human-AI co-creation.
• To assess the impact of audience expectations and prior knowledge on their perception of AI authenticity in real-time performance.

Proposed method

• The AI improviser, named A.L.Ex, is a sequence-to-sequence LSTM-based neural network trained on 100k film subtitles with a 50k-word vocabulary.
• A.L.Ex uses speech recognition and text-to-speech systems to enable real-time spoken dialogue with human performers.
• The system employs heuristics for turn-taking, sentence selection, and sentiment analysis to generate contextually appropriate responses.
• A humanoid robot (by EZ-Robot) physically embodies A.L.Ex to enhance audience focus and presence during performances.
• A Wizard-of-Oz setup was used in select shows to deceive audiences into believing a non-human AI was controlling dialogue, later revealing the human operator.
• Performances included audience suggestions for scene context, with scenes lasting 3–6 minutes and ending naturally by human performers.

Experimental results

Research questions

• RQ1 To what extent can audiences reliably detect that an AI is controlling a theatrical performance in real time?
• RQ2 How does the presence of a humanoid robot affect audience immersion and belief in the AI’s agency during improvisational scenes?
• RQ3 What role does emotional connection to the AI character play in sustaining audience suspension of disbelief?
• RQ4 How do audience perceptions shift when explicitly informed that an AI is performing versus when deceived into believing it is autonomous?
• RQ5 What factors contribute to the illusion of intelligence and emotional understanding in AI-generated dialogue during live improvisation?

Key findings

• In a Turing test-like setup, nearly all audience members incorrectly identified the AI as human when not forewarned, indicating strong illusion of intelligence.
• Approximately half the audience believed the AI was performing flawlessly in a Wizard-of-Oz scenario, even when the system was controlled by a human operator.
• Audiences frequently suspended disbelief and formed emotional attachments to the AI character, treating it as a believable narrative partner.
• The introduction of a humanoid robot with a human-like voice significantly increased audience immersion and perceived authenticity of the AI performance.
• Post-show discussions revealed that audiences trusted performers’ claims about AI control, suggesting that narrative framing and performer credibility heavily influence perception.
• The study highlights the public’s limited awareness of current AI capabilities, underscoring the need for clearer communication about AI progress and limitations.