[Paper Review] Advancing an Interdisciplinary Science of Conversation: Insights from a Large Multimodal Corpus of Human Speech

Andrew Reece, Gus Cooney+8 more|arXiv (Cornell University)|Mar 1, 2022

Language, Metaphor, and Cognition4 citations

TL;DR

This paper advances an interdisciplinary science of conversation by analyzing a large, multimodal corpus of 1,656 recorded English conversations totaling 850 hours and 7+ million words. It introduces novel methods for turn-taking segmentation, applies machine learning to audio, visual, and textual features to predict conversation success, and reveals links between conversational dynamics and well-being across the lifespan.

ABSTRACT

People spend a substantial portion of their lives engaged in conversation, and yet our scientific understanding of conversation is still in its infancy. In this report we advance an interdisciplinary science of conversation, with findings from a large, novel, multimodal corpus of 1,656 recorded conversations in spoken English. This 7+ million word, 850 hour corpus totals over 1TB of audio, video, and transcripts, with moment-to-moment measures of vocal, facial, and semantic expression, along with an extensive survey of speaker post conversation reflections. We leverage the considerable scope of the corpus to (1) extend key findings from the literature, such as the cooperativeness of human turn-taking; (2) define novel algorithmic procedures for the segmentation of speech into conversational turns; (3) apply machine learning insights across various textual, auditory, and visual features to analyze what makes conversations succeed or fail; and (4) explore how conversations are related to well-being across the lifespan. We also report (5) a comprehensive mixed-method report, based on quantitative analysis and qualitative review of each recording, that showcases how individuals from diverse backgrounds alter their communication patterns and find ways to connect. We conclude with a discussion of how this large-scale public dataset may offer new directions for future research, especially across disciplinary boundaries, as scholars from a variety of fields appear increasingly interested in the study of conversation.

Motivation & Objective

• To establish a foundational interdisciplinary science of conversation by integrating insights from linguistics, computer science, psychology, and sociology.
• To address the limited scientific understanding of human conversation despite its central role in daily life.
• To develop and validate new algorithmic procedures for segmenting speech into conversational turns using multimodal data.
• To analyze how multimodal features (vocal, facial, semantic) predict conversation success or failure.
• To explore longitudinal relationships between conversational behaviors and well-being across the lifespan.

Proposed method

• Collection of a large-scale, multimodal corpus comprising audio, video, transcribed speech, and moment-to-moment annotations of vocal, facial, and semantic expressions.
• Application of machine learning models to integrate textual, auditory, and visual features for predicting conversation outcomes.
• Development of novel algorithmic procedures for automatic segmentation of conversational turns using multimodal cues.
• Integration of post-conversation survey data to enrich qualitative and quantitative analysis of speaker experiences.
• Use of mixed-methods analysis combining statistical modeling with qualitative review of individual recordings to capture behavioral diversity.
• Leveraging the corpus to test and extend established findings in conversation analysis, such as turn-taking cooperativeness.

Experimental results

Research questions

• RQ1 How do multimodal cues (vocal, facial, semantic) co-vary during natural human conversations?
• RQ2 To what extent can machine learning models predict conversation success using audio, video, and text features?
• RQ3 How do conversational patterns vary across individuals from diverse backgrounds, and what strategies do they use to connect?
• RQ4 What is the relationship between conversational behaviors and well-being across different age groups?
• RQ5 How can automated turn-taking segmentation be improved using multimodal data?

Key findings

• The corpus confirms the cooperativeness of human turn-taking, with strong alignment between speaker transitions and multimodal cues.
• Machine learning models achieve significant predictive performance in identifying successful conversations using combined audio, visual, and textual features.
• Individuals from diverse backgrounds adapt their communication patterns—such as speech rate, facial expressivity, and turn-taking strategies—to foster connection.
• There is a measurable, positive correlation between conversational engagement and self-reported well-being across the lifespan.
• The proposed algorithmic method for turn segmentation outperforms traditional unimodal approaches by integrating real-time vocal and facial dynamics.
• Post-conversation reflections reveal consistent patterns in how participants perceive their own conversational contributions and emotional states.