[Paper Review] Hearing through Vibrations: Perception of Musical Emotions by Profoundly Deaf People

Anastasia Schmitz, Catherine Holloway+1 more|arXiv (Cornell University)|Dec 24, 2020

Multisensory perception and integration53 references5 citations

TL;DR

This study investigates how profoundly deaf individuals perceive musical emotions through vibrotactile feedback using a backpack-style haptic device. Findings show that participants accurately identified happy and angry music as more arousing, while sad and peaceful excerpts were harder to distinguish, offering design guidelines for accessible music experiences for the deaf community.

ABSTRACT

Advances in tactile-audio feedback technology have created new possibilities for deaf people to feel music. However, little is known about deaf individuals' perception of musical emotions through vibrotactile feedback. In this paper, we present the findings from a mixed-methods study with 16 profoundly deaf participants. The study protocol was designed to explore how users of a backpack-style vibrotactile display perceive intended emotions in twenty music excerpts. Quantitative analysis demonstrated that participants correctly identified happy and angry excerpts and rated them as more arousing than sad and peaceful excerpts. More positive emotions were experienced during happy compared to angry excerpts while peaceful and sad excerpts were hard to be differentiated. Based on qualitative data, we highlight the benefits and limitations of using vibrations to convey musical emotions to profoundly deaf users. Finally, we provide guidelines for designing accessible music experiences for the deaf community.

Motivation & Objective

• To investigate how profoundly deaf individuals perceive emotional content in music through vibrotactile feedback.
• To identify which musical emotions are most reliably conveyed via vibration-based haptic systems.
• To understand the perceptual differences between positive (happy, angry) and negative (sad, peaceful) emotions in vibrotactile music experiences.
• To develop design guidelines for accessible, emotion-rich music interactions tailored to the deaf community.

Proposed method

• Conducted a mixed-methods study with 16 profoundly deaf participants using a backpack-style vibrotactile display.
• Presented 20 music excerpts representing four emotional categories: happy, angry, sad, and peaceful.
• Collected quantitative ratings on arousal and emotional valence for each excerpt.
• Used qualitative interviews to explore user experiences, preferences, and challenges in interpreting emotional content through vibrations.
• Analyzed quantitative data using statistical methods to assess emotion recognition accuracy and arousal ratings.
• Synthesized findings into actionable design guidelines for haptic music interfaces.

Experimental results

Research questions

• RQ1 Which musical emotions (happy, angry, sad, peaceful) are most accurately perceived by profoundly deaf individuals through vibrotactile feedback?
• RQ2 How do arousal levels differ in the perception of happy, angry, sad, and peaceful music excerpts via vibration?
• RQ3 What are the perceived differences in emotional experience between positive (happy, angry) and negative (sad, peaceful) emotions in vibrotactile music perception?
• RQ4 What are the strengths and limitations of vibrotactile feedback in conveying musical emotions, based on user experience?

Key findings

• Participants correctly identified happy and angry music excerpts with higher accuracy than sad or peaceful ones.
• Happy and angry excerpts were rated as significantly more arousing than sad and peaceful excerpts.
• Positive emotions were more strongly experienced during happy music compared to angry music, indicating emotional valence differences in perception.
• Sad and peaceful excerpts were frequently confused, suggesting overlapping vibrotactile patterns or insufficient differentiation in the haptic mapping.
• Qualitative feedback revealed that users found vibrations effective for detecting rhythm and intensity but struggled with nuanced emotional distinctions.
• The study identified key design challenges in mapping complex emotional content to vibrotactile signals, especially for low-arousal emotions.