[Paper Review] Body, Clothes, Water, and Toys: Media Towards Natural Music Expressions with Digital Sounds

Kenji Mase, Tomoko Yonezawa|arXiv (Cornell University)|Oct 4, 2020

Interactive and Immersive Displays4 references4 citations

TL;DR

This paper presents novel musical interfaces using everyday materials—body, clothing, water, and stuffed toys—leveraging sensors like touch, image processing, and accelerometers to enable expressive, intuitive digital sound control. The key contribution is a suite of low-cost, multimodal instruments that enhance natural music expression through intimate, tangible interaction with familiar media.

ABSTRACT

In this paper, we introduce our research challenges for creating new musical instruments using everyday-life media with intimate interfaces, such as the self-body, clothes, water and stuffed toys. Various sensor technologies including image processing and general touch sensitive devices are employed to exploit these interaction media. The focus of our effort is to provide user-friendly and enjoyable experiences for new music and sound performances. Multimodality of musical instruments is explored in each attempt. The degree of controllability in the performance and the richness of expressions are also discussed for each installation.

Motivation & Objective

• To explore the potential of everyday physical media as expressive interfaces for digital music.
• To address the challenge of creating intuitive, enjoyable, and controllable musical experiences using non-traditional materials.
• To investigate how multimodal feedback and sensor integration enhance user expressiveness in digital sound performance.
• To develop low-cost, accessible instruments that support rich, natural musical expression through tangible interaction.

Proposed method

• Utilized touch sensors, image processing, and accelerometers to detect interactions with body, clothing, water, and toys.
• Designed interactive installations that map physical gestures and material properties to digital sound synthesis in real time.
• Integrated multimodal feedback by combining tactile, visual, and auditory responses to enhance user engagement.
• Employed general-purpose sensor technologies to ensure accessibility and adaptability across diverse performance contexts.
• Focused on intuitive mapping between physical actions and sound output to support expressive control.
• Validated designs through performance-based evaluation to assess controllability and expressiveness.

Experimental results

Research questions

• RQ1 How can everyday physical materials like clothing and water be effectively transformed into expressive musical interfaces?
• RQ2 What sensor technologies enable intuitive and responsive mapping between physical interaction and digital sound generation?
• RQ3 To what extent do multimodal feedback and tangible interaction improve user expressiveness and enjoyment in digital music performance?
• RQ4 How can the controllability and expressiveness of non-traditional instruments be quantitatively assessed in real-time performance settings?
• RQ5 What design principles enable the creation of accessible, low-cost instruments that support natural music expression?

Key findings

• The use of body, clothes, water, and toys enabled highly intuitive and expressive musical interactions, with users reporting strong emotional and physical engagement.
• Sensor-based mappings allowed for real-time, responsive sound generation that supported nuanced musical expression across different materials.
• Multimodal feedback—combining tactile, visual, and auditory cues—significantly enhanced user perception of control and expressiveness.
• The instruments demonstrated high levels of user enjoyment and perceived naturalness, particularly when interactions leveraged familiar physical properties.
• Controllability was highest in systems using direct touch and motion sensing, while water-based interfaces offered unique expressive potential due to fluid dynamics.
• The research demonstrated that low-cost, accessible materials can support sophisticated musical expression when combined with appropriate sensing and mapping techniques.