[Paper Review] Designing Sound Collaboratively - Perceptually Motivated Audio Synthesis

Niklas Klügel, Timo Becker+1 more|arXiv (Cornell University)|Jun 23, 2014

Music Technology and Sound Studies26 references3 citations

TL;DR

This paper presents a collaborative, perceptually grounded audio synthesis system using a multi-touch tabletop, where machine learning maps perceptual audio features to synthesis parameters for intuitive real-time sound design. A comparative study shows the system fosters creativity, flow, and collaboration, though navigation and awareness challenges remain due to non-linear timbral mappings and unclear user role attribution.

ABSTRACT

In this contribution, we will discuss a prototype that allows a group of users to design sound collaboratively in real time using a multi-touch tabletop. We make use of a machine learning method to generate a mapping from perceptual audio features to synthesis parameters. This mapping is then used for visualization and interaction. Finally, we discuss the results of a comparative evaluation study.

Motivation & Objective

• To enable intuitive, collaborative sound design by mapping perceptual audio features to synthesis parameters using machine learning.
• To support group creativity through real-time, multi-user interaction on a multi-touch tabletop.
• To address the cognitive difficulty of non-intuitive synthesis parameter spaces, especially for novices.
• To evaluate the system’s impact on user engagement, collaboration, and perceptual control in a real-world setting.
• To identify technical limitations in timbral navigation and user awareness for future refinement.

Proposed method

• A Gaussian Process Latent Variable Model (GPLVM) is used to learn a low-dimensional manifold from perceptual audio features to synthesis parameters.
• The resulting mapping is visualized as a 2D 'Timbre Surface' on a multi-touch tabletop for direct manipulation.
• Real-time interpolation using nearest neighbors ensures smooth transitions between timbres based on user touch position.
• A user study compares the system against a traditional synthesis interface to assess usability, collaboration, and user experience.
• Participants interact with private audio spaces and shared visualizations, enabling both individual experimentation and group coordination.
• Feature selection and GTM parameter tuning are explored to improve the coherence and precision of the timbral mapping.

Experimental results

Research questions

• RQ1 How does a perceptually motivated, machine learning-based audio synthesis interface support collaborative sound design on a multi-touch tabletop?
• RQ2 To what extent does the system enhance user engagement, creativity, and group flow compared to traditional synthesis methods?
• RQ3 What are the key usability challenges in navigating perceptual timbral spaces and maintaining group awareness during collaborative sound design?
• RQ4 How do users perceive the expressiveness, intuitiveness, and aesthetic quality of the synthesized sounds?
• RQ5 Can feature selection and interpolation techniques improve the coherence and precision of the timbral mapping?

Key findings

• Participants found the system more musical, expressive, and inspiring than traditional synthesis methods, with strong positive feedback on usability and aesthetics.
• The application successfully fostered collaboration and was perceived as fun, with users reporting high levels of immersion and flow.
• Users experienced confusion regarding group awareness, particularly in identifying which user contributed which sound, due to unclear role attribution in shared audio space.
• Navigation was perceived as incoherent, as small movements on the tabletop did not yield proportional or predictable changes in timbre, especially due to non-linearities in the GTM projection.
• Feature selection and improved interpolation methods—such as gradient-weighted interpolation—were shown to significantly enhance the quality and coherence of the Timbre Surface.
• The majority of users considered private audio spaces essential, and suggested enhancements like user-specific mixers or muting controls to improve awareness and control.