[Paper Review] Kinetic surface friction rendering for interactive sonification: an initial exploration

Staas de Jong|arXiv (Cornell University)|Jul 29, 2021

Tactile and Sensory Interactions6 references3 citations

TL;DR

This paper introduces a tactile system that renders programmable kinetic surface friction in real time to enhance interactive sonification, enabling users to perceive virtual data as both sound and dynamic haptic feedback during physical interaction. The prototype maps 1D signal data to both audio and friction patterns, demonstrating that friction rendering can give virtual sonified objects a tangible presence, with pilot evaluation highlighting its potential for richer multimodal interaction.

ABSTRACT

Inspired by the role sound and friction play in interactions with everyday objects, this work aims to identify some of the ways in which kinetic surface friction rendering can complement interactive sonification controlled by movable objects. In order to do this, a tactile system is presented which implements a movable physical object with programmable friction. Important aspects of this system include the capacity to display high-resolution kinetic friction patterns, the ability to algorithmically define interactions directly in terms of physical units, and the complete integration of audio and tactile synthesis. A prototype interaction spatially mapping arbitrary 1D signal data on a surface and directly converting these to sound and friction during movements across the surface is described. The results of a pilot evaluation of this interaction indicate how kinetic surface friction rendering can be a means for giving dynamically created virtual objects for sonification a tangible presence. Some specific possible roles for movement input and friction output are identified, as well as issues to be considered when applying and further developing this type of haptic feedback in the context of interactive sonification.

Motivation & Objective

• To investigate how kinetic surface friction can enhance interactive sonification by providing tangible feedback.
• To develop a system capable of dynamically rendering high-resolution friction patterns in sync with audio output.
• To integrate physical movement, audio synthesis, and tactile feedback into a unified interactive framework.
• To evaluate the role of friction in giving virtual sonified objects a perceptible, physical presence.
• To identify design considerations and potential applications of friction-based haptic feedback in sonification.

Proposed method

• A physical movable object is equipped with a programmable friction interface to simulate varying surface textures.
• Friction patterns are algorithmically defined using physical units (e.g., N/m) to ensure consistency and precision.
• 1D signal data is spatially mapped across a surface and directly converted into both sound and friction output during user movement.
• Audio and tactile synthesis are fully integrated, enabling synchronized real-time feedback.
• The system supports dynamic, user-driven interaction where movement across the surface triggers both auditory and haptic responses.
• A prototype implementation demonstrates the mapping of arbitrary signal data to combined audio and friction feedback.

Experimental results

Research questions

• RQ1 How can kinetic surface friction be effectively rendered in real time to complement interactive sonification?
• RQ2 What role does friction play in enhancing the perception of virtual objects during sonification?
• RQ3 How do users perceive the integration of tactile and auditory feedback in data exploration?
• RQ4 What design principles govern the mapping of signal data to friction and sound?
• RQ5 What are the technical and perceptual challenges in implementing such a multimodal feedback system?

Key findings

• The prototype successfully demonstrated real-time synchronization of audio and friction feedback during user movement.
• Pilot evaluation indicated that friction rendering contributes to a stronger sense of tangible presence for virtual sonified objects.
• Users reported enhanced perceptual engagement when both sound and friction were present during interaction.
• The system supports high-resolution friction patterns, enabling fine-grained haptic feedback.
• The integration of physical units in friction control allows for precise, reproducible interaction design.
• The study identified key challenges in perceptual alignment between audio and tactile feedback, suggesting further refinement is needed.