[Paper Review] Evaluation of short range depth sonifications for visual-to-auditory sensory substitution

Louis Commère, Jean Rouat|arXiv (Cornell University)|Apr 11, 2023

Tactile and Sensory Interactions4 citations

TL;DR

This study evaluates five depth sonification techniques—frequency, amplitude, repetition rate of beeps, reverberation, and signal-to-noise ratio—for visual-to-auditory sensory substitution in blind users. Using sighted blindfolded participants, it finds that beep repetition rate provides the most accurate and memorable depth perception, outperforming other methods in positioning accuracy and user perception.

ABSTRACT

Visual to auditory sensory substitution devices convert visual information into sound and can provide valuable assistance for blind people. Recent iterations of these devices rely on depth sensors. Rules for converting depth into sound (i.e. the sonifications) are often designed arbitrarily, with no strong evidence for choosing one over another. The purpose of this work is to compare and understand the effectiveness of five depth sonifications in order to assist the design process of future visual to auditory systems for blind people which rely on depth sensors. The frequency, amplitude and reverberation of the sound as well as the repetition rate of short high-pitched sounds and the signal-to-noise ratio of a mixture between pure sound and noise are studied. We conducted positioning experiments with twenty-eight sighted blindfolded participants. Stage 1 incorporates learning phases followed by depth estimation tasks. Stage 2 adds the additional challenge of azimuth estimation to the first stage's protocol. Stage 3 tests learning retention by incorporating a 10-minute break before re-testing depth estimation. The best depth estimates in stage 1 were obtained with the sound frequency and the repetition rate of beeps. In stage 2, the beep repetition rate yielded the best depth estimation and no significant difference was observed for the azimuth estimation. Results of stage 3 showed that the beep repetition rate was the easiest sonification to memorize. Based on statistical analysis of the results, we discuss the effectiveness of each sonification and compare with other studies that encode depth into sounds. Finally we provide recommendations for the design of depth encoding.

Motivation & Objective

• To evaluate and compare the effectiveness of five depth sonification strategies in visual-to-auditory sensory substitution systems.
• To determine which sonification method enables the most accurate and memorable depth estimation for blind users using 3D sensor data.
• To provide evidence-based design recommendations for future sensory substitution devices relying on depth sensors.
• To investigate whether sonification performance differs under conditions of learning, task complexity, and memory retention.
• To assess the role of perceptual naturalness and cross-modal correspondences in sonification design.

Proposed method

• Conducted three experimental stages: (1) depth estimation with learning, (2) combined depth and azimuth estimation, and (3) memory retention after a 10-minute break.
• Used 28 sighted, blindfolded participants to eliminate visual feedback during sonification tasks.
• Evaluated five sonification methods: frequency, amplitude, repetition rate of short high-pitched beeps, reverberation, and signal-to-noise ratio of pure tone mixed with noise.
• Measured absolute positioning error and used statistical analysis to compare performance across sonifications.
• Computed chance-level error as (b−a)/3 to establish baseline performance for significance testing.
• Collected qualitative feedback on perceived naturalness and intuitiveness of each sonification.

Experimental results

Research questions

• RQ1 Which depth sonification method yields the most accurate depth estimation in a short-range (1-meter) environment?
• RQ2 How does the inclusion of azimuth estimation affect the performance of different sonification strategies?
• RQ3 Which sonification is most easily remembered after a 10-minute delay, indicating long-term usability?
• RQ4 How do perceptual naturalness and cross-modal correspondences influence user preference and performance?
• RQ5 Can the results from sighted blindfolded participants be generalized to blind users in real-world applications?

Key findings

• The repetition rate of beeps yielded the most accurate depth estimates in Stage 1, with significantly lower absolute positioning error compared to other sonifications.
• In Stage 2, which included azimuth estimation, the beep repetition rate remained the best-performing sonification, with no significant difference in azimuth estimation accuracy across methods.
• The beep repetition rate was rated as the most natural and intuitive method for encoding depth, with participants consistently preferring it over others.
• Performance in Stage 3 demonstrated that the beep repetition rate was the easiest sonification to memorize, with the lowest error after a 10-minute break.
• The sonification based on sound frequency provided good quantitative accuracy but was perceived as less intuitive and less naturally associated with depth.
• The mixture of a pure tone with white noise received strong qualitative feedback and showed potential, though it was less accurate than the beep repetition rate.