[Paper Review] Audio-Visual Self-Supervised Terrain Type Discovery for Mobile Platforms

Akiyoshi Kurobe, Yoshikatsu Nakajima+2 more|arXiv (Cornell University)|Oct 13, 2020

Video Surveillance and Tracking Methods36 references4 citations

TL;DR

This paper proposes a self-supervised, multi-modal approach that alternates between audio and visual features to discover terrain types on mobile platforms, using adaptive clustering to generate pseudo-labels for training a vision-based CNN. The method achieves over 85% accuracy in terrain classification, significantly outperforming single-modality baselines by leveraging audio's robustness to visual noise and vision's discriminative power through feature switching.

ABSTRACT

The ability to both recognize and discover terrain characteristics is an important function required for many autonomous ground robots such as social robots, assistive robots, autonomous vehicles, and ground exploration robots. Recognizing and discovering terrain characteristics is challenging because similar terrains may have very different appearances (e.g., carpet comes in many colors), while terrains with very similar appearance may have very different physical properties (e.g. mulch versus dirt). In order to address the inherent ambiguity in vision-based terrain recognition and discovery, we propose a multi-modal self-supervised learning technique that switches between audio features extracted from a mic attached to the underside of a mobile platform and image features extracted by a camera on the platform to cluster terrain types. The terrain cluster labels are then used to train an image-based convolutional neural network to predict changes in terrain types. Through experiments, we demonstrate that the proposed self-supervised terrain type discovery method achieves over 80% accuracy, which greatly outperforms several baselines and suggests strong potential for assistive applications.

Motivation & Objective

• To address the ambiguity in vision-based terrain recognition, where similar appearances mask different physical properties (e.g., mulch vs. dirt).
• To overcome limitations of single-modality sensing (e.g., visual noise from lighting or objects, audio interference) by fusing audio and visual modalities in a self-supervised framework.
• To develop a practical, low-cost method for terrain type discovery using only onboard mic and camera, suitable for assistive and exploration robots.
• To create a free-to-use dataset with synchronized audio and image data for terrain friction and visual appearance.
• To train a vision-based CNN using pseudo-labels derived from multi-modal clustering for robust terrain classification.

Proposed method

• The method first performs audio-based clustering to generate initial temporal segments of terrain types, using features from a microphone mounted under the robot.
• Within each temporal segment, audio and visual features are averaged to reduce noise from occlusions or environmental interference.
• A second, larger-scale clustering is applied to the averaged features to produce more stable and coherent terrain segments.
• The system adaptively switches between audio and visual features during clustering to leverage the strengths of each modality—audio for consistent physical property detection, vision for appearance-based discrimination.
• Pseudo-labels from the final clustering are used to train a ResNet50-based CNN for end-to-end terrain type prediction in a self-supervised manner.
• The framework uses sequence detection and agglomerative clustering to refine segment boundaries and improve clustering robustness.

Experimental results

Research questions

• RQ1 Can multi-modal self-supervised learning improve terrain type discovery by combining audio and visual cues more effectively than single-modality approaches?
• RQ2 How does adaptive switching between audio and visual features during clustering affect the accuracy and robustness of terrain clustering?
• RQ3 To what extent can pseudo-labels generated from audio-visual clustering improve the performance of a vision-based terrain classifier?
• RQ4 Does the proposed method generalize across diverse indoor and outdoor environments with varying terrain appearances and physical properties?

Key findings

• The proposed self-supervised multi-modal clustering method achieved over 85% average accuracy in terrain type classification, significantly outperforming single-modality baselines.
• The method achieved 87.3% accuracy on carpet, 95.7% on asphalt, and 95.5% on pavement, demonstrating strong performance across diverse terrain types.
• Feature switching between audio and visual modalities improved clustering accuracy compared to concatenating features, as shown by higher NMI scores and better qualitative segmentation.
• The CNN trained on pseudo-labels from the proposed clustering framework correctly classified similar-looking terrains (e.g., different carpet types) by learning from multi-modal consistency.
• Qualitative results showed that the switching method reduced over-segmentation (common in audio-only) and noise sensitivity (common in visual-only), especially in scenes with changing lighting or floor patterns.
• The dataset, containing synchronized audio and image data from real-world indoor and outdoor environments, is publicly available and supports future research in terrain perception.