[Paper Review] AU Dataset for Visuo-Haptic Object Recognition for Robots

Lasse Emil R. Bonner, Daniel Daugaard Buhl+2 more|arXiv (Cornell University)|Jan 1, 2021

Robot Manipulation and Learning4 citations

TL;DR

This paper introduces the AU Dataset for Visuo-Haptic Object Recognition, a multimodal dataset containing visual, kinesthetic, and tactile (vibration-based) data from 63 objects with controlled visual and haptic ambiguities. Collected using a robotic hand (RH8D) and NAO robot with contact microphones, the dataset enables research in sensor fusion for robot object recognition under realistic perceptual challenges.

ABSTRACT

This directory contains the data, object list, thimble's 3D design and the technical report (Bonner2021AU.pdf) describing object exploration procedures, experimental set, folder structure, etc.<br><br><b>Citing this dataset:</b><br>Bonner, L. E. R., Buhl, D. D., Kristensen, K., &amp; Navarro-Guerrero, N. (2021). AU Dataset for Visuo-Haptic Object Recognition for Robots. figshare. https://doi.org/10.6084/m9.figshare.14222486<br><br><b>Related References:</b><br>Toprak, S., Navarro-Guerrero, N., &amp; Wermter, S. (2018). Evaluating Integration Strategies for Visuo-Haptic Object Recognition. Cognitive Computation, 10(3), 408–425. https://doi.org/10.1007/s12559-017-9536-7.<br><br><b>Contact:</b><br>For more information, please get in touch with Nicolás Navarro-Guerrero.

Motivation & Objective

• To address the scarcity of publicly available, large-scale multimodal datasets for visuo-haptic object recognition in robotics.
• To create a dataset with controlled visual and haptic ambiguities to challenge and evaluate multimodal fusion techniques.
• To provide a standardized benchmark for testing sensor fusion algorithms in robot perception.
• To support research in tactile perception using vibration signals captured via contact microphones on robotic hands and arms.

Proposed method

• Data collection used exploratory procedures from Lederman and Klatzky: visual scanning, kinesthetic lifting and enclosure, and tactile lateral motion and pressure application.
• A human-sized robotic hand (RH8D) captured kinesthetic data via joint current (mA) and finger position (degrees), with baseline at 30mA.
• Vibrations from tactile exploration were recorded using five clip-on contact microphones on the NAO robot and RH8D hand at 400kHz sampling rate.
• Visual data consisted of four high-resolution (4640×3472) images per object: three faces and one background image for subtraction.
• Background noise was recorded for each procedure to enable noise compensation during signal processing.
• All data—visual, kinesthetic, and vibration—was stored in structured CSV and image files with consistent naming across three repositionings per object.

Experimental results

Research questions

• RQ1 How effective are multimodal fusion strategies in resolving visual and haptic ambiguities in object recognition?
• RQ2 To what extent can vibration signals from contact microphones replace traditional tactile sensors in object perception?
• RQ3 Can kinesthetic data (current and joint positions) reliably infer object weight and shape under controlled robotic manipulation?
• RQ4 How does the inclusion of background noise recordings improve the robustness of tactile signal analysis?
• RQ5 What is the impact of object fill material (e.g., Play-Doh, salt, coffee beans) on haptic perception and classification?

Key findings

• The dataset includes 63 objects with visual and haptic ambiguities, such as a yellow ball and yellow lemon, or a velvet bag filled with different materials.
• Kinesthetic data captured via RH8D hand current (30mA baseline) and finger positions (−180° to 180°) enables inference of object weight and global shape.
• Tactile exploration via NAO robot’s lateral motion and pressure application generated vibration data in mV at 400kHz, stored across five microphone channels.
• Background noise was recorded for each procedure to support noise subtraction in vibration signal processing.
• The dataset comprises 60 kinesthetic CSV files, 30 vibration CSV files, and 189 visual images (63 objects × 3 positions), with structured folder organization.
• The dataset is publicly available via figshare with DOI: 10.6084/m9.figshare.14222486, supporting reproducible research in robot perception.