[Paper Review] Learning to Read Braille: Bridging the Tactile Reality Gap with Diffusion Models

Carolina Higuera, Byron Boots+1 more|arXiv (Cornell University)|Apr 3, 2023

Tactile and Sensory Interactions9 citations

TL;DR

The paper uses conditional diffusion models to render realistic tactile images from simulated depth, enabling zero-shot Braille reading with a DIGIT sensor and achieving 75.74% accuracy on real data.

ABSTRACT

Simulating vision-based tactile sensors enables learning models for contact-rich tasks when collecting real world data at scale can be prohibitive. However, modeling the optical response of the gel deformation as well as incorporating the dynamics of the contact makes sim2real challenging. Prior works have explored data augmentation, fine-tuning, or learning generative models to reduce the sim2real gap. In this work, we present the first method to leverage probabilistic diffusion models for capturing complex illumination changes from gel deformations. Our tactile diffusion model is able to generate realistic tactile images from simulated contact depth bridging the reality gap for vision-based tactile sensing. On real braille reading task with a DIGIT sensor, a classifier trained with our diffusion model achieves 75.74% accuracy outperforming classifiers trained with simulation and other approaches. Project page: https://github.com/carolinahiguera/Tactile-Diffusion

Motivation & Objective

• Motivate realistic rendering of vision-based tactile data to close the sim-to-real gap.
• Develop a tactile diffusion model conditioned on simulated height maps to generate realistic tactile images.
• Demonstrate zero-shot transfer to real Braille reading using a DIGIT sensor after diffusion-based rendering.
• Quantify improvements over simulation-only, data augmentation, and GAN-based baselines in downstream Braille classification.

Proposed method

• Formulate image-to-image translation as a conditional diffusion process p(y|x) with x being the simulated depth image and y the real-like tactile image.
• Train a conditional U-Net decoder with a denoising score-matching objective to recover y0 from noisy yT conditioned on x (Eq. 3).
• Pre-train the diffusion model on 180k sim-depth / real tactile image pairs from the YCB-Slide dataset.
• Fine-tune the model with 20% real braille data to adapt to task-specific textures and braille bumps.
• During inference, start from Gaussian noise and iteratively denoise conditioned on a sim-depth input to generate realistic tactile images (Eq. 4–6).
• Evaluate using SSIM and MSE against real tactile images and compare downstream Braille classification performance.

Experimental results

Research questions

• RQ1 Can a diffusion-based renderer capture illumination and gel-deformation effects in vision-based tactile sensors to bridge the sim-to-real gap?
• RQ2 Does diffusion-based post-processing enable zero-shot transfer for Braille reading with a DIGIT sensor when trained primarily on simulated data?
• RQ3 How does tactile diffusion compare to GAN-based approaches for realism and downstream task accuracy?
• RQ4 What is the impact of fine-tuning with limited real data on downstream Braille classification performance?

Key findings

• Tactile diffusion achieves SSIM generally above 0.80 on test trajectories, indicating realistic structure of real tactile images.
• Fine-tuned tactile diffusion on 27 braille characters yields SSIM 0.908 and MSE 36.02 (max 255 per channel), outperforming cGAN (SSIM 0.879, MSE 47.99).
• Braille classifier trained with diffusion-generated data achieves zero-shot real data accuracy of 75.74%.
• Sim-only training with data augmentation or real data fine-tuning improves accuracy but diffusion-based approach requires less real data to reach strong performance.
• Diffusion-based rendering provides better preservation of braille indentations than cGAN, reducing misclassification due to texture artifacts.