[Paper Review] Taxim: An Example-based Simulation Model for GelSight Tactile Sensors

Zilin Si, Wenzhen Yuan|arXiv (Cornell University)|Sep 9, 2021

Advanced Sensor and Energy Harvesting Materials4 citations

TL;DR

Taxim is an example-based, high-speed simulation framework for GelSight tactile sensors that combines optical response simulation via a polynomial lookup table and marker motion field simulation using linear displacement and superposition. It achieves state-of-the-art optical accuracy with sub-pixel intensity error and runs efficiently on CPU, enabling realistic, computationally lightweight tactile simulation for robotics research.

ABSTRACT

Simulation is widely used in robotics for system verification and large-scale data collection. However, simulating sensors, including tactile sensors, has been a long-standing challenge. In this paper, we propose Taxim, a realistic and high-speed simulation model for a vision-based tactile sensor, GelSight. A GelSight sensor uses a piece of soft elastomer as the medium of contact and embeds optical structures to capture the deformation of the elastomer, which infers the geometry and forces applied at the contact surface. We propose an example-based method for simulating GelSight: we simulate the optical response to the deformation with a polynomial look-up table. This table maps the deformed geometries to pixel intensity sampled by the embedded camera. In order to simulate the surface markers' motion that is caused by the surface stretch of the elastomer, we apply the linear elastic deformation theory and the superposition principle. The simulation model is calibrated with less than 100 data points from a real sensor. The example-based approach enables the model to easily migrate to other GelSight sensors or its variations. To the best of our knowledge, our simulation framework is the first to incorporate marker motion field simulation that derives from elastomer deformation together with the optical simulation, creating a comprehensive and computationally efficient tactile simulation framework. Experiments reveal that our optical simulation has the lowest pixel-wise intensity errors compared to prior work and can run online with CPU computing. Our code and supplementary materials are open-sourced at https://github.com/CMURoboTouch/Taxim.

Motivation & Objective

• To address the lack of realistic, efficient tactile sensor simulation in robotics simulation frameworks.
• To develop a simulation model that captures both optical response and marker motion in vision-based tactile sensors like GelSight.
• To create a computationally efficient, calibration-based method that generalizes across GelSight variants with minimal real-world data.
• To integrate both optical and mechanical response simulation in a single, unified framework for sim-to-real tactile perception.
• To enable large-scale data generation and system verification for tactile-based manipulation without costly real-world experiments.

Proposed method

• A polynomial lookup table maps contact geometries to pixel intensity in tactile images, simulating the optical response of the GelSight sensor.
• The marker motion field is simulated using the linear displacement relationship and superposition principle to model elastomer deformation.
• The model is calibrated using fewer than 100 real sensor contact examples, enabling fast adaptation to new sensors.
• Shadow effects from illumination are simulated using an accumulation approach to enhance realism.
• The framework is designed for integration into robot simulation engines like PyBullet and Gazebo.
• The entire simulation runs online on CPU, ensuring high-speed performance for real-time applications.

Experimental results

Research questions

• RQ1 Can a simulation model accurately replicate the optical response of a GelSight sensor using a data-driven, example-based approach?
• RQ2 Can the marker motion field induced by elastomer deformation be simulated with high fidelity using linear displacement and superposition principles?
• RQ3 How does the proposed simulation compare in accuracy and speed to existing physics-based or learning-based tactile simulators?
• RQ4 To what extent can the model generalize across different GelSight sensors with minimal re-calibration?
• RQ5 Can the simulation effectively capture sensor noise and illumination effects through calibration with real data?

Key findings

• The optical simulation achieves the lowest pixel-wise intensity error compared to prior work, with a mean error of 3.58×10⁻³ mm in X, 3.32×10⁻³ mm in Y, and 5.43×10⁻³ mm in Z direction.
• The marker motion field simulation shows a mean L1 error of 3.96×10⁻³ mm when compared to FEM simulation, indicating strong alignment with physics-based deformation.
• The simulation runs in 9.22 seconds on average on CPU, significantly faster than FEM simulations (2–4 hours) and competitive with GPU-accelerated baselines.
• The weighted mean angular L1 error between real and simulated marker motion is 14.57°, close to the 12.94° error between real and FEM data, indicating high fidelity.
• The model successfully captures illumination effects and sensor noise through calibration with less than 100 real examples, enabling accurate sim-to-real transfer.
• Taxim is the first framework to simultaneously simulate both optical response and marker motion field with high accuracy and low computational cost.