[Paper Review] In-Hand Gravitational Pivoting Using Tactile Sensing

Jason Toskov, R. Newbury+3 more|arXiv (Cornell University)|Oct 11, 2022

Tactile and Sensory Interactions4 citations

TL;DR

This paper proposes a tactile-only approach for in-hand gravitational pivoting using an LSTM-based model (RSE-LSTM) to estimate object angular position and velocity from force-based tactile sensors. The method enables a parallel gripper to rotate objects to target angles without vision or prior object knowledge, achieving a 16.6% performance improvement via fine-tuning and outperforming a baseline MLP in zero-shot generalization on unseen household objects.

ABSTRACT

We study gravitational pivoting, a constrained version of in-hand manipulation, where we aim to control the rotation of an object around the grip point of a parallel gripper. To achieve this, instead of controlling the gripper to avoid slip, we embrace slip to allow the object to rotate in-hand. We collect two real-world datasets, a static tracking dataset and a controller-in-the loop dataset, both annotated with object angle and angular velocity labels. Both datasets contain force-based tactile information on ten different household objects. We train an LSTM model to predict the angular position and velocity of the held object from purely tactile data. We integrate this model with a controller that opens and closes the gripper allowing the object to rotate to desired relative angles. We conduct real-world experiments where the robot is tasked to achieve a relative target angle. We show that our approach outperforms a sliding-window based MLP in a zero-shot generalization setting with unseen objects. Furthermore, we show a 16.6% improvement in performance when the LSTM model is fine-tuned on a small set of data collected with both the LSTM model and the controller in-the-loop. Code and videos are available at https://rhys-newbury.github.io/projects/pivoting/

Motivation & Objective

• To enable robust in-hand object reorientation using only tactile feedback, avoiding reliance on vision or object-specific models.
• To address the challenge of tracking object angular position and velocity during gravitational pivoting with slip as a control mechanism.
• To develop a generalizable gripper controller that operates without prior knowledge of object mass, shape, or friction.
• To collect and release a real-world tactile dataset with 10 household objects for gravitational pivoting tasks.
• To evaluate zero-shot generalization and fine-tuning performance on unseen objects in real-world experiments.

Proposed method

• An RSE-LSTM neural network is trained to predict object angular position and velocity from force-based tactile sensor data collected during pivoting.
• The gripper controller modulates grip width based on RSE-LSTM predictions to induce gravitational pivoting toward a target angle.
• A two-dataset collection protocol is used: a static tracking dataset and a controller-in-the-loop dataset, both annotated with angular position and velocity.
• The system uses forward prediction to compensate for a 0.83s delay in gripper closing, adjusting stopping conditions based on velocity estimates.
• Fine-tuning is performed using data collected with the RSE-LSTM model and controller in the loop to improve real-world performance.
• A sliding-window MLP baseline is used for comparison, trained on the same tactile input to evaluate generalization.

Experimental results

Research questions

• RQ1 Can tactile sensing alone accurately estimate object angular position and velocity during gravitational pivoting?
• RQ2 Can a gripper controller using only tactile feedback achieve target-angle reorientation without vision or object prior knowledge?
• RQ3 How does the RSE-LSTM model generalize to unseen objects in real-world settings compared to a baseline MLP?
• RQ4 To what extent does fine-tuning on in-the-loop data improve controller performance on unseen objects?
• RQ5 What are the key failure modes and performance bottlenecks in tactile-based pivoting, particularly due to control delay and velocity estimation errors?

Key findings

• The RSE-LSTM model outperformed the MLP baseline in zero-shot generalization on unseen objects, demonstrating superior generalization capability.
• Fine-tuning the RSE-LSTM model on in-the-loop data improved performance by 16.6% compared to the non-fine-tuned version, reducing target error from 15.2° to 12.67°.
• The LSTM model achieved a failure rate of 3.3% in real-world experiments, significantly lower than the MLP’s 6.7%.
• The Oracle controller (using ground truth visual feedback) achieved a target error of 3.95°, serving as an upper bound, while the best RSE-LSTM model reached 12.67°.
• The system exhibited higher tracking error in real-world testing than in the test set, likely due to unmodeled dynamics like overshooting and incomplete gripper closure.
• Performance degradation was observed on certain objects, suggesting sensitivity to object-specific properties such as mass or surface friction, highlighting the need for broader data collection.