[Paper Review] Vision-based Manipulation of Deformable and Rigid Objects Using Subspace Projections of 2D Contours

Jihong Zhu, David Navarro-Alarcón+2 more|arXiv (Cornell University)|Jun 16, 2020

Advanced Vision and Imaging36 references4 citations

TL;DR

This paper presents a model-free, vision-based framework for unified manipulation of both rigid and deformable objects using subspace projections of 2D image contours. It automatically learns visual features and interaction matrices online from raw image data, enabling real-time control without camera calibration, prior knowledge of object properties, or extensive training, and successfully demonstrates 'move and shape' tasks on a sponge in both free-space and contact-based configurations.

ABSTRACT

This paper proposes a unified vision-based manipulation framework using image contours of deformable/rigid objects. Instead of using human-defined cues, the robot automatically learns the features from processed vision data. Our method simultaneously generates -- from the same data -- both, visual features and the interaction matrix that relates them to the robot control inputs. Extraction of the feature vector and control commands is done online and adaptively, with little data for initialization. The method allows the robot to manipulate an object without knowing whether it is rigid or deformable. To validate our approach, we conduct numerical simulations and experiments with both deformable and rigid objects.

Motivation & Objective

• To develop a unified vision-based control framework that handles both rigid and deformable object manipulation without prior assumptions about object mechanics.
• To eliminate the need for manual feature definition or physical modeling by learning visual features and interaction models directly from image data.
• To enable online, adaptive feature extraction and control input generation with minimal initialization data.
• To validate the framework on both simulation and real-world experiments involving moving and shaping objects, including contact-based deformation.
• To demonstrate robustness to initial misalignments and false contours in visual data during manipulation tasks.

Proposed method

• The method uses Principal Component Analysis (PCA) to project 2D object contours into a low-dimensional subspace, extracting a compact visual feature vector from image data.
• It formulates a direct visual servoing approach using photometric moments and subspace projections, avoiding explicit feature parameterization.
• The interaction matrix relating visual features to robot control inputs is learned online from concurrent vision and manipulation data, enabling real-time adaptation.
• The framework operates without camera calibration or knowledge of camera pose, object size, or shape, relying only on image luminance and contour data.
• A subspace-based error metric (ASE) is used to evaluate convergence, with a threshold of ASE = 1 for termination.
• The system is validated through numerical simulations and real experiments using a sponge as both rigid (free space) and deformable (contact-based) object.

Experimental results

Research questions

• RQ1 Can a unified vision-based control framework be developed that handles both rigid and deformable object manipulation without prior knowledge of object properties?
• RQ2 How can visual features and interaction models be learned simultaneously and online from raw image data without explicit feature engineering?
• RQ3 To what extent can the framework operate without camera calibration or knowledge of camera pose?
• RQ4 Can the system successfully perform 'move and shape' tasks requiring both motion and deformation of an object?
• RQ5 How robust is the method to visual noise, such as false contours, and initial misalignment during contact-based manipulation?

Key findings

• The framework successfully performs 'move and shape' tasks on a sponge, transitioning from free-space manipulation to contact-based shaping, with both experiments achieving convergence to the target contour.
• The system achieves convergence with ASE below the threshold of 1, as shown in Fig. 21, even when initial contact is misaligned.
• The method operates without camera calibration or prior knowledge of object size, shape, or rigidity, demonstrating robustness to varying initial conditions.
• False contours in the image data cause noise in the ASE metric, but the system still achieves acceptable final configurations, indicating resilience to visual artifacts.
• The framework is model-free and requires minimal initialization data, enabling fast online adaptation during manipulation.
• The approach is validated in both simulation and real-world experiments, confirming its feasibility and robustness across different object types and control scenarios.