[Paper Review] A Survey on Robotic Manipulation of Deformable Objects: Recent Advances, Open Challenges and New Frontiers

Feida Gu, Yanmin Zhou+3 more|arXiv (Cornell University)|Dec 16, 2023

Robot Manipulation and Learning10 citations

TL;DR

This survey reviews data-driven and analytical approaches to perception, modeling, and manipulation of deformable objects (DOs) in robotics, highlighting recent advances and open challenges, including the role of large language models (LLMs).

ABSTRACT

Deformable object manipulation (DOM) for robots has a wide range of applications in various fields such as industrial, service and health care sectors. However, compared to manipulation of rigid objects, DOM poses significant challenges for robotic perception, modeling and manipulation, due to the infinite dimensionality of the state space of deformable objects (DOs) and the complexity of their dynamics. The development of computer graphics and machine learning has enabled novel techniques for DOM. These techniques, based on data-driven paradigms, can address some of the challenges that analytical approaches of DOM face. However, some existing reviews do not include all aspects of DOM, and some previous reviews do not summarize data-driven approaches adequately. In this article, we survey more than 150 relevant studies (data-driven approaches mainly) and summarize recent advances, open challenges, and new frontiers for aspects of perception, modeling and manipulation for DOs. Particularly, we summarize initial progress made by Large Language Models (LLMs) in robotic manipulation, and indicates some valuable directions for further research. We believe that integrating data-driven approaches and analytical approaches can provide viable solutions to open challenges of DOM.

Motivation & Objective

• Survey over 150 studies on deformable object manipulation to synthesize current knowledge.
• Highlight advances in perception, modeling, and manipulation for DOs with emphasis on data-driven methods.
• Discuss open challenges and propose future directions, including multimodal perception and LLMs in DOM.
• Bridge analytical and data-driven approaches to offer viable solutions for DOM tasks.

Proposed method

• Systematic literature review of recent works on DOM across perception, modeling, and manipulation.
• Classification of perception into visual, tactile, and multimodal modalities with dataset and simulator considerations.
• Comparison of analytical modeling (MSD, PBD, continuum mechanics) versus data-driven models (Jacobian-based and GNN-based).
• Overview of manipulation strategies including traditional planning/control and learning-based methods.
• Discussion of LLMs in robotic manipulation and their potential impact on task definition, planning, rewards, and uncertainty alignment.
• Discussion of data requirements, datasets, tactile simulators, and sim-to-real considerations.

Figure 1: Applications involving DOM. (a) Manufacturing industry [ 3 , 4 , 5 , 6 ] (b) Medical surgery [ 7 , 8 , 9 , 10 ] (c) Food processing [ 11 , 12 , 13 , 14 ] (d) Daily living activities [ 15 , 16 , 17 , 18 ]

Experimental results

Research questions

• RQ1 What are the recent advances in perception, modeling, and manipulation for deformable object manipulation in robotics?
• RQ2 What open challenges remain in DOM, and what frontiers are identified for future research?
• RQ3 How can data-driven approaches be integrated with analytical models to address DOM complexity?
• RQ4 What is the potential impact of Large Language Models (LLMs) on robotic manipulation of DOs?
• RQ5 What role do datasets and tactile simulators play in advancing multimodal DOM perception?

Key findings

• The survey covers more than 150 relevant studies with a focus on data-driven approaches.
• Multimodal perception combining vision and tactile sensing is emphasized as essential for DOM.
• GNN-based models and Jacobian-matrix approaches are presented as key data-driven modeling techniques.
• RL and imitation learning methods are highlighted for DOM manipulation, alongside traditional planning and control.
• LLMs are discussed as a nascent but promising direction for task definition, planning, and reward design in DOM.
• Integration of data-driven and analytical approaches is proposed as a viable path to address open DOM challenges.

Figure 2: A typical robotic system for handling DOs including robotic hardware, perception hardware, robotic hands, tools used for DOM, algorithms for various functions, etc.