[Paper Review] Motion Switching with Sensory and Instruction Signals by designing Dynamical Systems using Deep Neural Network

Kanata Suzuki, Hiroki MORI+1 more|arXiv (Cornell University)|Dec 14, 2017

Robot Manipulation and Learning4 citations

TL;DR

This paper proposes a deep learning framework that designs dynamical systems using point attractors to enable robots to switch between subtasks in manipulation tasks via sensory and instruction signals. By integrating a convolutional autoencoder and a multiple time-scale RNN, the model learns to combine visual feedback and external instructions for flexible, adaptive motion control, successfully demonstrating cloth-folding with dynamic switching across untrained object positions.

ABSTRACT

To ensure that a robot is able to accomplish an extensive range of tasks, it is necessary to achieve a flexible combination of multiple behaviors. This is because the design of task motions suited to each situation would become increasingly difficult as the number of situations and the types of tasks performed by them increase. To handle the switching and combination of multiple behaviors, we propose a method to design dynamical systems based on point attractors that accept (i) "instruction signals" for instruction-driven switching. We incorporate the (ii) "instruction phase" to form a point attractor and divide the target task into multiple subtasks. By forming an instruction phase that consists of point attractors, the model embeds a subtask in the form of trajectory dynamics that can be manipulated using sensory and instruction signals. Our model comprises two deep neural networks: a convolutional autoencoder and a multiple time-scale recurrent neural network. In this study, we apply the proposed method to manipulate soft materials. To evaluate our model, we design a cloth-folding task that consists of four subtasks and three patterns of instruction signals, which indicate the direction of motion. The results depict that the robot can perform the required task by combining subtasks based on sensory and instruction signals. And, our model determined the relations among these signals using its internal dynamics.

Motivation & Objective

• To address the challenge of designing flexible, scalable motion policies for complex manipulation tasks involving soft materials.
• To enable dynamic switching between subtasks using both sensory feedback and external instruction signals, overcoming limitations of purely model-based or sensory-driven control.
• To develop a unified framework that embeds task sequences as trajectory dynamics in a hierarchical RNN structure, allowing generalization across diverse object positions and task variations.
• To investigate how internal neural dynamics can encode and combine sensory and instruction signals for robust, interactive task execution.

Proposed method

• Designs a dynamical system using point attractors as switching phases to enable stable, switchable subtask dynamics.
• Employs a two-stream deep neural network: a convolutional autoencoder for sensory feature extraction and a multiple time-scale RNN for temporal dynamics modeling.
• Uses the RNN's context layers (C_f and C_s) to separately encode instruction signals (C_f) and sensory-motor sequences (C_s), enabling hierarchical signal integration.
• Trains the model on a cloth-folding task with five subtasks, using image sequences and instruction signals indicating motion direction.
• Applies point attractor constraints to align internal states at switching points, ensuring smooth transitions between subtasks.
• Visualizes internal states via PCA to analyze how sensory and instruction signals are encoded and combined in the network's latent space.

Experimental results

Research questions

• RQ1 Can a deep learning-based dynamical system effectively switch between multiple subtasks using both sensory input and external instruction signals?
• RQ2 How can internal neural dynamics represent and combine sensory feedback and instruction signals for adaptive task execution?
• RQ3 To what extent can the model generalize to untrained object positions and shapes in a manipulation task?
• RQ4 Can the model learn to switch between subtasks based on a combination of visual feedback and instruction signals without explicit retraining?
• RQ5 How do the distinct internal states of the RNN (C_f and C_s) contribute to instruction-driven and sensory-driven switching, respectively?

Key findings

• The model successfully performed a cloth-folding task with four subtasks and three instruction patterns, demonstrating robust switching across untrained object positions.
• The C_f neurons encoded instruction signals, forming three distinct clusters corresponding to motion direction, enabling immediate response to instructions regardless of object position.
• The C_s neurons captured sensory-motor dynamics, forming four distinct clusters corresponding to subtasks, indicating learning of visual and motion differences.
• The model exhibited interactive switching by combining sensory and instruction signals, with the internal state space showing clear separation and integration of both signal types.
• The hierarchical structure allowed the model to generalize task execution across diverse configurations, confirming the method’s robustness and adaptability.
• Visualization of internal states confirmed that the model learned to embed and switch between task dynamics based on both sensory feedback and instruction signals.