[Paper Review] Predictive Coding-based Deep Dynamic Neural Network for Visuomotor Learning

Jungsik Hwang, Jinhyung Kim+3 more|arXiv (Cornell University)|Jun 8, 2017

Action Observation and Synchronization22 references3 citations

TL;DR

This paper proposes a predictive coding-based deep dynamic neural network that enables visuomotor learning by minimizing prediction error to infer intentions and simulate dynamic sensory patterns. The model learns to mentally simulate incoming visuomotor inputs and infer underlying intentions, demonstrating cross-modal representation recall and supporting the predictive coding account of mirror neuron systems in a robotic imitation task.

ABSTRACT

This study presents a dynamic neural network model based on the predictive coding framework for perceiving and predicting the dynamic visuo-proprioceptive patterns. In our previous study [1], we have shown that the deep dynamic neural network model was able to coordinate visual perception and action generation in a seamless manner. In the current study, we extended the previous model under the predictive coding framework to endow the model with a capability of perceiving and predicting dynamic visuo-proprioceptive patterns as well as a capability of inferring intention behind the perceived visuomotor information through minimizing prediction error. A set of synthetic experiments were conducted in which a robot learned to imitate the gestures of another robot in a simulation environment. The experimental results showed that with given intention states, the model was able to mentally simulate the possible incoming dynamic visuo-proprioceptive patterns in a top-down process without the inputs from the external environment. Moreover, the results highlighted the role of minimizing prediction error in inferring underlying intention of the perceived visuo-proprioceptive patterns, supporting the predictive coding account of the mirror neuron systems. The results also revealed that minimizing prediction error in one modality induced the recall of the corresponding representation of another modality acquired during the consolidative learning of raw-level visuo-proprioceptive patterns.

Motivation & Objective

• To develop a dynamic neural network model that integrates perception and action through predictive coding for visuomotor learning.
• To enable the model to infer hidden intentions behind observed visuomotor behaviors by minimizing prediction error.
• To investigate how minimizing prediction error in one sensory modality triggers recall of representations in another modality during consolidative learning.
• To validate the model's ability to simulate dynamic visuo-proprioceptive patterns in a top-down manner without external input.
• To support the predictive coding framework as an explanation for mirror neuron system functionality in social cognition.

Proposed method

• The model is built on a deep dynamic neural network architecture that processes both visual and proprioceptive sensory inputs.
• It employs a predictive coding framework where prediction errors across multiple layers are minimized to refine internal representations.
• The network uses top-down feedback connections to simulate expected sensory patterns based on inferred intentions.
• During training, raw-level visuo-proprioceptive patterns are consolidated into shared internal representations across modalities.
• Prediction error minimization drives both sensory prediction and intention inference, enabling mental simulation of unseen behaviors.
• The model is trained in a simulation environment where a robot imitates gestures from another robot, using intention states as supervisory signals.

Experimental results

Research questions

• RQ1 Can a deep dynamic neural network infer the underlying intention of observed visuomotor behaviors through prediction error minimization?
• RQ2 To what extent can the model simulate dynamic visuo-proprioceptive patterns in the absence of external sensory input?
• RQ3 Does minimizing prediction error in one modality (e.g., visual) trigger the recall of corresponding representations in another modality (e.g., proprioceptive)?
• RQ4 How does the predictive coding framework support the integration of perception and action in a continuous visuomotor learning process?
• RQ5 Can the model’s internal dynamics emulate the functional properties of mirror neuron systems as described by predictive coding?

Key findings

• The model successfully generated mental simulations of dynamic visuo-proprioceptive patterns in a top-down manner without external sensory input, given intention states.
• Prediction error minimization enabled accurate inference of hidden intentions behind observed visuomotor sequences, supporting the predictive coding account of mirror neurons.
• Minimizing prediction error in one modality induced the recall of corresponding representations in another modality, demonstrating cross-modal integration.
• The model achieved seamless coordination between perception and action through end-to-end learning under the predictive coding framework.
• The results highlight the role of prediction error as a unifying mechanism for perception, inference, and mental simulation in visuomotor learning.
• The model demonstrated robust performance in a robotic imitation task, validating its applicability to epigenetic and developmental robotics.