[Paper Review] Deep Predictive Coding Network for Object Recognition

Haiguang Wen, Kuan Han+4 more|arXiv (Cornell University)|Feb 13, 2018

Cell Image Analysis Techniques53 references41 citations

TL;DR

A bi-directional, recurrent neural network based on predictive coding that performs recursive bottom-up and top-down processing to enhance object recognition, outperforming a feedforward-only baseline on standard datasets as computation cycles increase.

ABSTRACT

Based on the predictive coding theory in neuroscience, we designed a bi-directional and recurrent neural net, namely deep predictive coding networks (PCN). It has feedforward, feedback, and recurrent connections. Feedback connections from a higher layer carry the prediction of its lower-layer representation; feedforward connections carry the prediction errors to its higher-layer. Given image input, PCN runs recursive cycles of bottom-up and top-down computation to update its internal representations and reduce the difference between bottom-up input and top-down prediction at every layer. After multiple cycles of recursive updating, the representation is used for image classification. With benchmark data (CIFAR-10/100, SVHN, and MNIST), PCN was found to always outperform its feedforward-only counterpart: a model without any mechanism for recurrent dynamics. Its performance tended to improve given more cycles of computation over time. In short, PCN reuses a single architecture to recursively run bottom-up and top-down processes. As a dynamical system, PCN can be unfolded to a feedforward model that becomes deeper and deeper over time, while refining it representation towards more accurate and definitive object recognition.

Motivation & Objective

• Translate predictive coding theory into a deep network with feedforward, feedback, and recurrent connections for object recognition.
• Develop a model that updates internal representations through recursive bottom-up and top-down cycles to minimize prediction error at each layer.
• Evaluate the approach on standard benchmarks (CIFAR-10/100, SVHN, MNIST) and compare to a feedforward-only baseline.
• Analyze how increasing the number of computation cycles affects recognition performance.

Proposed method

• Design a bi-directional, recurrent neural network called deep predictive coding network (PCN) with feedback from higher layers and recurrent connections.
• Higher-layer feedback carries predictions of lower-layer representations while bottom-up signals carry prediction errors to higher layers.
• Iteratively run bottom-up and top-down processing cycles to reduce the difference between input and top-down predictions across layers.
• Unfold the dynamical system to a progressively deeper feedforward model over time as representations refine for object recognition.
• Empirically evaluate PCN on CIFAR-10/100, SVHN, and MNIST and compare against a non-recurrent feedforward counterpart.

Experimental results

Research questions

• RQ1 Can predictive-coding–based networks outperform traditional feedforward networks on standard object recognition benchmarks?
• RQ2 How does the number of recursive computation cycles influence recognition accuracy on these benchmarks?
• RQ3 Do higher-layer feedback and recurrent connections contribute to more accurate internal representations for classification?

Key findings

• PCN outperforms its feedforward-only counterpart across benchmark datasets.
• Performance improves as more computation cycles are executed over time.
• The model reuses a single architecture to perform recursive bottom-up and top-down processing, refining representations for more accurate recognition.