[Paper Review] Predictive Encoding of Contextual Relationships for Perceptual Inference, Interpolation and Prediction

M. Zhao, Chengxu Zhuang+2 more|arXiv (Cornell University)|Nov 14, 2014

Neural dynamics and brain function25 references3 citations

TL;DR

This paper proposes a neurally-inspired predictive coding model that learns spatiotemporal contextual relationships to improve perceptual inference, interpolation, and prediction in image sequences. By using prediction error to update latent contextual representations—without altering feedforward inputs—it enables bidirectional inference, outperforms gated Boltzmann machines in prediction, and uniquely supports missing frame interpolation.

ABSTRACT

We propose a new neurally-inspired model that can learn to encode the global relationship context of visual events across time and space and to use the contextual information to modulate the analysis by synthesis process in a predictive coding framework. The model learns latent contextual representations by maximizing the predictability of visual events based on local and global contextual information through both top-down and bottom-up processes. In contrast to standard predictive coding models, the prediction error in this model is used to update the contextual representation but does not alter the feedforward input for the next layer, and is thus more consistent with neurophysiological observations. We establish the computational feasibility of this model by demonstrating its ability in several aspects. We show that our model can outperform state-of-art performances of gated Boltzmann machines (GBM) in estimation of contextual information. Our model can also interpolate missing events or predict future events in image sequences while simultaneously estimating contextual information. We show it achieves state-of-art performances in terms of prediction accuracy in a variety of tasks and possesses the ability to interpolate missing frames, a function that is lacking in GBM.

Motivation & Objective

• To develop a biologically plausible model that learns contextual relationships across space and time in visual sequences.
• To address the limitation of standard predictive coding and gated Boltzmann machines (GBM) in handling missing frame interpolation.
• To enable joint estimation of contextual information and generation of predictions or reconstructions using a unified framework.
• To improve inference and prediction accuracy by modulating image synthesis through learned latent contextual representations.
• To demonstrate that prediction error can be used to update contextual representations without replacing feedforward input, enhancing biological plausibility.

Proposed method

• The model uses a predictive coding framework where top-down feedback generates predictions and bottom-up error signals update contextual representations.
• Latent contextual representations are learned by maximizing the predictability of visual events using both top-down and bottom-up processes.
• Prediction error is used solely to update the contextual representation, not to alter the feedforward input to the next layer, enhancing biological plausibility.
• The model employs spatiotemporal filtering in the feedforward path, avoiding the need for N-way multiplicative interactions or neural synchrony required in GBMs.
• Contextual modulation rescales basis functions during image synthesis, enabling adaptive prediction and reconstruction.
• The framework is trained end-to-end by minimizing the prediction error between synthesized and observed image sequences.

Experimental results

Research questions

• RQ1 Can a predictive coding model learn and utilize spatiotemporal contextual relationships to improve perceptual inference and prediction?
• RQ2 Can the model perform interpolation of missing frames in image sequences, a capability lacking in standard GBM models?
• RQ3 Is the use of prediction error to update contextual representations only (without altering feedforward input) more biologically plausible than standard predictive coding?
• RQ4 How does the model’s performance in prediction and interpolation compare to state-of-the-art models like gated Boltzmann machines?
• RQ5 Can the model generalize to sequences with limited data, such as the NORB dataset, without requiring large-scale training?

Key findings

• The model achieves state-of-the-art performance in prediction accuracy, slightly outperforming gated Boltzmann machines (GBM) on tested image sequences.
• The model uniquely supports interpolation of missing frames in sequences, a capability not available in GBM due to its reliance on two-frame transformations.
• Quantitative results in Table 4.3 show the model's root mean square error (RMSE) for prediction and interpolation is comparable or superior to GBM and GAE on the NORB dataset.
• The model's use of prediction error to update only the contextual representation—without replacing feedforward input—aligns better with neurophysiological observations than standard predictive coding.
• Temporal resolution was improved by training filters with a quadrature phase relationship, yielding reasonable interpolation results as shown in Figure 6.
• The model demonstrates that contextual modulation via learned latent variables can effectively support generative and constructive aspects of visual perception beyond simple reconstruction.