Predictive Coding Based Multiscale Network with Encoder-Decoder LSTM for Video Prediction

by   Chaofan Ling, et al.

We are introducing a multi-scale predictive model for video prediction here, whose design is inspired by the "Predictive Coding" theories and "Coarse to Fine" approach. As a predictive coding model, it is updated by a combination of bottom-up and top-down information flows, which is different from traditional bottom-up training style. Its advantage is to reduce the dependence on input information and improve its ability to predict and generate images. Importantly, we achieve with a multi-scale approach – higher level neurons generate coarser predictions (lower resolution), while the lower level generate finer predictions (higher resolution). This is different from the traditional predictive coding framework in which higher level predict the activity of neurons in lower level. To improve the predictive ability, we integrate an encoder-decoder network in the LSTM architecture and share the final encoded high-level semantic information between different levels. Additionally, since the output of each network level is an RGB image, a smaller LSTM hidden state can be used to retain and update the only necessary hidden information, avoiding being mapped to an overly discrete and complex space. In this way, we can reduce the difficulty of prediction and the computational overhead. Finally, we further explore the training strategies, to address the instability in adversarial training and mismatch between training and testing in long-term prediction. Code is available at


page 1

page 4

page 5

page 8

page 9

page 10


Pyramidal Predictive Network: A Model for Visual-frame Prediction Based on Predictive Coding Theory

Visual-frame prediction is a pixel-dense prediction task that infers fut...

Multi Resolution LSTM For Long Term Prediction In Neural Activity Video

Epileptic seizures are caused by abnormal, overly syn- chronized, electr...

Analyzing and Improving the Pyramidal Predictive Network for Future Video Frame Prediction

The pyramidal predictive network (PPNV1) proposes an interesting tempora...

Encoding Longer-term Contextual Multi-modal Information in a Predictive Coding Model

Studies suggest that within the hierarchical architecture, the topologic...

Rethinking Coarse-to-Fine Approach in Single Image Deblurring

Coarse-to-fine strategies have been extensively used for the architectur...

Small Lesion Segmentation in Brain MRIs with Subpixel Embedding

We present a method to segment MRI scans of the human brain into ischemi...

Learning Accurate Extended-Horizon Predictions of High Dimensional Trajectories

We present a novel predictive model architecture based on the principles...

Please sign up or login with your details

Forgot password? Click here to reset