Precipitation nowcasting refers to the problem of providing very short range (e.g., 0-6 hours) forecast of the rainfall intensity in a local region based on radar echo maps111The radar echo maps are Constant Altitude Plan Position Indicator (CAPPI) images which can be converted to rainfall intensity maps using the Marshall-Palmer relationship or Z-R relationship marshall1948distribution ., rain gauge and other observation data as well as the Numerical Weather Prediction
(NWP) models. It significantly impacts the daily lives of many and plays a vital role in many real-world applications. Among other possibilities, it helps to facilitate drivers by predicting road conditions, enhances flight safety by providing weather guidance for regional aviation, and avoids casualties by issuing citywide rainfall alerts. In addition to the inherent complexities of the atmosphere and relevant dynamical processes, the ever-growing need for real-time, large-scale, and fine-grained precipitation nowcasting poses extra challenges to the meteorological community and has aroused research interest in the machine learning communityxingjian2015convolutional ; sun2014use .
The conventional approaches to precipitation nowcasting used by existing operational systems rely on optical flow woo2017operational
. In a modern day nowcasting system, the convective cloud movements are first estimated from the observed radar echo maps by optical flow and are then used to predict the future radar echo maps using semi-Lagrangian advection. However, these methods are unsupervised from the machine learning point of view in that they do not take advantage of the vast amount of existing radar echo data. Recently, progress has been made by utilizing superviseddeep learning lecun2015deep techniques for precipitation nowcasting. Shi et al. xingjian2015convolutional formulated precipitation nowcasting as a spatiotemporal sequence forecasting problem and proposed the
Convolutional Long Short-Term Memory(ConvLSTM) model, which extends the LSTM hochreiter1997long by having convolutional structures in both the input-to-state and state-to-state transitions, to solve the problem. Using the radar echo sequences for model training, the authors showed that ConvLSTM is better at capturing the spatiotemporal correlations than the fully-connected LSTM and gives more accurate predictions than the Real-time Optical flow by Variational methods for Echoes of Radar (ROVER) algorithm woo2017operational currently used by the Hong Kong Observatory (HKO).
However, despite their pioneering effort in this interesting direction, the paper has some deficiencies. First, the deep learning model is only evaluated on a relatively small dataset containing 97 rainy days and only the nowcasting skill score at the 0.5mm/h rain-rate threshold is compared. As real-world precipitation nowcasting systems need to pay additional attention to heavier rainfall events such as rainstorms which cause more threat to the society, the performance at the 0.5mm/h threshold (indicating raining or not) alone is not sufficient for demonstrating the algorithm’s overall performance woo2017operational . In fact, as the area Deep Learning for Precipitation Nowcasting is still in its early stage, it is not clear how models should be evaluated to meet the need of real-world applications. Second, although the convolutional recurrence structure used in ConvLSTM is better than the fully-connected recurrent structure in capturing spatiotemporal correlations, it is not optimal and leaves room for improvement. For motion patterns like rotation and scaling, the local correlation structure of consecutive frames will be different for different spatial locations and timestamps. It is thus inefficient to use convolution which uses a location-invariant filter to represent such location-variant
relationship. Previous attempts have tried to solve the problem by revising the output of a recurrent neural network (RNN) from the raw prediction to be some location-variant transformation of the input, like optical flow or dynamic local filterfinn2016unsupervised ; de2016dynamic . However, not much research has been conducted to address the problem by revising the recurrent structure itself.
In this paper, we aim to address these two problems by proposing both a benchmark and a new model for precipitation nowcasting. For the new benchmark, we build the HKO-7 dataset which contains radar echo data from 2009 to 2015 near Hong Kong. Since the radar echo maps arrive in a stream in the real-world scenario, the nowcasting algorithms can adopt online learning to adapt to the newly emerging patterns dynamically. To take into account this setting, we use two testing protocols in our benchmark: the offline setting in which the algorithm can only use a fixed window of the previous radar echo maps and the online setting in which the algorithm is free to use all the historical data and any online learning algorithm. Another issue for the precipitation nowcasting task is that the proportions of rainfall events at different rain-rate thresholds are highly imbalanced. Heavier rainfall occurs less often but has a higher real-world impact. We thus propose the Balanced Mean Squared Error (B-MSE) and Balanced Mean Absolute Error
(B-MAE) measures for training and evaluation, which assign more weights to heavier rainfalls in the calculation of MSE and MAE. We empirically find that the balanced variants of the loss functions are more consistent with the overall nowcasting performance at multiple rain-rate thresholds than the original loss functions. Moreover, our experiments show that training with the balanced loss functions is essential for deep learning models to achieve good performance at higher rain-rate thresholds. For the new model, we propose the
Trajectory Gated Recurrent Unit(TrajGRU) model which uses a subnetwork to output the state-to-state connection structures before state transitions. TrajGRU allows the state to be aggregated along some learned trajectories and thus is more flexible than the Convolutional GRU (ConvGRU) ballas2016delving whose connection structure is fixed. We show that TrajGRU outperforms ConvGRU, Dynamic Filter Network (DFN) de2016dynamic as well as 2D and 3D Convolutional Neural Networks (CNNs) mathieu2015deep ; vondrick2016generating in both a synthetic MovingMNIST++ dataset and the HKO-7 dataset.
Using the new dataset, testing protocols, training loss and model, we provide extensive empirical evaluation of seven models, including a simple baseline model which always predicts the last frame, two optical flow based models (ROVER and its nonlinear variant), and four representative deep learning models (TrajGRU, ConvGRU, 2D CNN, and 3D CNN). We also provide a large-scale benchmark for precipitation nowcasting. Our experimental validation shows that (1) all the deep learning models outperform the optical flow based models, (2) TrajGRU attains the best overall performance among all the deep learning models, and (3) after applying online fine-tuning, the models tested in the online setting consistently outperform those in the offline setting. To the best of our knowledge, this is the first comprehensive benchmark of deep learning models for the precipitation nowcasting problem. Besides, since precipitation nowcasting can be viewed as a video prediction problem ranzato2014video ; vondrick2016generating , our work is the first to provide evidence and justification that online learning could potentially be helpful for video prediction in general.
2 Related Work
Deep learning for precipitation nowcasting and video prediction
For the precipitation nowcasting problem, the reflectivity factors in radar echo maps are first transformed to grayscale images before being fed into the prediction algorithm xingjian2015convolutional . Thus, precipitation nowcasting can be viewed as a type of video prediction problem with a fixed “camera”, which is the weather radar. Therefore, methods proposed for predicting future frames in natural videos are also applicable to precipitation nowcasting and are related to our paper. There are three types of general architecture for video prediction: RNN based models, 2D CNN based models, and 3D CNN based models. Ranzato et al. ranzato2014video proposed the first RNN based model for video prediction, which uses a convolutional RNN with state-state kernel to encode the observed frames. Srivastava et al. srivastava2015unsupervised proposed the LSTM encoder-decoder network which uses one LSTM to encode the input frames and another LSTM to predict multiple frames ahead. The model was generalized in xingjian2015convolutional by replacing the fully-connected LSTM with ConvLSTM to capture the spatiotemporal correlations better. Later, Finn et al. finn2016unsupervised and De Brabandere et al. de2016dynamic extended the model in xingjian2015convolutional by making the network predict the transformation of the input frame instead of directly predicting the raw pixels. Ruben et al. villegas2017decomposing proposed to use both an RNN that captures the motion and a CNN that captures the content to generate the prediction. Along with RNN based models, 2D and 3D CNN based models were proposed in mathieu2015deep and vondrick2016generating respectively. Mathieu et al. mathieu2015deep treated the frame sequence as multiple channels and applied 2D CNN to generate the prediction while vondrick2016generating treated them as the depth and applied 3D CNN. Both papers show that Generative Adversarial Network (GAN) goodfellow2014generative is helpful for generating sharp predictions.
Structured recurrent connection for spatiotemporal modeling
From a higher-level perspective, precipitation nowcasting and video prediction are intrinsically spatiotemporal sequence forecasting problems in which both the input and output are spatiotemporal sequences xingjian2015convolutional . Recently, there is a trend of replacing the fully-connected structure in the recurrent connections of RNN with other topologies to enhance the network’s ability to model the spatiotemporal relationship. Other than the ConvLSTM which replaces the full-connection with convolution and is designed for dense videos, the SocialLSTM alahi2016social and the Structural-RNN (S-RNN) jain2016structural have been proposed sharing a similar notion. SocialLSTM defines the topology based on the distance between different people and is designed for human trajectory prediction while S-RNN defines the structure based on the given spatiotemporal graph. All these models are different from our TrajGRU in that our model actively learns the recurrent connection structure. Liang et al. liang2017interpretable have proposed the Structure-evolving LSTM, which also has the ability to learn the connection structure of RNNs. However, their model is designed for the semantic object parsing task and learns how to merge the graph nodes automatically. It is thus different from TrajGRU which aims at learning the local correlation structure for spatiotemporal data.
Benchmark for video tasks
There exist benchmarks for several video tasks like online object tracking wu2013online and video object segmentation perazzi2016benchmark . However, there is no benchmark for the precipitation nowcasting problem, which is also a video task but has its unique properties since radar echo map is a completely different type of data and the data is highly imbalanced (as mentioned in Section 1). The large-scale benchmark created as part of this work could help fill the gap.
In this section, we present our new model for precipitation nowcasting. We first introduce the general encoding-forecasting structure used in this paper. Then we review the ConvGRU model and present our new TrajGRU model.
3.1 Encoding-forecasting Structure
We adopt a similar formulation of the precipitation nowcasting problem as in xingjian2015convolutional . Assume that the radar echo maps form a spatiotemporal sequence . At a given timestamp , our model generates the most likely -step predictions, , based on the previous observations including the current one: . Our encoding-forecasting network first encodes the observations into layers of RNN states: , and then uses another layers of RNNs to generate the predictions based on these encoded states: . Figure 2 illustrates our encoding-forecasting structure for
. We insert downsampling and upsampling layers between the RNNs, which are implemented by convolution and deconvolution with stride. The reason to reverse the order of the forecasting network is that the high-level states, which have captured the global spatiotemporal representation, could guide the update of the low-level states. Moreover, the low-level states could further influence the prediction. This structure is more reasonable than the previous structurexingjian2015convolutional
which does not reverse the link of the forecasting network because we are free to plug in additional RNN layers on top and no skip-connection is required to aggregate the low-level information. One can choose any type of RNNs like ConvGRU or our newly proposed TrajGRU in this general encoding-forecasting structure as long as their states correspond to tensors.
3.2 Convolutional GRU
The main formulas of the ConvGRU used in this paper are given as follows:
The bias terms are omitted for notational simplicity. ‘’ is the convolution operation and ‘’ is the Hadamard product. Here, are the memory state, reset gate, update gate, and new information, respectively. is the input and
is the activation, which is chosen to be leaky ReLU with negative slope equals to 0.2maas2013rectifier througout the paper. are the height and width of the state and input tensors and are the channel sizes of the state and input tensors, respectively. Every time a new input arrives, the reset gate will control whether to clear the previous state and the update gate will control how much the new information will be written to the state.
3.3 Trajectory GRU
When used for capturing spatiotemporal correlations, the deficiency of ConvGRU and other ConvRNNs is that the connection structure and weights are fixed for all the locations. The convolution operation basically applies a location-invariant filter to the input. If the inputs are all zero and the reset gates are all one, we could rewrite the computation process of the new information at a specific location at timestamp , i.e, , as follows:
Here, is the ordered neighborhood set at locationyu2016multi . is the th neighborhood location of position . The
function concatenates the inner vectors in the set andis the matrix representation of the state-to-state convolution weights.
As the hyperparameter of convolution is fixed, the neighborhood set stays the same for all locations. However, most motion patterns have different neighborhood sets for different locations. For example, rotation and scaling generate flow fields with different angles pointing to different directions. It would thus be more reasonable to have a location-variant connection structure as
where is the total number of local links, is the th neighborhood parameterized by .
Based on this observation, we propose the TrajGRU, which uses the current input and previous state to generate the local neighborhood set for each location at each timestamp. Since the location indices are discrete and non-differentiable, we use a set of continuous optical flows to represent these “indices”. The main formulas of TrajGRU are given as follows:
Here, is the total number of allowed links. are the flow fields that store the local connection structure generated by the structure generating network . The are the weights for projecting the channels, which are implemented by convolutions. The function selects the positions pointed out by from via the bilinear sampling kernel jaderberg2015spatial ; ilg2017flownet . If we denote where and , we have:
The advantage of such a structure is that we could learn the connection topology by learning the parameters of the subnetwork . In our experiments, takes the concatenation of and as the input and is fixed to be a one-hidden-layer convolutional neural network with kernel size and 32 feature maps. Thus, has only a small number of parameters and adds nearly no cost to the overall computation. Compared to a ConvGRU with state-to-state convolution, TrajGRU is able to learn a more efficient connection structure with . For ConvGRU and TrajGRU, the number of model parameters is dominated by the size of the state-to-state weights, which is for TrajGRU and for ConvGRU. If is chosen to be smaller than , the number of parameters of TrajGRU can also be smaller than the ConvGRU and the TrajGRU model is able to use the parameters more efficiently. Illustration of the recurrent connection structures of ConvGRU and TrajGRU is given in Figure 2. Recently, Jeon & Kim jeon2017active has used similar ideas to extend the convolution operations in CNN. However, their proposed Active Convolution Unit (ACU) focuses on the images where the need for location-variant filters is limited. Our TrajGRU focuses on videos where location-variant filters are crucial for handling motion patterns like rotations. Moreover, we are revising the structure of the recurrent connection and have tested different number of links while jeon2017active fixes the link number to 9.
4 Experiments on MovingMNIST++
Before evaluating our model on the more challenging precipitation nowcasting task, we first compare TrajGRU with ConvGRU, DFN and 2D/3D CNNs on a synthetic video prediction dataset to justify its effectiveness.
The previous MovingMNIST dataset srivastava2015unsupervised ; xingjian2015convolutional only moves the digits with a constant speed and is not suitable for evaluating different models’ ability in capturing more complicated motion patterns. We thus design the MovingMNIST++ dataset by extending MovingMNIST to allow random rotations, scale changes, and illumination changes. Each frame is of size and contains three moving digits. We use 10 frames as input to predict the next 10 frames. As the frames have illumination changes, we use MSE instead of cross-entropy for training and evaluation 222The MSE for the MovingMNIST++ experiment is averaged by both the frame size and the length of the predicted sequence.. We train all models using the Adam optimizer kingma2014adam with learning rate equal to and momentum equal to 0.5. For the RNN models, we use the encoding-forecasting structure introduced previously with three RNN layers. All RNNs are either ConvGRU or TrajGRU and all use the same set of hyperparameters. For TrajGRU, we initialize the weight of the output layer of the structure generating network to zero. The strides of the middle downsampling and upsampling layers are chosen to be . The numbers of filters for the three RNNs are respectively. For the DFN model, we replace the output layer of ConvGRU with a local filter and transform the previous frame to get the prediction. For the RNN models, we train them for 200,000 iterations with norm clipping threshold equal to 10 and batch size equal to 4. For the CNN models, we train them for 100,000 iterations with norm clipping threshold equal to 50 and batch size equal to 32. The detailed experimental configuration of the models for the MovingMNIST++ experiment can be found in the appendix. We have also tried to use conditional GAN for the 2D and 3D models but have failed to get reasonable results.
Table 1 gives the results of different models on the same test set that contains 10,000 sequences. We train all models using three different seeds to report the standard deviation. We can find that TrajGRU with only 5 links outperforms ConvGRU with state-to-state kernel size and dilation (9 links). Also, the performance of TrajGRU improves as the number of links increases. TrajGRU with outperforms ConvGRU with state-to-state kernel and yet has fewer parameters. Another observation from the table is that DFN does not perform well in this synthetic dataset. This is because DFN uses softmax to enhance the sparsity of the learned local filters, which fails to model illumination change because the maximum value always gets smaller after convolving with a positive kernel whose weights sum up to 1. For DFN, when the pixel values get smaller, it is impossible for them to increase again. Figure 3 visualizes the learned structures of TrajGRU. We can see that the network has learned reasonable local link patterns.
5 Benchmark for Precipitation Nowcasting
5.1 HKO-7 Dataset
The HKO-7 dataset used in the benchmark contains radar echo data from 2009 to 2015 collected by HKO. The radar CAPPI reflectivity images, which have resolution of pixels, are taken from an altitude of 2km and cover a
area centered in Hong Kong. The data are recorded every 6 minutes and hence there are 240 frames per day. The raw logarithmic radar reflectivity factors are linearly transformed to pixel values viaand are clipped to be between 0 and 255. The raw radar echo images generated by Doppler weather radar are noisy due to factors like ground clutter, sea clutter, anomalous propagation and electromagnetic interference lee2017ensemble . To alleviate the impact of noise in training and evaluation, we filter the noisy pixels in the dataset and generate the noise masks by a two-stage process described in the appendix.
As rainfall events occur sparsely, we select the rainy days based on the rain barrel information to form our final dataset, which has 812 days for training, 50 days for validation and 131 days for testing. Our current treatment is close to the real-life scenario as we are able to train an additional model that classifies whether or not it will rain on the next day and applies our precipitation nowcasting model if this coarser-level model predicts that it will be rainy. The radar reflectivity values are converted to rainfall intensity values (mm/h) using the Z-R relationship:where is the rain-rate level, and . The overall statistics and the average monthly rainfall distribution of the HKO-7 dataset are given in the appendix.
5.2 Evaluation Methodology
As the radar echo maps arrive in a stream, nowcasting algorithms can apply online learning to adapt to the newly emerging spatiotemporal patterns. We propose two settings in our evaluation protocol: (1) the offline setting in which the algorithm always receives 5 frames as input and predicts 20 frames ahead, and (2) the online setting in which the algorithm receives segments of length 5 sequentially and predicts 20 frames ahead for each new segment received. The evaluation protocol is described more systematically in the appendix. The testing environment guarantees that the same set of sequences is tested in both the offline and online settings for fair comparison.
For both settings, we evaluate the skill scores for multiple thresholds that correspond to different rainfall levels to give an all-round evaluation of the algorithms’ nowcasting performance. Table 2 shows the distribution of different rainfall levels in our dataset. We choose to use the thresholds 0.5, 2, 5, 10, 30 to calculate the CSI and Heidke Skill Score (HSS) hogan2010equitability . For calculating the skill score at a specific threshold , which is 0.5, 2, 5, 10 or 30, we first convert the pixel values in prediction and ground-truth to 0/1 by thresholding with . We then calculate the TP (prediction=1, truth=1), FN (prediction=0, truth=1), FP (prediction=1, truth=0), and TN (prediction=0, truth=0). The CSI score is calculated as and the HSS score is calculated as . During the computation, the masked points are ignored.
As shown in Table 2, the frequencies of different rainfall levels are highly imbalanced. We propose to use the weighted loss function to help solve this problem. Specifically, we assign a weight to each pixel according to its rainfall intensity : . Also, the masked pixels have weight 0. The resulting B-MSE and B-MAE scores are computed as and , where is the total number of frames and is the weight corresponding to the th pixel in the th frame. For the conventional MSE and MAE measures, we simply set all the weights to 1 except the masked points.
|Rain Rate (mm/h)||Proportion (%)||Rainfall Level|
|90.25||No / Hardly noticeable|
|2.46||Light to moderate|
|1.14||Moderate to heavy|
5.3 Evaluated Algorithms
We have evaluated seven nowcasting algorithms, including the simplest model which always predicts the last frame, two optical flow based methods (ROVER and its nonlinear variant), and four deep learning methods (TrajGRU, ConvGRU, 2D CNN, and 3D CNN). Specifically, we have evaluated the performance of deep learning models in the online setting by fine-tuning the algorithms using AdaGrad duchi2011adaptive with learning rate equal to . We optimize the sum of B-MSE and B-MAE during offline training and online fine-tuning. During the offline training process, all models are optimized by the Adam optimizer with learning rate equal to and momentum equal to and we train these models with early-stopping on the sum of B-MSE and B-MAE. For RNN models, the training batch size is set to 4. For the CNN models, the training batch size is set to 8. For TrajGRU and ConvGRU models, we use a 3-layer encoding-forecasting structure with the number of filters for the RNNs set to . We use kernel size equal to for the ConvGRU models while the number of links is set to for the TrajGRU model. We also train the ConvGRU model with the original MSE and MAE loss, which is named “ConvGRU-nobal”, to evaluate the improvement by training with the B-MSE and B-MAE loss. The other model configurations including ROVER, ROVER-nonlinear and deep models are included in the appendix.
5.4 Evaluation Results
The overall evaluation results are summarized in Table 3
. In order to analyze the confidence interval of the results, we train 2D CNN, 3D CNN, ConvGRU and TrajGRU models using three different random seeds and report the standard deviation in Table4. We find that training with balanced loss functions is essential for good nowcasting performance of heavier rainfall. The ConvGRU model that is trained without balanced loss, which best represents the model in xingjian2015convolutional , has worse nowcasting score than the optical flow based methods at the 10mm/h and 30mm/h thresholds. Also, we find that all the deep learning models that are trained with the balanced loss outperform the optical flow based models. Among the deep learning models, TrajGRU performs the best and 3D CNN outperforms 2D CNN, which shows that an appropriate network structure is crucial to achieving good performance. The improvement of TrajGRU over the other models is statistically significant because the differences in B-MSE and B-MAE are larger than three times their standard deviation. Moreover, the performance with online fine-tuning enabled is consistently better than that without online fine-tuning, which verifies the effectiveness of online learning at least for this task.
Based on the evaluation results, we also compute the Kendall’s coefficients kendall1938new between the MSE, MAE, B-MSE, B-MAE and the CSI, HSS at different thresholds. As shown in Table 5, B-MSE and B-MAE have stronger correlation with the CSI and HSS in most cases.
|ROVER + Linear||0.4762||0.4089||0.3151||0.2146||0.1067||0.6038||0.5473||0.4516||0.3301||0.1762||11651||23437|
|ROVER + Non-linear||0.4655||0.4074||0.3226||0.2164||0.0951||0.5896||0.5436||0.4590||0.3318||0.1576||10945||22857|
6 Conclusion and Future Work
In this paper, we have provided the first large-scale benchmark for precipitation nowcasting and have proposed a new TrajGRU model with the ability of learning the recurrent connection structure. We have shown TrajGRU is more efficient in capturing the spatiotemporal correlations than ConvGRU. For future work, we plan to test if TrajGRU helps improve other spatiotemporal learning tasks like visual object tracking and video segmentation. We will also try to build an operational nowcasting system using the proposed algorithm.
This research has been supported by General Research Fund 16207316 from the Research Grants Council and Innovation and Technology Fund ITS/205/15FP from the Innovation and Technology Commission in Hong Kong. The first author has also been supported by the Hong Kong PhD Fellowship.
-  Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social LSTM: Human trajectory prediction in crowded spaces. In CVPR, 2016.
-  Nicolas Ballas, Li Yao, Chris Pal, and Aaron Courville. Delving deeper into convolutional networks for learning video representations. In ICLR, 2016.
-  Bert De Brabandere, Xu Jia, Tinne Tuytelaars, and Luc Van Gool. Dynamic filter networks. In NIPS, 2016.
-  John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul):2121–2159, 2011.
-  Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In NIPS, 2016.
-  Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.In ICCV, 2015.
-  Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997.
-  Robin J Hogan, Christopher AT Ferro, Ian T Jolliffe, and David B Stephenson. Equitability revisited: Why the “equitable threat score” is not equitable. Weather and Forecasting, 25(2):710–726, 2010.
-  Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In CVPR, 2017.
-  S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
-  Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. In NIPS, 2015.
-  Ashesh Jain, Amir R Zamir, Silvio Savarese, and Ashutosh Saxena. Structural-RNN: Deep learning on spatio-temporal graphs. In CVPR, 2016.
-  Yunho Jeon and Junmo Kim. Active convolution: Learning the shape of convolution for image classification. In CVPR, 2017.
-  Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81–93, 1938.
-  Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
-  Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436–444, 2015.
-  Hansoo Lee and Sungshin Kim. Ensemble classification for anomalous propagation echo detection with clustering-based subset-selection method. Atmosphere, 8(1):11, 2017.
-  Xiaodan Liang, Liang Lin, Xiaohui Shen, Jiashi Feng, Shuicheng Yan, and Eric P Xing. Interpretable structure-evolving LSTM. In CVPR, 2017.
-  Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlinearities improve neural network acoustic models. In ICML, 2013.
-  John S Marshall and W Mc K Palmer. The distribution of raindrops with size. Journal of Meteorology, 5(4):165–166, 1948.
-  Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. In ICLR, 2016.
-  Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung. A benchmark dataset and evaluation methodology for video object segmentation. In CVPR, 2016.
-  MarcAurelio Ranzato, Arthur Szlam, Joan Bruna, Michael Mathieu, Ronan Collobert, and Sumit Chopra. Video (language) modeling: a baseline for generative models of natural videos. arXiv preprint arXiv:1412.6604, 2014.
-  Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-kin Wong, and Wang-chun Woo. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In NIPS, 2015.
-  Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov. Unsupervised learning of video representations using LSTMs. In ICML, 2015.
-  Juanzhen Sun, Ming Xue, James W Wilson, Isztar Zawadzki, Sue P Ballard, Jeanette Onvlee-Hooimeyer, Paul Joe, Dale M Barker, Ping-Wah Li, Brian Golding, et al. Use of NWP for nowcasting convective precipitation: Recent progress and challenges. Bulletin of the American Meteorological Society, 95(3):409–426, 2014.
-  Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. In ICLR, 2017.
-  Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In NIPS, 2016.
-  Wang-chun Woo and Wai-kin Wong. Operational application of optical flow techniques to radar-based rainfall nowcasting. Atmosphere, 8(3):48, 2017.
-  Yi Wu, Jongwoo Lim, and Ming-Hsuan Yang. Online object tracking: A benchmark. In CVPR, 2013.
-  Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. In ICLR, 2016.
Appendix A Weight Initialization
The weights and biases of all models are initialized with the MSRA initializer  except that the weights and biases of the structure generating network in TrajGRUs are initialized to be zero.
Appendix B Structure Generating Network in TrajGRU
The structure generating network takes the concatenation of the state tensor and the input tensor as the input. We fix the network to have two convolution layers. The first convolution layer uses kernel size, padding size, 32 filters and uses the leaky ReLU activation. The second convolution layer uses kernel size, padding and filters where is the number of links.
Appendix C Details about the MovingMNIST++ Experiment
c.1 Generation Process
For each sequence, we choose three digits randomly from the MNIST dataset333MNIST dataset:http://yann.lecun.com/exdb/mnist/. Each digit will move, rotate, scale up or down at a randomly sampled speed. Also, we multiply the pixel values by an illumination factor every time to make the digits have time-varying appearances. The hyperparameters of the generation process are given in Table 6. In our experiment, we always generate a length-20 sequence and use the first 10 frames to predict the last 10 frames.
|Number of digits||3|
c.2 Network Structures
Appendix D Details about the HKO-7 Benchmark
d.1 Overall Data Statistics
d.2 Denoising Process
We first remove the ground clutter and sun spikes, which appear at a fixed position, by detecting the out-lier locations in the image. For each in-boundary location in the frame, we use the ratio of its pixel value equal to as the feature and estimate these features’ sample mean and covariance matrix . We then calculate the Mahalanobis distance 444We use Moore-Penrose pseudoinverse in the implementation. of these features using the estimated mean and covariance. Locations that have the Mahalanobis distances higher than the mean distance plus three times the standard deviation are classified as outliers. After out-lier detection, the locations in the image are divided into 177316 inliers, 2824 outliers and 50260 out-of-boundary points. The outlier detection process is illustrated in Figure
of these features using the estimated mean and covariance. Locations that have the Mahalanobis distances higher than the mean distance plus three times the standard deviation are classified as outliers. After out-lier detection, the
locations in the image are divided into 177316 inliers, 2824 outliers and 50260 out-of-boundary points. The outlier detection process is illustrated in Figure6. After out-lier detection, we further remove other types of noise, like sea clutter, by filtering out the pixels with value smaller than 71 and larger than 0. Two examples that compare the original radar echo sequence and the denoised sequence are included in the attached “denoising” folder.
d.3 Evaluation Protocol
We illustrate our evaluation protocol in Algorithm 1. We can choose the evaluation type to be ‘offline’ or ‘online’. In the online setting, the model is able to store the previously seen sequences in a buffer and fine-tune the parameters using the sampled training batches from the buffer. For algorithms that are tested in the online setting in the paper, we sample the last 25 consecutive frames in the buffer to update the model if these frames are available. The buffer will be made empty once a new episode flag is received, which indicates that the newly observed 5-frame segment is not consecutive to the previous frames.