Learning Spectral-Spatial-Temporal Features via a Recurrent Convolutional Neural Network for Change Detection in Multispectral Imagery

by   Lichao Mou, et al.
Università di Trento

Change detection is one of the central problems in earth observation and was extensively investigated over recent decades. In this paper, we propose a novel recurrent convolutional neural network (ReCNN) architecture, which is trained to learn a joint spectral-spatial-temporal feature representation in a unified framework for change detection in multispectral images. To this end, we bring together a convolutional neural network (CNN) and a recurrent neural network (RNN) into one end-to-end network. The former is able to generate rich spectral-spatial feature representations, while the latter effectively analyzes temporal dependency in bi-temporal images. In comparison with previous approaches to change detection, the proposed network architecture possesses three distinctive properties: 1) It is end-to-end trainable, in contrast to most existing methods whose components are separately trained or computed; 2) it naturally harnesses spatial information that has been proven to be beneficial to change detection task; 3) it is capable of adaptively learning the temporal dependency between multitemporal images, unlike most of algorithms that use fairly simple operation like image differencing or stacking. As far as we know, this is the first time that a recurrent convolutional network architecture has been proposed for multitemporal remote sensing image analysis. The proposed network is validated on real multispectral data sets. Both visual and quantitative analysis of experimental results demonstrates competitive performance in the proposed mode.



There are no comments yet.


page 1

page 3

page 4

page 5

page 6

page 8

page 10

page 11


Missing Data Reconstruction in Remote Sensing image with a Unified Spatial-Temporal-Spectral Deep Convolutional Neural Network

Because of the internal malfunction of satellite sensors and poor atmosp...

HybridSN: Exploring 3D-2D CNN Feature Hierarchy for Hyperspectral Image Classification

Hyperspectral image (HSI) classification is widely used for the analysis...

Hyperspectral Image Denoising Employing a Spatial-Spectral Deep Residual Convolutional Neural Network

Hyperspectral image (HSI) denoising is a crucial preprocessing procedure...

Sea Level Anomaly Prediction using Recurrent Neural Networks

Sea level change, one of the most dire impacts of anthropogenic global w...

Graph WaveNet for Deep Spatial-Temporal Graph Modeling

Spatial-temporal graph modeling is an important task to analyze the spat...

Summarized Network Behavior Prediction

This work studies the entity-wise topical behavior from massive network ...

Bi-Temporal Semantic Reasoning for the Semantic Change Detection of HR Remote Sensing Images

Semantic change detection (SCD) extends the change detection (CD) task t...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

With the development of remote sensing technology, every day massive amounts of remotely sensed data are produced from a rich number of spaceborne and airborne sensors; e.g., the Landsat 8 satellite is capable of imaging the entire Earth every 16 days in an eight-day offset from Landsat 7, and every 10 days the Sentinel-2 mission can provide a global coverage of the Earth’s land surface. For the Sentinel-2 mission alone, to date about 3.4 petabytes of data have been acquired. Triggered by these exciting existing and future observation capabilities, methodological research on the multitemporal data analysis is of great importance [1, 2]. Change detection is very crucial in the field of multitemporal image analysis, as it is able to identify land use or land cover differences in the same geographical area across a period of time and can be used in a large number of applications, to name a few, urban expansion, disaster assessment, resource management, and monitoring dynamics of land use [3, 4, 5].

In the literature, many methods have been proposed to better identify land cover changes [1]

. Among them, a widely used model is based on image algebra approaches. A classic one is change vector analysis (CVA) proposed by Malila in 1980 

[6]. CVA is designed to analyze possible multiple changes in pairs of multi-spectral pixels of bi-temporal images. Bovolo and Bruzzone [7] propose a formal definition and a theoretical study to of CVA in the polar domain. Later some extensions of the CVA model have been proposed, e.g., compressed CVA (CVA) [8]. CVA is used together with unsupervised threshold selection techniques based on different possible models of the data distribution. For example, the Rayleigh-Rice mixture density model [9]

has been recently used in the framework of the Expectation-Maximization (EM) algorithm.

In addition, some image transformation-based models have been proposed in change detection to improve detection performance. These approaches mainly aim at learning a new, transformed feature representation from the original spectral domain, in order to suppress unchanged regions and highlight the presence of changes in the new feature space. For example, principal component analysis (PCA), Gram-Schmidt transformation, multivariate alteration detection (MAD), slow feature analysis (SFA), sparse learning, and deep belief network (DBN) use transformation algorithms in change detection methods 

[10, 11, 12, 13, 14, 15]. PCA is one of the best known subspace learning algorithms and can be used on both difference images and stacked images [10, 16]

. The goal of Gram-Schmidt transformation is to reduce data correlation. MAD makes an attempt at maximizing variance of independently transformed variables 

[12] and is invariant to linear scaling of the input data. SFA [13] is able to extract the most temporally invariant component from multitemporal images to transform data into a new feature space and, in this space, differences in unchanged pixels are suppressed so that changed regions can be better separated. In [14], the authors apply sparse learning on stacked multitemporal images and expect that resulting sparse solutions do not vary greatly between the multitemporal data. In [15], the authors learn feature representations of two images with DBNs. Feature vectors issued from the two image acquisitions are stacked and used to learn a representation, where changes stand out more clearly. Using such feature representation, changes are more easily detected by image differencing.

Another important branch of change detection methods is based on classification approaches. For example, Bruzzone and Serpico [17] propose a supervised nonparametric model, based on the compound classification rule for minimum error, to detect land cover transitions between two remote sensing images acquired at different times. The main idea of this approach is to consider the temporal correlation between images in the classification without requiring complex training data. In [18], the authors use the Bayes rule for minimum error in the compound classification framework for detecting land cover transitions between pairs multisource images gathered at two different dates. In [19]

, the authors propose a multi-classifier architecture, which is composed of an ensemble of partially unsupervised classifiers, to detect changes or update land cover maps. Later, Bruzzone et al. 

[20] develop an effective system that employs an ensemble of nonparametric multitemporal classifiers to address the problem of detecting land cover transitions in multitemporal images. All these techniques consider different tradeoffs between modeling the temporal correlation in the training of the system and requiring complex training data.

One crucial issue in change detection is modeling the temporal correlation between bi-temporal images. Various atmospheric scattering conditions, complicated light scattering mechanisms, and intra-class variability lead to change detection being inherently nonlinear. Thus sophisticated, task-driven, learning-based methods are desirable.

Deep neural networks have recently been shown to be very successful on a variety of computer vision and remote sensing tasks 

[21, 22]

. They can also provide the opportunity for change detection, where one would like to extract joint spectral-temporal features from a bi-temporal image sequence in an end-to-end manner. In this respect, as an important branch of deep learning family, a recurrent neural network (RNN) is a natural candidate to tackle the temporal connection between multitemporal sequence data in change detection tasks. Recently, Lyu et al. 

[23] make use of an end-to-end RNN to solve the multi/hyper-spectral image change detection task, since RNN is well known to be good at processing sequential data. In their framework, a long short-term memory (LSTM)-based RNN is employed to learn a joint spectral-temporal feature representation from a bi-temporal image sequence. In addition, the authors also show the versatility of their network by applying it to detect multi-class changes and pointing out a good transferability for change detection in an “unseen” scene without fine-tuning. The authors of [24] follow a similar idea, where an RNN based on LSTM units is used to extract dynamic spectral-temporal features but, in contrast to the change detection scenario, their goal is to address land cover classification of multitemporal image sequence.

In this paper, we would like to learn joint spectral-spatial-temporal features using an end-to-end network for change detection, which is named as recurrent convolutional neural network (ReCNN), since it combines convolutional neural network (CNN) and RNN. Although both CNN [25, 26, 27, 28, 29, 30, 31, 32, 33] and RNN [23, 34, 24] are well-established techniques for remote sensing applications, to the best of our knowledge, we are the first to combine them for multitemporal data analysis in the remote sensing community. Note that integrating CNN and RNN in an end-to-end manner has also been explored in hyperspectral image classification [35], where the network is only used for extracting spectral information to build a spectral classifier for the classification purpose. In our work, the CNN part transforms the input, a pair of 3D multispectral patches, to an abstract spectral-spatial feature representation, whereas the RNN part is not only employed for modeling temporal dependency, but is also used for predicting the final label (i.e., changed, unchanged, or change-type). In other words, the features from the proposed ReCNN encapsulate information related to spectral, spatial, and temporal components in bi-temporal images, making them useful for an holistic change detection task. For multitemporal image analysis, the proposed ReCNN contributes to the literature in three major aspects:

  • It is able to extract a spectral-spatial-temporal feature representation of multitemporal data through learning with a structured deep architecture.

  • It has the same property of 2D CNN used for multi/hyper-spectral data classification on learning informative spectral-spatial feature representations directly from multispectral data, requiring neither hand-crafted visual features nor pre-processing steps.

  • It has the same characteristic of RNN, being capable of modeling the temporal correlation between bi-temporal images using a sophisticated and task-driven approach to the extraction of temporal features in an end-to-end architecture, and finally producing labels for the image sequence.

The remainder of this paper is organized as follows. After the introductory Section I detailing change detection, Section II is dedicated to the details of the proposed recurrent convolutional network. Section III then provides data set information, network setup, experimental results, and discussion. Finally, Section IV concludes the paper.

Fig 1:

Overview of the proposed recurrent convolutional neural network (ReCNN). At the bottom of our network, convolutional layers automatically extract feature maps from each input. On top of the convolutional sub-network, a recurrent sub-network takes the feature representations produced by convolutional layers as inputs to exploit the temporal dependency in the bi-temporal images. The third part is two fully connected layers widely used in classification problems. Although ReCNN is composed of different kinds of network architectures, i.e., CNN, RNN, and fully connected network, it can be trained end-to-end by back-propagation with one loss function, due to the differential property of all these components.

Ii Methodology

Ii-a Network Architecture

The architecture of the proposed ReCNN, as shown in Fig. 1, is made up of three components, including a convolutional sub-network, a recurrent sub-network, and fully connected layers, from bottom to top.

To acquire a joint spectral-spatial-temporal feature representation for change detection, at the bottom of our network, convolutional layers automatically extract feature maps from each input. On top of the convolutional sub-network, a recurrent sub-network takes the feature representations produced by convolutional layers as inputs to exploit the temporal dependency in the bi-temporal images. The third part is two fully connected layers widely used in classification problems. Although ReCNN is composed of different kinds of network architectures (i.e., CNN, RNN, and fully connected network) it can be trained end-to-end by back-propagation with one loss function, due to the differential properties of all these components.

Let and represent a pair of multispectral images acquired over the same geographical area at two different times and , respectively. Let and be two patches taken from the exact same location in two images. is a label that indicates the category (i.e., changed, unchanged, or change-type) that the pair of patches belongs to. The flowchart of the proposed ReCNN can be summarized as follows:

  • First, the 3D multispectral patch is fed into branch of the convolutional sub-network, which transforms it to an abstract feature vector .

  • Then, the recurrent sub-network receives and calculates the hidden state information for the current input; it also restores that information in the meantime.

  • Subsequently, is input to branch for extracting spectral-spatial feature , it is fed into the recurrent layer simultaneously with the state information of , and the activation at time

    is computed by a linear interpolation between existing value and the activation of the previous time


  • Finally, fully connected layers of the ReCNN predict the label of the input bi-temporal multispectral patches by looping through the entire sequence.

The entire change detection map can be obtained by applying the network to all pixels in the image.

Fig 2: Illustration of traditional convolution operation (left) and 2-dilated convolution (right). Traditional convolution corresponds to dilated convolution with dilation rate 1. Employing dilated convolution operation enlarges the network’s field of view.

Ii-B Spectral-Spatial Feature Extraction via the Convolutional Sub-Network

As we have mentioned, spectral-spatial information is of great importance for change detection. Some of the previous widely used unsupervised image algebra-based and image transformation-based methods cannot totally capture task specialized features which may be discriminative for a specific change detection task. Features directly learned from data and driven by tasks are supposed to be better [22]. This advantage leads to our usage of a trainable feature generator.

Though trainable, early and fairly simple 1D neural network models, such as DBN [15]

and multilayer perceptron (MLP), suffer from huge amount of learnable parameters, since those architectures are totally equipped with fully connected layers, which is an undesirable case given that available annotated training samples for change detection are often very limited. Moreover, another disadvantage of such networks is that they treat the multispectral data as vectors, ignoring the 2D property of imagery in the spatial domain.

CNNs, which are a significant branch of deep learning, have been attracting attention, due to the fact that they are capable of automatically discovering relevant contextual 2D spatial features as well as spectral features for multi/hyper-spectral data. In addition, a CNN makes use of local connections to deal with spatial dependencies via sharing weights, and thus can significantly reduce the number of parameters of the network in comparison with the conventional 1D fully connected neural networks, e.g., DBN and MLP. Recently, CNNs used for hyperspectral image classification have proven their effectiveness in extracting useful spectral-spatial features [36, 29]. Triggered by this, adopting a CNN in our architecture is natural.

However, a direct use of CNNs commonly used in typical recognition tasks, e.g., AlexNet [37], VGG Nets [38], and GoogLeNet [39]

, is not possible in our task, as we believe that a simpler network architecture is more appropriate for our problem due to the following reasons. First, change detection aims to distinguish only several classes (two for binary change detection), which requires much less model complexity than general visual recognition problems in computer vision, such as ImageNet classification with 1,000 categories. Second, since spatial resolution of multispectral imagery is limited, it is desirable to make input size small, which reduces the depth of the network naturally. Third, a smaller network is obviously more efficient in change detection problems, where testing may be performed in a large-scale area. Finally, the above-mentioned networks are not suitable to be used on multispectral images with a large number of spectral channels.

The convolutional sub-network receives a sequence of multispectral patches as the input and has two separate, yet identical convolutional branches (i.e., branch and branch [cf. Fig. 1]) which process and in parallel, respectively. The learned features are fed into the following recurrent sub-network. Using this two-branch architecture, the convolutional recurrent neural network is constrained to first learn meaningful spectral-spatial representations of input patches, and to combine them on a higher level for modeling temporal dependency. More specifically, we make use of convolutional filters with a very small receptive field of , rather than using larger ones such as

. Moreover, we do not adopt max-pooling after convolution or spatial padding for convolutional layers. The depth of the convolutional sub-network is such that the output size of the last layer is


Regarding convolution, we make use of dilated convolution to construct convolutional layers in the network because, for our task, it is able to offer a slightly better performance than a traditional convolution operation. The dilated convolution [40] was originally designed for the efficient computation of the undecimated wavelet transform in the “algorithme à trous” scheme [41]. This algorithm makes it possible to calculate responses of any layer at any desirable resolution and can be applied post-hoc, once a network has been trained. Let be a discrete function. Let and let be a discrete filter of size . The traditional discrete convolution operation can be defined as follows:


This operation can be generalized. Let be a dilation rate and let be defined as


We will refer to as a dilated convolution or an -dilated convolution. Fig. 2 shows differences between the conventional convolution and the dilated convolution.

The usage of dilated convolution in our network allows us to exponentially enlarge the field of view with linearly increasing number of parameters, providing a significant parameter reduction while increasing effective field of view. Note that a very recent study [42] found that large field of view actually plays an important role. This can be easily understood by an analogy that states the fact that humans usually confirm the category of a pixel by referring to its surrounding context region.

Fig 3: Graphic models of fully connected RNN, LSTM, and GRU. In LSTM, , , , , and are output gates, forget gates, input gates, new memory cell contents, and memory cells, respectively. In GRU, the reset and update gates are denoted by and , and and are the candidate activation and final activation.

Ii-C Modeling Temporal Dependency by the Recurrent Sub-Network

The impressive success of recent deep learning systems has been predominantly achieved by feedforward neural network architectures like CNN. In such networks, we implicitly assume that all inputs are independent of each other. However, for tasks that involve processing time sequence (e.g., change detection), that is not a good assumption. RNNs are a kind of neural networks that extend the conventional feedforward neural networks with loops in connections. Unlike a feedforward network, an RNN is capable of dealing with dependent, sequential inputs by having a recurrent hidden state whose activation at each time step depends on that of the previous time. By doing so, the network can exhibit dynamic temporal behavior, which is in line with our purpose; i.e., modeling temporal dependency between the and

data. To this end, three types of RNN architectures, namely, fully connected RNN, LSTM, and gated recurrent unit (GRU), are used to construct the recurrent sub-network in our ReCNN.

Fully Connected RNN. Given feature vectors and learned from the convolutional sub-network, a fully connected RNN updates its recurrent hidden state by



is a nonlinear activation function, such as a hyperbolic tangent function or logistic sigmoid function. The recurrent layer will output a sequence

. For our task, we only need the last one as input to the fully connected layers for predicting label.

In the fully connected RNN model, the update of the recurrent hidden state in Eq. (3) is implemented as


where and are the coefficient matrices for the activation of recurrent hidden units at the previous time step and for the input at the present time, respectively.

Fully connected RNN is the concisest RNN model, and it can reflect the essence of RNNs; i.e., an RNN is capable of modeling a probability distribution over the next element of the sequence data, given its present state

, by capturing a distribution over sequence data. Let be the sequence probability, which can be decomposed into


Then, the conditional probability distribution can be modeled with an RNN:


where is obtained from Eq. (3). Our motivation in this work is apparent here: bi-temporal images act as true sequential data instead of a simple difference image or stacked image and, therefore, an RNN can be used to model the temporal dependency.

LSTM. LSTM is a special type of recurrent hidden unit and was initially proposed by Hochreiter and Schmidhuber [43]. Since then, a couple of minor modifications to the original version have been made. In this work, we follow the implementation of LSTM as used in [44]. As shown in Eq. (3), recurrent hidden units in a fully connected RNN simply compute a weight sum of inputs and then apply a nonlinear function. In contrast, an LSTM-based recurrent layer maintains a series of memory cells at time step . The activation of LSTM units can be calculated by


where is the hyperbolic tangent function and is the output gates that control the amount of memory content exposure. The output gates are updated by


where the terms represent coefficient matrices; e.g., and are the input-output weight matrix and memory-output weight matrix, respectively.

The memory cells are updated by partially discarding the present memory contents and adding new contents of the memory cells :


where is an element-wise multiplication. The new memory contents are


where is input-memory weight matrix and represents hidden-memory coefficient matrix.

The and are input gates and forget gates, respectively. The former modulates the extent to which the new memory information is added to the memory cell, whereas the latter controls the degree to which contents of the existing memory cells are forgotten. Specifically, gates are computed as follows:

Fig 4: True-color composites of the and images in the Taizhou data set as well as ground truths (GTs).

GRU. Similarly to LSTM, a GRU makes use of a linear sum between the existing state and the newly computed state. It, however, directly exposes whole state values at each time step, instead of controlling what part of the state information will be exposed.

The activation of GRUs at time step is a linear interpolation between the previous activation and the candidate activation :


where the update gates determine how much GRUs update their activations or contents. Update gates can be computed by


where and are the input-update coefficient matrix and hidden-update weight matrix, respectively.

The candidate activation is computed similarly to that of the fully connected RNN (cf. Eq. (3)) and as follows


where is the set of reset gates. When reset gates are totally off (i.e., is ), GRUs will completely forget the activation of the recurrent layer at previous time and only receive existing input. When open, reset gates will partially keep the information of the previously computed state. Reset gates are calculated similarly to update gates:


where is the input-reset weight matrix and represents the hidden-reset coefficient matrix.

Fig. 3 shows graphic models of fully connected RNN, LSTM, and GRU through time.

Ii-D Network Training

The network training is based on the TensorFlow framework. We chose Nesterov Adam 

[45, 46]

as the optimizer to train the network since, for this task, it shows much faster convergence than standard stochastic gradient descent (SGD) with momentum 

[47] or Adam [48]. We fixed almost all of parameters of Nesterov Aadam as recommended in [45]: , , , and a schedule decay of 0.004, making use of a fairly small learning rate of . All network weights are initialized with a Glorot uniform initializer [49]

that draws samples from a uniform distribution. We utilize sigmoid and softmax as activation functions of the last fully connected layer for the binary and multi-class change detections, respectively. Finally, we train our network on a single NVIDIA GeForce GTX TITAN with 12 GB of GPU memory.

Iii Experimental Results and Discussion

Iii-a Data Description

The performance of the proposed network is evaluated on two data sets, which were acquired by the Landsat Enhanced Thematic Mapper Plus (ETM+) sensor with six bands and a spatial resolution of 30m. Before feeding data into models, digital numbers (DNs) of the original data were converted into absolute radiance (i.e., all of the data sets used in the experiments were normalized into a range of [0,1]).

Iii-A1 Taizhou Data

This data set consists of two images covering the city of Taizhou, China, in March 2000 and February 2003, with a WGS-84 projection and a coordinate range of 311456N–312739N, 1200224E–1210745E. These two images both consist of pixels, and changes between them mainly involve city expansion. The available manually annotated samples of this data set for multi-class change detection cover four classes of interest (cf. Fig. 4); i.e., unchanged area, city expansion (bare soils, grasslands, or cultivated fields to buildings or roads), soil change (cultivated field to bare soil), and water change (non-water regions to water regions). Table I provides information about different classes and their corresponding training and test samples.

Class Name Training Test
Binary Changed region 500 4055
Unchanged region 500 16961
TOTAL 1000 21016
Multiple Unchanged region 500 16961
City expansion 500 2875
Soil change 500 104
Water change 500 75
TOTAL 2000 20015
TABLE I: Number of Training and Test Samples in the Taizhou Data Set

Iii-A2 Eppalock Lake

The second data set was acquired over the Eppalock lake, Victoria, Australia, in February 1991 and March 2009, with a WGS-84 projection and a coordinate range of 364910S–370052S, 1442752E–1443735E. Both images in this data set are pixels. Similar to the Taizhou data, four multi-class change types are considered in the Eppalock lake scene, and they are unchanged region, city expansion (bare soils, grasslands, or cultivated fields to buildings or roads), water loss (water regions to bare soils), and soil change (vegetative covers or artificial buildings to bare soils). Fig. 5 shows tow true-color composite images and their corresponding reference samples. The number of training and test samples is displayed in Table II.

Fig 5: Eppalock lake data set.

Iii-B General Information

To evaluate the performance of different change detection algorithms, we utilize the following evaluation criteria:

  • Overall accuracy (OA): This index shows the number of bi-temporal pixels that are classified correctly, divided by the number of test samples.

  • Kappa coefficient: This metric is a statistical measurement of agreement between the final change detection map and the ground truth map. It is the percentage agreement corrected by the level of agreement that could be expected due to change alone. In general, it is thought to be a more robust measure than a simple percent agreement computation, as takes into account the agreement occurring by chance.

To validate the effectiveness of the proposed ReCNN model, it is compared with the most widely used change detection methods. These methods are summarized as follows:

  • CVA [7], which is an effective unsupervised approach for multispectral image change detection tasks.

  • PCA [10], which is simple in computation and can be applied to real-time applications.

  • MAD [12], which is a classical image transformation-based unsupervised algorithm for bi-temporal multispectral image change detection.

  • Iteratively-reweighted multivariate alteration detection (IRMAD) [50], which is an extension to MAD by introducing an iterative scheme.

  • Decision tree (DT), which is a non-parametric supervised learning method used for classification and regression. Its goal is to create a model that predicts the value of a target variable by learning simple decision rules inferred from data features.

  • Support vector machine (SVM), which works by mapping data to a kernel-included high-dimensional feature space seeking an optimal decision hyperplane that can best separate data samples, when data points are not linearly separable. Here, we use an SVM with RBF kernel. The optimal hyperplane parameters (parameter that controls the amount of penalty during the SVM optimization) and (spread of the RBF kernel) have been traced in the range of and using five-fold cross validation.

  • RNN [23], which has recently shown promising performance in classification and change detection.

  • ReCNN-FC, which uses fully connected RNN as recurrent sub-network in ReCNN model.

  • ReCNN-GRU, which uses GRU architecture in the recurrent sub-network.

  • ReCNN-LSTM, which is the ReCNN model with LSTM as recurrent component.

Among these methods, CVA, PCA, MAD, IRMAD, and RNN are used in binary change detection experiments, and DT, SVM, and RNN are compared to the proposed network in multi-class change detection experiments. Moreover, k-means algorithm is used to automatically select threshold for unsupervised methods in the binary change detection task.

Class Name Training Test
Binary Changed region 500 3380
Unchanged region 500 4515
TOTAL 1000 7895
Multiple Unchanged region 300 4715
Water loss 300 2817
Soil change 300 341
City expansion 50 72
TOTAL 950 7945
TABLE II: Number of Training and Test Samples in the Eppalock Lake Data Set
Fig 6: Comparisons of different RNN architectures in terms of model size. Here, 128 recurrent units are used in each architecture.
Taizhou City Eppalock Lake
OA Kappa Unchanged Changed OA Kappa Unchanged Changed
CVA [7] 83.82 0.3202 97.38 27.10 81.28 0.6353 69.24 97.37
PCA [10] 94.63 0.8181 99.79 74.51 74.68 0.5044 64.98 87.63
MAD [12] 94.62 0.8168 98.47 78.52 91.10 0.8138 99.14 80.36
IRMAD [50] 95.14 0.8313 99.35 77.53 91.27 0.8174 99.49 80.30
RNN [23] 96.50 0.8884 97.58 91.96 95.21 0.9018 97.03 92.78
ReCNN-FC 98.35 0.9470 98.94 95.86 98.40 0.9674 98.56 98.20
ReCNN-GRU 98.67 0.9571 99.23 96.30 98.64 0.9723 99.22 97.87
ReCNN-LSTM 98.73 0.9592 99.20 96.77 98.67 0.9728 98.83 98.46
TABLE III: Accuracy Comparison of Binary Change Detection on the Two Experimental Data Sets.
OA Kappa Unchanged City expansion Soil change Water change/loss
Taizhou City Decision Tree 85.19 0.5846 84.64 88.49 82.69 86.67
SVM 93.90 0.7927 94.69 89.32 92.31 93.33
RNN [23] 95.48 0.8374 97.04 86.92 85.58 85.33
ReCNN-FC 97.37 0.9039 97.95 94.12 95.19 92.00
ReCNN-GRU 97.52 0.9097 98.05 94.54 95.19 96.00
ReCNN-LSTM 98.04 0.9279 98.36 96.31 94.23 97.33
Eppalock Lake Decision Tree 87.56 0.7811 81.31 41.67 89.15 99.01
SVM 95.86 0.9228 94.46 72.22 97.65 98.58
RNN [23] 96.34 0.9392 95.55 41.67 96.48 99.04
ReCNN-FC 98.45 0.9705 98.01 80.56 100 99.47
ReCNN-GRU 98.49 0.9712 98.24 79.17 100 99.22
ReCNN-LSTM 98.70 0.9752 98.49 84.72 100 99.25
TABLE IV: Accuracy Comparison of Multi-Class Change Detection on the Two Experimental Data Sets.

Iii-C Analysis of Recurrent Sub-network: Comparisons between Fully Connected RNN, LSTM, and GRU

The most prominent trait shared between fully connected RNN, LSTM, and GRU is that there exists an additive loop of their update from to , which is lacking in the conventional feedforward neural networks such as CNNs. In contrast, compared to the fully connected RNN like Eq. (4), both LSTM and GRU keep the current content and add the new content on top of it (cf. Eq. (9) and Eq. (13)). These two RNN architectures, however, have a number of differences as well. LSTM makes use of three gates and a cell, namely, an input gate, forget gate, output gate, and memory cell, to control the exposure of memory content; whereas GRU only utilizes two gates to control the information flow. Therefore, the total number of parameters in GRU is reduced by about 25% compared to that in LSTM. Fig. 6 shows the number of total trainable parameters in different RNN architectures.

Table III and Table IV list binary and multi-class change detection results obtained in our experiments, respectively. For both data sets, ReCNN-LSTM outperforms ReCNN-FC and ReCNN-GRU on all indexes (i.e., OA and Kappa coefficient). For example, in the binary change detection, ReCNN-LSTM increases the accuracy by 0.38% of OA and 0.0122 of Kappa on the Taizhou data set, in comparison with ReCNN-FC; by 0.06% of OA and 0.0021 of Kappa on the same data set, compared to ReCNN-GRU. However, we can see that on these data sets, all three variations of the proposed ReCNN perform closely to each other. On the other hand, the proposed networks with gating RNN architectures as the recurrent sub-network (ReCNN-LSTM and ReCNN-GRU) slightly outperforms the more traditional ReCNN-FC on both of data sets and change detection tasks.

Fig 7: Change detection maps generated by the proposed ReCNN-LSTM model.
Fig 8: Comparison between spectral-temporal model (RNN) and spectral-spatial-temporal method (ReCNN-LSTM) on a region of the Taizhou city. From left to right and top to bottom: image, image, ground truth, change detection map obtained from RNN, and change detection map produced by ReCNN-LSTM. It can be clearly seen that there are a number of noisy scatter points of wrong detection (see ellipses in the lower left image) in the change detection map of RNN. While our spectral-spatial-temporal model RCNN-LSTM addresses this problem by eliminating those points.

Iii-D Analysis of Spatial Component: RNN vs ReCNN-LSTM

In the case of spectral-spatial-temporal change detection, the proposed recurrent convolutional network is able to significantly improve the spectral-temporal-based RNN model. As shown in Table III, compared to RNN, ReCNN-LSTM increases the accuracy of binary change detection considerably by 2.23% of OA and 0.0708 of Kappa coefficient, respectively, on the Taizhou data set. For the Eppalock lake scene, the accuracy increments on OA and Kappa coefficient are 3.46% and 0.071, respectively. Table IV compares the performance of RNN and ReCNN-LSTM in terms of multi-class change detection task. The latter can improve the former by 2.56% of OA and 0.0905 of Kappa coefficient, respectively, on the Taizhou scene; by 2.36% of OA and 0.036 of Kappa, respectively, on the Eppalock lake data. These results reveal the fact that the usage of spatial cue in our model can construct a more powerful spectral-spatial-temporal change detector.

Furthermore, as shown in Fig. 8, it is obvious that the spectral-temporal change detection method (RNN) always results in noisy scatter points in the change detection map. However, our spectral-spatial-temporal model ReCNN-LSTM addresses this problem by eliminating noisy scattered points of wrong detection.

Iii-E Comparison with Other Approaches

The OAs and Kappa coefficients of all competitors and the proposed networks on binary change detection task can be found in Table III. The classical change detection algorithms, CVA, PCA, MAD, and IRMAD, all achieve a good performance, especially IRMAD, which has the best performance among them. Compared to IRMAD, improvements in OA and Kappa coefficient achieved by ReCNN-LSTM are 3.59% and 0.1279, respectively, on the Taizhou data set, and increments of OA and Kappa obtained by ReCNN-LSTM on the Eppalock lake scene are 7.4% and 0.1554, respectively. However, the cost of such accuracy improvements is that we have to manually label some training data for supervised learning.

Table IV presents accuracy indexes on multi-class change detection task. Analysis of the detection accuracies indicates that SVM with RBF kernel outperforms DT, mainly because the kernel SVM generally handles nonlinear inputs more efficiently than DT. It can be seen that the proposed recurrent convolutional network ReCNN-LSTM outperforms SVM and RNN in terms of OA and Kappa coefficient on both the Taizhou and Eppalock lake data. Compared to SVM and RNN, ReCNN-LSTM increases OA by 4.14% and 2.56%, respectively, on the Taizhou data set; by 2.84% and 2.36%, respectively, on the Eppalock lake data.

Fig. 7 shows change detection results of the Taizhou city and Eppalock lake obtained by our model.

Iv Conclusion

In this paper, we have proposed a novel neural network architecture, called recurrent convolutional neural network (ReCNN), which integrates merits of both convolutional neural network (CNN) and recurrent neural network (RNN). ReCNN is capable of extracting joint spectral-spatial-temporal features from bi-temporal multispectral images and predicts change types. Moreover, it is end-to-end trainable. All these properties make ReCNN an excellent approach for multitemporal remote sensing data analysis.

The experiments on real multispectral images demonstrate that ReCNN achieves competitive performance, compared with conventional change detection models as well as spectral-temporal-based RNN algorithm. This confirms advantages of the proposed recurrent convolutional network. In addition, ReCNN is a general framework; therefore, it can be applied to other domains and problems (such as multitemporal hyper/multi-spectral data classification) that involve sequence prediction in remote sensing sequence data.

Future works will focus on new architectures based on ReCNN, for example, a semi-supervised ReCNN that can also use arbitrary amounts of unlabeled data for training – typically a small amount of labeled data with a large amount of unlabeled data.


The authors would like to express their appreciation to Dr. Chen Wu for providing the Taizhou data set.


  • [1] F. Bovolo and L. Bruzzone, “The time variable in data fusion: A change detection perspective,” IEEE Geoscience and Remote Sensing Magazine, vol. 3, no. 3, pp. 8–26, 2015.
  • [2] N. Yokoya, X. X. Zhu, and A. Plaza, “Multisensor coupled spectral unmixing for time-series analysis,” IEEE Transactions on Geoscience and Remote Sensing, vol. 55, no. 5, pp. 2842–2857, 2017.
  • [3] J. Yang, P. J. Weisberg, and N. A. Bristow, “Landsat remote sensing approaches for monitoring long-term tree cover dynamics in semi-arid woodlands: Comparison of vegetation indices and spectral mixture analysis,” Remote Sensing of Environment, vol. 119, pp. 62–71, 2012.
  • [4] G. Xian and C. Homer, “Updating the 2001 national land cover database impervious surface products to 2006 using Landsat imagery change detection methods,” Remote Sensing of Environment, vol. 114, no. 8, pp. 1676–1686, 2010.
  • [5] B. Liang and Q. Weng, “Assessing urban environmental quality change of Indianapolis, United States, by the remote sensing and gis integration,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 4, no. 1, pp. 43–55, 2011.
  • [6] W. A. Malila, “Change vector analysis: An approach for detecting forest changes with Landsat,” in Machine Processing of Remotely Sensed Data Symposium, 1980.
  • [7] F. Bovolo and L. Bruzzone, “A theoretical framework for unsupervised change detection based on change vector analysis in the polar domain,” IEEE Transactions on Geoscience and Remote Sensing, vol. 45, no. 1, pp. 218–236, 2007.
  • [8] F. Bovolo, S. Marchesi, and L. Bruzzone, “A framework for automatic and unsupervised detection of multiple changes in multitemporal images,” IEEE Transactions on Geoscience and Remote Sensing, vol. 50, no. 6, pp. 2196–2212, 2012.
  • [9]

    M. Zanetti, F. Bovolo, and L. Bruzzone, “Rayleigh-Rice mixture parameter estimation via em algorithm for change detection in multispectral images,”

    IEEE Transactions on Image Processing, vol. 24, no. 12, pp. 5004–5016, 2015.
  • [10] J. S. Deng, K. Wang, Y. H. Deng, and G. J. Qi, “PCA-based land-use change detection and analysis using multitemporal and multisensor satellite data,” International Journal of Remote Sensing, vol. 29, no. 16, pp. 4823–4838, 2008.
  • [11] J. B. Collins and C. E. Woodcock, “An assessment of several linear change detection techniques for mapping forest mortality using multitemporal Landsat TM data,” Remote Sensing of Environment, vol. 56, no. 1, pp. 66–77, 1996.
  • [12] A. A. Nielsen, K. Conradsen, and J. J. Simpson, “Multivariate alteration detection (MAD) and MAF postprocessing in multispectral, bitemporal image data: New approaches to change detection studies,” Remote Sensing of Environment, vol. 64, no. 1, pp. 1–19, 1998.
  • [13] C. Wu, B. Du, and L. Zhang, “Slow feature analysis for change detection in multispectral imagery,” IEEE Transactions on Geoscience and Remote Sensing, vol. 52, no. 5, pp. 2858–2874, 2014.
  • [14] A. Ertürk, M.-D. Iordache, and A. Plaza, “Sparse unmixing-based change detection for multitemporal hyperspectral images,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 9, no. 2, pp. 708–719, 2016.
  • [15] M. Gong, T. Zhan, P. Zhang, and Q. Miao, “Superpixel-based difference representation learning for change detection in multispectral remote sensing images,” IEEE Transactions on Geoscience and Remote Sensing, vol. 55, no. 5, pp. 2658–2673, 2017.
  • [16] X. Li and A. G. O. Yeh, “Principal component analysis of stacked multitemporal images for the monitoring of rapid urban expansion in the Pearl River Delta,” International Journal of Remote Sensing, vol. 19, no. 8, pp. 1501–1518, 1998.
  • [17] L. Bruzzone and S. B. Serpico, “An iterative technique for the detection of land-cover transitions in multitemporal remote-sensing images,” IEEE Transactions on Geoscience and Remote Sensing, vol. 35, no. 4, pp. 858–867, 1997.
  • [18] L. Bruzzone, D. F. Prieto, and S. B. Serpico, “A neural-statistical approach to multitemporal and multisource remote-sensing image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 37, no. 3, pp. 1350–1359, 1999.
  • [19] L. Bruzzone and R. Cossu, “A multiple cascade-classifier system for a robust a partially unsupervised updating of land-cover maps,” IEEE Transactions on Geoscience and Remote Sensing, vol. 40, no. 9, pp. 1984–1996, 2002.
  • [20] L. Bruzzone, R. Cossu, and G. Vernazza, “Detection of land-cover transitions by combining multidate classifiers,” Pattern Recogniton Letters, vol. 25, no. 13, pp. 1491–1500, 2004.
  • [21] X. X. Zhu, D. Tuia, L. Mou, G.-S. Xia, L. Zhang, F. Xu, and F. Fraundorfer, “Deep learning in remote sensing: A comprehensive review and list of resources,” IEEE Geoscience and Remote Sensing Magazine, vol. 5, no. 4, pp. 8–36, 2017.
  • [22] ——, “Deep learning in remote sensing: A review,” arXiv:1710.03959, 2017.
  • [23] H. Lyu, H. Lu, and L. Mou, “Learning a transferable change rule from a recurrent neural network for land cover change detection,” Remote Sensing, vol. 8, no. 6, p. 506, 2016.
  • [24] M. Russwurm and M. Körner, “Temporal vegetation modelling using long short-term memory networks for crop identification from medium-resolution multi-spectral satellite images,” in

    IEEE International Conference on Computer Vision and Pattern Recognition (CVPR) workshop

    , 2017.
  • [25] L. Mou and X. X. Zhu, “IM2HEIGHT: Height estimation from single monocular imagery via fully residual convolutional-deconvolutional network,” arXiv:1802.10249, 2018.
  • [26]

    J. Hu, L. Mou, A. Schmitt, and X. X. Zhu, “FusioNet: A two-stream convolutional neural network for urban scene classification using PolSAR and hyperspectral data,” in

    Joint Urban Remote Sensing Event (JURSE), 2017.
  • [27] L. Mou, M. Schmitt, Y. Wang, and X. X. Zhu, “A CNN for the identification of corresponding patches in SAR and optical imagery of urban scenes,” in Joint Urban Remote Sensing Event (JURSE), 2017.
  • [28] M. Volpi and D. Tuia, “Dense semantic labeling of subdecimeter resolution images with convolutional neural networks,” IEEE Transactions on Geoscience and Remote Sensing, vol. 55, no. 2, pp. 881–893, 2017.
  • [29] L. Mou, P. Ghamisi, and X. Zhu, “Unsupervised spectral-spatial feature learning via deep residual conv-deconv network for hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 56, no. 1, pp. 391–406, 2018.
  • [30] E. Maggiori, Y. Tarabalka, G. Charpiat, and P. Alliez, “Convolutional neural networks for large-scale remote-sesning image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 55, no. 2, pp. 645–657, 2017.
  • [31] L. Mou and X. X. Zhu, “Spatiotemporal scene interpretation of space videos via deep neural network and tracklet analysis,” in IEEE International Geoscience and Remote Sensing Symposium (IGARSS), 2016.
  • [32] L. Hughes, M. Schmitt, L. Mou, Y. Wang, and X. X. Zhu, “Identifying corresponding patches in SAR and optical images with a pseudo-Siamese CNN,” arXiv:1801.08467, 2018.
  • [33] L. Mou, X. X. Zhu, M. Vakalopoulou, K. Karantzalos, N. Paragios, B. L. Saux, G. Moser, and D. Tuia, “Multitemporal very high resolution from space: Outcome of the 2016 IEEE GRSS Data Fusion Contest,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 10, no. 8, pp. 3435–3447, 2017.
  • [34] L. Mou, P. Ghamisi, and X. Zhu, “Deep recurrent neural networks for hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 55, no. 7, pp. 3639–3655, 2017.
  • [35] H. Wu and S. Prasad, “Convolutional recurrent neural networks for hyperspectral data classification,” Remote Sensing, vol. 9, no. 3, p. 298, 2017.
  • [36] Y. Li, H. Zhang, and Q. Shen, “Spectral-spatial classification of hyperspectral imagery with 3D convolutional neural network,” Remote Sensing, vol. 18, no. 7, pp. 1527–1554, 2016.
  • [37] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems (NIPS), 2012.
  • [38] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in IEEE International Conference on Learning Representation (ICLR), 2015.
  • [39] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  • [40] L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs,” arXiv:1606.00915, 2016.
  • [41] M. Holschneider, R. Kronland-Martinet, J. Morlet, and P. Tchamitchian, “A real-time algorithm for signal analysis with the help of the wavelet transform,” in Wavelets: Time-Frequency Methods and Phase Space, 1989.
  • [42] C. Peng, X. Zhang, G. Yu, G. Luo, and J. Sun, “Large kernel matters – improve semantic segmentation by global convolutional network,” in IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  • [43] S. Hochreiter and J. Schmidhuber, “Long short-term memory,,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997.
  • [44] A. Graves, “Generating sequences with recurrent neural networks,,” arXiv:1308.0850, 2013.
  • [45]

    T. Dozat, “Incorporating Nesterov momentum into Adam,”

    http://cs229.stanford.edu/proj2015/054_report.pdf, online.
  • [46] I. Sutskever, J. Martens, G. Dahl, and G. Hinton, “On the importance of initialization and momentum in deep learning,” in

    IEEE International Conference on Machine Learning (ICML)

    , 2013.
  • [47]

    Y. LeCun, B. Boser, J. Denker, D. Henderson, R. Howard, W. Hubbard, and L. Jackel, “Backpropagation applied to handwritten zip code recognition,”

    Neural Computation, vol. 1, no. 4, pp. 541–551, 1989.
  • [48] D. P. Kingma and J. L. Ba, “Adam: A method for stochastic optimization,” in IEEE International Conference on Learning Representations (ICLR), 2015.
  • [49] X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in

    International Conference on Artificial Intelligence and Statistics (AISTATS)

    , 2010.
  • [50] A. A. Nielsen, “The regularized iteratively reweighted MAD method for change detection in multi- and hyperspectral data,” IEEE Transactions on Image Processing, vol. 16, no. 2, pp. 463–478, 2007.