Novel view synthesis is a long-standing problem in computer vision. Earlier works in this area synthesize views of single objects or natural scenes with small variation in viewing angle. For generating views of single objects in a uniform background or scenes, the task regards learning a mapping or transformation across views. With a small camera movement, there is a high degree of overlap in field of views resulting in images with high content overlap. The generative models can learn to copy large parts of the image content from the input to the output and perform the synthesis task satisfactorily. Despite this, view synthesis task is very challenging due to presence of multiple objects in the scene. The network needs to learn the object relations and occlusions in the scene.
Generating cross-view natural scenes conditioned on images from drastically different views (e.g., generating top-view from street view scene) is very painstaking. This is mainly because there is very little overlap between the corresponding field of views. Thus, simply copying and pasting pixels from one view to another would not be a solution. Rather, it is needed to learn the object classes present in the input view and to understand the correspondences in the target view with appropriate object relations and transformations (i.e., geometric reasoning).
In this work, we address the problem of synthesizing ground-level images from overhead imagery and vice versa using conditional Generative Adversarial Networks [Mirza and Osindero (2014)]. Also, when possible, we guide the generative networks by feeding homography transformed images as inputs to improve the synthesized results. Basically, conditional GANs try to generate new images from conditioning variables as input. The conditioning variables could be other images, text descriptions, class labels, etc.
Our first approach exploits the success of the first GAN-based image-to-image translation network put forward byIsola et al. (2017) as a general purpose architecture on multiple image translation tasks. This work translates images of objects or scenes which are represented by RGB images, gradient fields, edge maps, aerial images, sketches, etc across these representations. Thus, it essentially operates on different representations of images in a single view. We use this architecture as a starting point (base model) in our task and obtain encouraging results. The limitation of this approach to our problem, however, is that the images to be transformed to are very different as they come from two drastically different views, have small fields of view overlap, and objects in the images might be occluded. As a result, learning to map the pixels between the views is difficult as the corresponding pixels in two views may represent different object classes. To address this challenge, here we propose to use the semantic segmentation maps of target view images to regularize the training process. This helps the network to learn the semantic class of the pixels in the target view and to guide the network to generate the target pixels. By encouraging the networks to generate the segmentation maps in target view, the network learns the semantic classes of each pixel which are important cues in assisting to generate cross-view images that preserve semantic information from the source view to the target view.
The next approach we take to solve the cross-view image synthesis task is to exploit the geometric relation between the views to guide the synthesis. For this, we first compute the homography transformation matrix between the views and then project the aerial images to street-view perspective. By doing so, we obtain an intermediate image that looks very close to the target view image but not as realistic and with some missing regions. Now, our problem reduces to preserving the scene layout and details while filling in the missing regions and adding realism to the transformed image. For this, we use cGAN architectures described in previous approach. We also use different cGANs that work specifically for inpainting and realism tasks to preserve the pixel information from the homography transformed image in a controlled manner.
To summarize, we propose the following methods. We start with the simple image-to-image translation network of Isola et al. (2017) as a baseline (here called Pix2pix). We then propose two new cGAN architectures that generate images as well as segmentation maps in the target view. Augmenting semantic segmentation generation to the architectures helps improve the quality of generated images. The first architecture, called X-Fork, is a slight modification of the baseline, forking at the penultimate block to generate two outputs, target view image and segmentation map. The second architecture, called X-Seq, has a sequence of two baseline networks connected. The target view image generated by the first network is fed to the second to generate its corresponding segmentation map. Once trained, both architectures are able to generate better images than the baseline that generates only the target view images. This implies that learning to generate segmentation map along with the image indeed improves the quality of generated images. We also use homography to transform the aerial image to ground view and feed the transformed image to these networks to further improve the results. Finally, we propose a method to preserve details from the homography transformed image in a controlled setting to generate the street view images. It constitutes two subtasks: a) generating missing regions by inpainting, and b) adding realism by using GAN to preserves details visible in aerial view into the street view image. We call this approach H-Regions. Throughout the paper, H in a method’s name indicates that the input is the homography transformed image.
2 Related Work
2.1 Relating Aerial and Ground-level Images
Zhai et al. (2017) explored the relationship between the cross-view images by learning to predict the semantic layout of the ground image from its corresponding aerial image. They used the predicted layout to synthesize ground-level panorama. Prior works relating the aerial and ground imageries have addressed problems such as cross-view co-localization [Lin et al. (2013); Vo and Hays (2016)], ground-to-aerial geo-localization [Lin et al. (2015); MH and Lee (2018)] and geo-tagging the cross-view images [Workman et al. (2015)]. Recently, the images generated by our cross-view image synthesis approach have been successfully used to bridge the domain gap between aerial and street-view images in geo-localization tasks [Regmi and Shah (2019)].
Cross-view relations have also been studied between egocentric (first person) and exocentric (surveillance or third-person) domains for different purposes. Human re-identification by matching viewers in top-view and egocentric cameras have been tackled by establishing the correspondences between the views in Ardeshir and Borji (2016). Soran et al. (2014) utilize the information from one egocentric camera and multiple exocentric cameras to solve the action recognition task.
2.2 Learning View Transformations
Existing works on viewpoint transformation have been conducted to synthesize novel views of the same objects [Dosovitskiy et al. (2017); Tatarchenko et al. (2016); Zhou et al. (2016)]. Zhou et al. (2016) proposed models that learn to copy the pixel information from the input view and utilize them to preserve the identity and structure of the objects to generate new views. Tatarchenko et al. (2016) trained an encode-decoder network to obtain 3D models of cars and chairs which they later used to generate different views of an unseen car or chair. Dosovitskiy et al. (2017)
learned generative models by training on 3D renderings of cars, chairs and tables and synthesized intermediate views and objects by interpolating between views and models.
2.3 GAN and cGAN
Goodfellow et al. (2014)
are the pioneers of Generative Adversarial Networks that are very successful at generating sharp and unblurred images, much better compared to existing methods such as Restricted Boltzmann Machines [Hinton et al. (2006); Smolensky (1986)] or deep Boltzmann Machines [Salakhutdinov and Hinton (2009)].
Conditional GANs synthesize images conditioned on different parameters during both training and testing. Examples include conditioning on labels of MNIST to generate digits by Mirza and Osindero (2014), on image representations to translate an image between different representations by Isola et al. (2017), and generating panoramic ground-level scenes from aerial images of the same location by Zhai et al. (2017). Reed et al. (2016) synthesize images conditioned on detailed textual descriptions of the objects in the scene, and Zhang et al. (2017) improved on that by using a two-stage Stacked GAN.
2.4 Cross-Domain Transformations using GANs
Kim et al. (2017) utilized the GAN networks to learn the relation between images in two different domains such that these learned relations can be transferred between the domains. Similar work by Zhu et al. (2017) learned mappings between unpaired images using cycle-consistency loss. They assume that a mapping from one domain to the other and back to the first should generate the original image. Both works exploited large unpaired datasets to learn the relation between domains and formulated the mapping task between images in different domains as a generation problem. Zhu et al. (2017) compare their generation task with previous works on paired datasets by Isola et al. (2017). They conclude that the result with paired images is the upper-bound for their unpaired examples.
2.5 Geometry-guided Synthesis
Song et al. (2017) propose geometry-guided adversarial networks to synthesize identity-preserving facial expressions. The facial geometry is used as a controlled input to guide the network to synthesize facial images with desired expressions. Similar work by Kossaifi et al. (2017) improves the visual quality of synthesized images by enforcing a mechanism to control the shapes of the objects. They map the generator’s output to a mean shape and implicitly enforce the geometry of the objects and also add skip connections to transfer priors to the generated objects.
2.6 Image Inpainting
tackle the problem of image inpainting by searching for the encoding of the corrupted image that is closest to another image in the latent space and passing it through the generator to reconstruct the image. The closeness is defined based on the weighted context loss of the corrupted image, and a prior loss that penalizes unrealistic images.Yang et al. (2017) propose a multi-scale patch synthesis approach for high-resolution image inpainting by jointly optimizing on image content and texture constraint.
3 Background on GANs
Th Generative Adversarial Network (GAN) proposed by Goodfellow et al. (2014)
consists of two adversarial networks: a generator and a discriminator that are trained simultaneously based on the min-max game theory. The generatoris optimized to map a
-dimensional noise vector (usually=100) to an image (i.e., synthesizing) that is close to the true data distribution. The discriminator , on the other hand, is optimized to accurately distinguish between the synthesized images coming from the generator and the real images coming from the true data distribution. The objective function of such a network is
where is real data sampled from data distribution and is a
-dimensional noise vector sampled from a Gaussian distribution.
Conditional GANs synthesize images looking into some auxiliary variable which may be labels [Mirza and Osindero (2014)], text embeddings [Zhang et al. (2017); Reed et al. (2016)] or images [Isola et al. (2017); Zhu et al. (2017); Kim et al. (2017)]. In conditional GANs, both the discriminator and the generator networks receive the conditioning variable represented by in Eqn. (2). The generator uses this additional information during image synthesis while the discriminator makes its decision by looking at the pair of conditioning variable and the image it receives. Real pair input to the discriminator consists of true image from distribution and its corresponding label while the fake pair consists of the synthesized image and the label. For conditional GAN, the objective function is
where = is the generated image.
In addition to the GAN loss, previous works [e.g., Isola et al. (2017); Zhu et al. (2017); Pathak et al. (2016)] have tried to minimize the or distances between real and generated image pairs. This step aids the generator to synthesize images very similar to the ground truth. Minimizing the distance generates less blurred images than minimizing the distance. That is, using the distance increases image sharpness in generation tasks. Therefore, we use the distance in our method. The expression to minimize the distance is
Considering the synthesis of the ground level imagery from aerial image , the conditional GAN loss and loss are represented as in Eqns. (4) and (5), respectively. For ground to aerial synthesis, the roles of and are reversed.
where, and are the balancing factors between the losses.
In this section, we discuss the baseline methods and the proposed architectures for the task of cross-view image synthesis.
The naive way to approach this task is to consider it as an image to image translation problem. We run the experiments in the following settings as our baselines.
4.1.1 Cross-view Image-to-Image Translation (X-Pix2pix)
For this, the generator is an encoder-decoder network that takes in an image in first view as input and learns to generate the image in the other view as output.
4.1.2 Cross-view Image-to-Image Translation with Stacked Outputs (X-SO)
The network takes an image in first view as input and generates a single output of 6 channels, the first 3 channels correspond to the RGB image and the next three channels represent the segmentation map. The L1 loss and adversarial loss are computed over six channels of output and the corresponding ground truth images.
4.2 Proposed Methods
4.2.1 Crossview Fork (X-Fork)
Our first architecture, known as the Crossview Fork, is shown in Figure 0(a). The generator network is forked to synthesize two outputs of 3 channels each, the first output is the RGB image and the second output is the segmentation map both in target view. The fork-generator architecture is shown in Figure 2. The first six blocks of decoder share the weights. This is because the image and segmentation map contain a lot of shared features. The number of kernels used in each layer (block) of the generator are shown below the blocks.
The inherent idea behind this architecture is multi-task learning by the generator network. When the generator is enforced to learn the semantic class of the pixels together with the image synthesis, this helps to improve the image synthesis task. The generated segmentation map serves as an auxiliary output. The objective function for this network is shown in Eqn. 7.
Note that the L loss is minimized for images as well as segmentation maps whereas the adversarial loss is optimized for images only. This is because we only care about pixel accuracy for segmentation maps.
4.2.2 Crossview Sequential (X-Seq)
Our second architecture uses a sequence of two cGAN networks as shown in Figure 0(b). The first network performs cross-view image synthesis and the generated image is fed to the second network as a conditioning input to synthesize its corresponding segmentation map. This two-stage end-to-end learning of image and segmentation map synthesis produces improved image quality compared to the network without second stage. The joint objective function for X-Seq architecture is shown in Eqn. 8 below.
4.2.3 Cross-view Pix2pix with Homography (H-Pix2pix)
Another approach that we take here is to feed the homography transformed image as an input to the translation network. Our hypothesis is that the majority of the scene from the first view is transformed into the second perspective using the homography and this should ease the synthesis task. The large missing regions in the transformed images are mostly related to sky and buildings.
4.2.4 Cross-view Stacked Outputs with Homography (H-SO)
The network takes the homography transformed image as input and generates target view image and the segmentation map stacked together as a 6-channel output; first 3 channels for image and the next three channels for its segmentation map.
4.2.5 Cross-view Fork with Homography (H-Fork)
Here, we use the homography transformed image I as input to the network architecture proposed in subsection 4.2.1. The hypothesis behind this idea is that the use of transformed images as input should ease the cross-view synthesis task compared to synthesizing by feeding the aerial images I directly.
4.2.6 Crossview Sequence with Homography (H-Seq)
In this setup, we feed the homography transformed image I rather than the original input image I as input to the X-Seq network of subsection 4.2.2.
4.2.7 Crossview Regions with Homography (H-Regions)
In this method, we attempt to preserve the structural details visible in the aerial view images and guide the network to transfer those details to the synthesized ground view images. For this, we use the homography transformed image as input to our method and solve the synthesis task in following subtasks:
Subtask I: The homography transformed image (I in Figure 3) has a large portion of missing region (R) in the image. Our first task is to fill in the missing region in the transformed images. We use an encoder-decoder network that takes I as input and generates only the upper half of the image (I).
Subtask II: The street-view images are recorded by a camera mounted on a car’s dashboard. Therefore, all the street-view images contain a part of the car’s hood around the lower central region (R) in them which can be seen in image I in Figure 3. Note that, this scenario is present in SVA dataset and can be avoided in real applications where the dash camera can be mounted such that the hood of the car is not captured in the frames. The homography transformed image I
also has a car around that region which has been transformed from the aerial view of the car but it does not realistically represent a car in street-view. To address this, we mask a probable car-region and train a small network dedicated to learn mapping of the car region. This helps generate a realistic car region in ground view images (See region Rin image I).
Once we have images generated from the first two tasks, we copy them to their respective spatial locations in homography transformed image I creating a street-view image I and preserving the pixels of remaining regions. Copying pixels in this manner helps us preserve the structural information that has been transformed using homography. The problem with this approach is that images do not look realistic at region boundaries. So, we further train another network to add realism to this image.
Subtask III: GANs are successful at generating images that look very realistic to human eyes. Here, we train a conditional GAN architecture on I. For this, we first define bands around the region boundaries as shown in Figure 3
. We formulate the loss function to preserve (by copying) the pixel information outside the bands to the output image while at the same time adding realism to the whole image. This step helps a lot to improve the visual quality of the synthesized image.
We now define our loss functions for the subtasks. For subtask one, we use conditional GAN network to inpaint missing regions in I by optimizing the network on adversarial and L losses for the missing regions only. For subtask two, we only consider region R by masking out the remaining regions in input and output images and optimizing for adversarial and L losses for car region only. Once we have results from the above two subtasks, we compute I as shown in Eqn. 9.
Here, I is the image generated from the inpainting network of subtask one. I is the car image generated in subtask two. M and M are 3-channel binary masks for regions R and R, M being the 3-channel all ones image. The masks M and M are manually computed looking at the homography transformed image (I) in Figure 3. This was done for a single frame only and worked well for all the images in the dataset. If the hood of the car was not visible in the street-view image, we wouldn’t even need region R2 and correspondingly mask M. is fed to the realism network to generate the final image in the target view. is the element-wise product.
5 Experimental Setting
For the experiments in this work, we use three datasets described in the following.
Dayton. This cross-view image dataset is provided by Vo and Hays (2016). It consists of more than 1M pairs of street-view and overhead view images collected from 11 different US cities. We select 76,048 image pairs from Dayton images and create a train/test split of 55,000/21,048 pairs. We call it Dayton Dataset. The images in the original dataset have resolution of 354354. We resize them to 256256. We use this dataset for experiments in both aerial to ground a2g and ground to aerial g2a directions.
CVUSA. We recruit this dataset [Workman et al. (2015)] for direct comparison of our work with Zhai et al. (2017). It consists of 35,532/8,884 train/test split of image pairs. Following Zhai et al. (2017), the aerial images are center-cropped to 224 224 and then resized to 256 256. We only generate a single camera-angle image rather than the panorama. To do so, we take the first quarter of the ground level images as well as segmentations from the dataset and resize them to 256 256 in our experiments.
SVA. The Surround Vehicle Awareness (SVA) dataset [Palazzi et al. (2017)] is a synthetic dataset collected from Grand Theft Auto V (GTAV) video game. The game camera is toggled between frontal and bird’s eye view to simultaneously capture images in the two views at each game time step. We use the train/test split as provided in the dataset. The original dataset has 100 sets of training set images and 50 sets of test set images. The consecutive frames in each set are very similar to each other, so we use every tenth frame to remove redundancy in the dataset. Finally, we have a training set of 46,030 image pairs and a test set of 22,254 image pairs. The images are resized to 256 256 for experiments in this work. Sample images from SVA dataset are shown in the leftmost and rightmost columns of Figure 7. We use this dataset for experiments in aerial to ground a2g direction only. Note that, homography related experiments are performed on this dataset only.
The proposed Fork and Sequence networks learn to generate the target view images and segmentation maps conditioned on source view image or their homography transformed image. Training procedure requires the images as well as their semantic segmentation maps. The CVUSA dataset has annotated segmentation maps for ground view images, but for SVA and Dayton datasets such information is not available. To compensate, we use one of the leading semantic segmentation methods, known as the RefineNet [Lin et al. (2017)]. This network is pre-trained on outdoor scenes of the Cityscapes dataset [Cordts et al. (2016)] and is used to generate the segmentation maps that are utilized as ground truth maps. These semantic maps have pixel labels from 20 classes (e.g., road, sidewalk, building, vegetation, sky, void, etc). Figure 4 shows image pairs from the Dayton dataset and their segmentation masks overlaid in both views. As can be seen, the segmentation mask (label) generation process is not perfect since it is unable to segment parts of buildings, roads, cars, etcetera in images.
5.2 Implementation Details
We use homography as a preprocessing step to transform the visual features from aerial images to ground perspective. Since the locations of aerial and ground view camera are fixed in the SVA dataset, first we randomly pick a pair of images from the dataset. We then manually select four points in the aerial image and find their corresponding locations in the ground-view image and use them to compute the homography matrix that transforms the aerial image to ground view and vice versa. Surprisingly, this method works well and avoids expensive computations for homography estimation usually done for each pair of images separately by computing SIFT features of the images and then finding the keypoints in the images. We also tried computing the SIFT features and then finding keypoints in two images and transforming images based on those points. This method could not find the corresponding points in two views, most likely because of very large perspective variation between the views.
We use the conditional GAN architecture very similar to Isola et al. (2017) as the base architecture. The generator is an encoder-decoder network with blocks of Convolution, Batch Normalization [Ioffe and Szegedy (2015)
] and activation layers. Leaky ReLU with a slope of 0.2 is used as the activation function in the encoder, whereas the decoder has ReLU activation except for its final layer where Tanh is used. The first three blocks of the decoder have a Dropout layer in between Batch normalization and activation layer, with dropout rate of 50%. The discriminator is taken as it is from theIsola et al. (2017). For the experiments on the SVA dataset, we removed two blocks of CBL and UBDL from the generator architecture, primarily to save training time. We observed that removal of these blocks did not have much impact on the quality of synthesized images. Also, note that the synthesized semantic maps are 3-channel RGB images which effectively mitigated the class imbalance among the semantic classes. This was primarily done to consider all 20 semantic classes during the training and reduce bias towards dominant classes like houses, trees, road and sky. Had the semantic classes been limited to the dominant ones only, it would regularize the synthesized images to not learn the less prevalent objects in the target view images. Also, we brought in some confidence by the success of the pix2pix in synthesizing 3-channel segmentation maps from RGB images.
The convolutional kernels used in the networks are 42011)] implementation of , and upsamples the input by 2. The convolutional operation downsamples the images by 2. No pooling operation is used in the networks. The and used in the objective function for different networks are the balancing factors between the GAN loss and the loss. We fix at 1 and at 100 for Fork and Seq models. For realism task in the H-Regions method, =5 for adversarial loss and =2 for pixel-wise loss worked the best. Following the idea to smooth the labels by Szegedy et al. (2016) and demonstration of its effectiveness by Salimans et al. (2016)
, we use one-sided label smoothing to stabilize the training process, replacing 1 by 0.9 for real labels. During the training, we utilized different data augmentation methods such as random jitter and horizontal flipping of images. The network is trained end-to-end with weights initialized using a random Gaussian distribution with zero mean and 0.02 standard deviation. Our methods are implemented in Torch [Collobert et al. (2011)]. Our code and data is available online at: https://github.com/kregmi/cross-view-image-synthesis.git.
We train the networks for 100 epochs on low resolution and 35 epochs on high resolution images of Dayton dataset and for 20 epochs on SVA dataset. Experiments are conducted on CVUSA dataset for comparison with the work ofZhai et al. (2017)
. Following their setup, we train our architectures for 30 epochs using the Adam optimizer and moment parameters= 0.5 and = 0.999. We observed that the training for X-SO method on low resolution of Dayton dataset and CVUSA dataset was very unstable; so the qualitative and quantitative results are not very good for them. For H-Regions, we conduct experiments as follows. For subtask I, we train the network for 20 epochs. For subtask II, we train another network for one epoch only since the network needs to learn the mapping of the car and to preserve its color from source view to the target view. Eventually, we train a final realism network for 5 epochs (subtask III).
6 Evaluation and Model Comparison
We have conducted experiments in a2g (aerial-to-ground) and g2a (ground-to-aerial) directions on the Dayton dataset and a2g direction only on CVUSA and SVA datasets. We consider image resolutions of 6464 and 256256 on the Dayton dataset while for experiments on CVUSA and SVA datasets, 256256 resolution images are used.
6.1 Evaluation measures
It is not straightforward to evaluate the quality of synthesized images [Borji (2018)]. In fact, evaluation of GAN methods continues to be an open problem [Theis et al. (2016)]. Here we utilize four quantitative measures and one qualitative measure to evaluate our methods.
6.1.1 Qualitative measure
For the subjective evaluation of different methods, we run a user study on the images synthesized using these methods. We show an aerial image along with the corresponding images in ground view synthesized using seven different methods to 10 users. We specifically ask each user to select the most realistic image that also contains the most visual details from the aerial view image.
6.1.2 Quantitative measures
Inception Score: A common quantitative measure used in GAN evaluation is the Inception Score proposed by Salimans et al. (2016)
. The core idea behind the inception score is to assess how diverse the generated samples are within a class while being meaningfully representative of the class at the same time. One major criticism regarding the Inception score is that the CNN trained on ImageNet objects may not be adequate for other scene datasets (our case). To address this, here we use the AlexNet model [Krizhevsky et al. (2017)] trained on Places dataset [Zhou et al. (2017)] with 365 categories to compute the inception score. The Places dataset has images similar to those in our datasets.
We observe that the confidence scores predicted by the pre-trained model on our datasets are dispersed between classes for many samples and not all the categories are represented by the images. Therefore, we compute inception scores on Top-1 and Top-5 classes, where ”Top-k” means that top k predictions for each image are unchanged while the remaining predictions are smoothed by an epsilon equal to (1 -(top-k predictions))/(n-k classes).
Accuracy: In addition to inception score, we compute the top-k prediction accuracy between real and generated images. We use the same pre-trained Alexnet model to obtain annotations for real images and class predictions for generated images. We compute top-1 and top-5 accuracies. For each setting, accuracies are computed in two ways: 1) considering all images, and 2) considering real images whose top-1 (highest) prediction is greater than 0.5. Below each accuracy heading, the first column considers all images whereas the second column computes accuracies the second way.
KL(model data): We compute the KL divergence between the model generated images and the real data distribution for quantitative analysis of our work, similar to some generative works [Che et al. (2016); Nguyen et al. (2017)]. We again use the same pre-trained Alexnet described earlier. The lower KL score implies that the generated samples are closer to the real data distribution.
SSIM, PSNR and Sharpness Difference: We employ three measures from the image quality assessment literature to evaluate our models, similar to Mathieu et al. (2016); Ledig et al. (2017); Shi et al. (2016); Park et al. (2017). Structural-Similarity (SSIM) measures the similarity between the images based on their luminance, contrast and structural aspects. SSIM value ranges between -1 and +1. Peak Signal-to-Noise Ratio (PSNR) measures the peak signal-to-noise ratio between two images to assess the quality of a transformed (generated) image compared to its original version. Sharpness difference (SD) measures the loss of sharpness during image generation. For each of these score, the higher the the better. Please refer to Regmi and Borji (2018) for details on how to compute these scores.
FID Score: An alternative metric to evaluate the quality of generated images is by computing Frechet Inception Distance [Heusel et al. (2017)] between the generated samples and the real images. We use the same AlexNet model (as above) pretrained on the Places Dataset to compute the FID score. The lower value of FID score for a method, the better. The FID scores that we obtain in this work are relatively larger than numbers reported in other works mainly because of the variations in the image statistics of the Places Dataset used during the training and our test images.
6.2 Model Comparison
We conduct the qualitative and quantitative evaluation on 3 datasets. We report homography results on SVA dataset only. This is because the aerial view image for SVA dataset contains high overlap with the field of view of street-view image and thus application of homoraphy to preserve the details from aerial image seemed valid for this dataset compared to the other two.
We conduct the user studies to compare the images synthesized using different methods as well as illustrate qualitative results in Figures 5, 6 and 7 and conduct an in-depth quantitative evaluation on test images of the datasets.
6.2.1 Qualitative Comparison
Dayton dataset. For 6464 resolution experiments, the networks are modified by removing the last two blocks of CBL from discriminator and encoder, and the first two blocks of UBDR from decoder of the generator. We run experiments on all three methods. Qualitative results are depicted in Figure 5 (left). The results affirm that the networks have learned to transfer the image representations across the views. Generated ground level images clearly show details about road, trees, sky, clouds, and pedestrian lanes. Trees, grass, road, house roofs are well rendered in the synthesized aerial images.
For 256256 resolution synthesis, we conduct experiments on all three architectures and illustrate the qualitative results in Figure 5 (right). We observe that the images generated in high resolution contain more details of objects in both views and are less granulated than those in low resolution. Houses, trees, pedestrian lanes, and roads look more natural.
CVUSA dataset. Test results on the CVUSA dataset show that images generated by the proposed methods are visually better compared to Zhai et al. (2017) and Isola et al. (2017). Our proposed methods are more successful in transforming the pixel classes and thus generating correct class objects in target view compared to the baselines.
SVA dataset. The qualitative results on the SVA dataset for aerial to street-view synthesis is shown in Figure 7. The proposed methods are capable at generating roads, car-hood, markers on road, sky and other details in the images. We observe that the images generated by the proposed H-Regions method contain more details around the central regions. This is due to enforcing the network to preserve those details from the aerial images.
User Study. We conducted the perceptual test over 100 test images on 10 subjects to compare the images synthesized by different methods on the SVA dataset. The results are presented in Table 1. The most preferred method is H-Regions, closely contested by H-Seq and X-Seq methods. The results illustrate the following: a) The use of homography transformed input drastically outperforms corresponding experiments with untransformed aerial image as input, and b) Users preferred the images synthesized using H-Regions because of the method’s ability to preserve the pixel information onto the target view.
6.2.2 Quantitative Comparison
|Dir.||Methods||Dayton (64 64)||Dayton (256 256)||CVUSA|
|Zhai et al. (2017)||–||–||–||–||–||–||1.8434||1.5171||1.8666|
|Dir.||Methods||Dayton (64 64)||Dayton (256 256)||CVUSA|
|Accuracy (%)||Accuracy (%)||Accuracy (%)||Accuracy (%)||Accuracy (%)||Accuracy (%)|
|Zhai et al. (2017)||–||–||–||–||–||–||–||–||13.97||14.03||42.09||52.29|
We report the quantitative results on three datasets next.
Dayton dataset. The quantitative scores over different measures are provided in Tables 2, 3, 4, 5 and 6 under the columns Dayton (64 x 64) and Dayton (256 x 256) for low and high resolution synthesis.
Inception Score: The scores for X-Fork generated images are closest to that of real data distribution for Dayton dataset in low resolution in both directions and also in high resolution in a2g direction. The X-Seq method works best for g2a synthesis in high resolution for Dayton dataset. Inception scores on top-k classes follow a similar pattern as in all classes (except for Top-1 class on low resolution and Top-5 class on high resolution in g2a direction over Dayton dataset).
Accuracy: Results are shown in Table 3. We observe that the X-Fork method works better at low resolution whereas X-Seq is better at high resolution synthesis in both directions.
KL(model data): The scores are provided in Table 4. As it can be seen, our proposed methods generate much better results than the baselines. X-Fork generates images very similar to real distribution in all experiments except on the high resolution a2g experiment where X-Seq is slightly better than X-Fork.
FID Score: The FID scores are presented in Table 5. X-Fork performs the best on lower resolution images while X-Seq works the best on higher resolution images in both directions a2g and g2a.
SSIM, PSNR, and SD: The scores are reported in Table 6. The X-Seq model works the best in a2g direction while X-Fork outperforms the rest in the g2a direction.
Inception Score: The X-Seq method works best for CVUSA dataset in terms of inception score.
Accuracy: Results are shown in Table 3. Images generated with X-Fork method obtain the best accuracy closely followed by X-Seq method.
KL(model data): The scores are provided in Table 4. As it can be seen, our proposed methods generate much better results than the baseline. X-Fork generates images very similar to real distribution and X-Seq very close to it.
FID Score: X-Seq works the best on the CVUSA dataset closely followed by X-Fork network. See Table 5.
SSIM, PSNR, and SD: Images generated using X-Fork are better than other methods in terms of SSIM, PSNR and SD. We find that X-Fork improves over Zhai et al. by 5.03% in SSIM, 8.93% in PSNR, and 12.35% in SD.
SVA dataset. The quantitative evaluation on SVA dataset is illustrated in Table 7.
Inception Score: H-Regions generates images that have the inception score closest to that of real data.
Accuracy: H-Seq method performs best in terms of accuracy.
KL(model data): Images synthesized using X-Seq method have the closest distribution to the ground truth distribution among all the methods.
FID Score: The H-Regions performs the best in terms of FID score in SVA dataset. The H-models perform better than their X- counterparts.
SSIM, PSNR, and SD: X-Seq achieves the highest numbers in terms of SSIM and PSNR whereas H-Regions has the best SD.
Also, H-Pix2pix already performs very good compared to X-Pix2pix because homography simplified the learning task by transforming the image to the target view. H- methods outperform their X- counterparts for most of the evaluation metrics.
Because there is no consensus in evaluation of GANs, we had to use several scores. Theis et al. (2016) show that these scores often do not agree with each other and this was observed in our evaluations as well. Nonetheless, we find that the proposed methods are consistently superior to the baselines in terms of quantitative and qualitative evaluations.
|Zhai et al. (2017)||–||–|
|Zhai et al. (2017)||–||–|
|Dir.||Methods||Dayton (64 64)||Dayton (256 256)||CVUSA|
|Zhai et al. (2017)||–||–||–||–||–||–||0.4147||17.4886||16.6184|
|*Inception Score, all||2.0131||2.4951||2.1888||2.2232||2.1906||2.3202||2.3202||2.2394||2.6328|
|*Inception Score, Top-1||1.7221||1.8940||1.9776||1.9842||1.9507||1.9410||1.9525||1.9892||2.0732|
|*Inception Score, Top-5||2.2370||2.6634||2.3664||2.4344||2.4069||2.7340||2.3918||2.4385||2.8347|
|Accuracy (Top-1, all)||8.5961||7.5146||17.3794||19.5056||18.0706||5.2444||18.0182||20.7391||15.4803|
|Accuracy (Top-1, 0.5)||30.3288||30.9507||53.4725||57.1010||54.8068||26.4697||51.0756||57.5378||48.0767|
|Accuracy (Top-5, all)||9.0260||10.3905||23.8315||25.8807||23.4400||5.2544||26.6746||28.5517||21.8225|
|Accuracy (Top-5, 0.5)||29.9102||38.9822||63.5045||65.3005||62.3072||31.9527||62.8166||67.4649||56.8994|
*Inception Score for real (ground truth) data is 3.0347, 2.3886 and 3.3446 for all, top-1 and top-5 setups respectively.
7 Discussion and Conclusion
We explored image generation using conditional GANs between two drastically different views. Generating semantic segmentations together with images in the target view helps the networks learn better images compared to the baselines. Using homography to guide the cross-view synthesis allows preserving the overlapping regions between the views. Extensive qualitative and quantitative evaluations testify the effectiveness of our methods. Future research can explore other cues such as edge maps to facilitate synthesis tasks. Also, efforts can be put towards automatically finding regions that we defined manually. The challenging nature of the problem leaves room for further improvements. Potential application of this work can be in bridging the big domain-gap between street-view and aerial imagery for cross-view image geo-localization, multi-view object synthesis and so on.
- Ardeshir and Borji (2016) Ardeshir, S., Borji, A., 2016. Ego2top: Matching viewers in egocentric and top-view videos, in: Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V, pp. 253–268. doi:10.1007/978-3-319-46454-1_16.
- Borji (2018) Borji, A., 2018. Pros and cons of gan evaluation measures. arXiv preprint arXiv:1802.03446 .
- Che et al. (2016) Che, T., Li, Y., Jacob, A.P., Bengio, Y., Li, W., 2016. Mode regularized generative adversarial networks. arXiv e-prints abs/1612.02136.
Collobert et al. (2011)
Collobert, R., Kavukcuoglu, K.,
Farabet, C., 2011.
Torch7: A matlab-like environment for machine learning, in: BigLearn, NIPS Workshop.
- Cordts et al. (2016) Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B., 2016.
- Dosovitskiy et al. (2017) Dosovitskiy, A., Springenberg, J.T., Tatarchenko, M., Brox, T., 2017. Learning to generate chairs, tables and cars with convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 39, 692–705.
- Goodfellow et al. (2014) Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., 2014. Generative adversarial nets, in: Advances in neural information processing systems, pp. 2672–2680.
- Heusel et al. (2017) Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S., 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium, in: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (Eds.), Advances in Neural Information Processing Systems 30. Curran Associates, Inc., pp. 6626–6637.
- Hinton et al. (2006) Hinton, G.E., Osindero, S., Teh, Y.W., 2006. A fast learning algorithm for deep belief nets. Neural Comput. 18, 1527–1554. URL: http://dx.doi.org/10.1162/neco.2006.18.7.1527, doi:10.1162/neco.2006.18.7.1527.
- Ioffe and Szegedy (2015) Ioffe, S., Szegedy, C., 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift, in: Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37, JMLR.org. pp. 448–456. URL: http://dl.acm.org/citation.cfm?id=3045118.3045167.
Isola et al. (2017)
Isola, P., Zhu, J.Y.,
Zhou, T., Efros, A.A.,
Image-to-image translation with conditional adversarial networks.CVPR .
- Kim et al. (2017) Kim, T., Cha, M., Kim, H., Lee, J.K., Kim, J., 2017. Learning to discover cross-domain relations with generative adversarial networks, in: Precup, D., Teh, Y.W. (Eds.), Proceedings of the 34th International Conference on Machine Learning, PMLR, International Convention Centre, Sydney, Australia. pp. 1857–1865.
- Kossaifi et al. (2017) Kossaifi, J., Tran, L., Panagakis, Y., Pantic, M., 2017. Gagan: Geometry-aware generative adverserial networks. arXiv preprint arXiv:1712.00684 .
Krizhevsky et al. (2017)
Krizhevsky, A., Sutskever, I.,
Hinton, G.E., 2017.
Imagenet classification with deep convolutional neural networks.Commun. ACM 60, 84–90. URL: http://doi.acm.org/10.1145/3065386, doi:10.1145/3065386.
Ledig et al. (2017)
Ledig, C., Theis, L.,
Huszar, F., Caballero, J.,
Cunningham, A., Acosta, A.,
Aitken, A.P., Tejani, A.,
Totz, J., Wang, Z., Shi,
Photo-realistic single image super-resolution using a generative adversarial network, in: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 105–114.doi:10.1109/CVPR.2017.19.
- Lin et al. (2017) Lin, G., Milan, A., Shen, C., Reid, I., 2017. RefineNet: Multi-path refinement networks for high-resolution semantic segmentation, in: CVPR.
- Lin et al. (2015) Lin, T., Cui, Y., Belongie, S.J., Hays, J., 2015. Learning deep representations for ground-to-aerial geolocalization, in: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp. 5007–5015. doi:10.1109/CVPR.2015.7299135.
- Lin et al. (2013) Lin, T.Y., Belongie, S., Hays, J., 2013. Cross-view image geolocalization, in: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
- Mathieu et al. (2016) Mathieu, M., Couprie, C., LeCun, Y., 2016. Deep multi-scale video prediction beyond mean square error, in: 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
- MH and Lee (2018) MH, S.H.M.F.R., Lee, N.G.H., 2018. Cvm-net: Cross-view matching network for image-based ground-to-aerial geo-localization, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7258–7267.
- Mirza and Osindero (2014) Mirza, M., Osindero, S., 2014. Conditional generative adversarial nets. CoRR abs/1411.1784. URL: http://arxiv.org/abs/1411.1784.
- Nguyen et al. (2017) Nguyen, T., Le, T., Vu, H., Phung, D., 2017. Dual discriminator generative adversarial nets, in: Advances in Neural Information Processing Systems, pp. 2670–2680.
- Palazzi et al. (2017) Palazzi, A., Borghi, G., Abati, D., Calderara, S., Cucchiara, R., 2017. Learning to map vehicles into bird’s eye view, in: Battiato, S., Gallo, G., Schettini, R., Stanco, F. (Eds.), Image Analysis and Processing - ICIAP 2017, Springer International Publishing, Cham. pp. 233–243.
- Park et al. (2017) Park, E., Yang, J., Yumer, E., Ceylan, D., Berg, A.C., 2017. Transformation-grounded image generation network for novel 3d view synthesis. CoRR abs/1703.02921. URL: http://arxiv.org/abs/1703.02921.
- Pathak et al. (2016) Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A., 2016. Context encoders: Feature learning by inpainting, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2536–2544.
- Reed et al. (2016) Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., Lee, H., 2016. Generative adversarial text to image synthesis, in: Balcan, M.F., Weinberger, K.Q. (Eds.), Proceedings of The 33rd International Conference on Machine Learning, PMLR, New York, New York, USA. pp. 1060–1069.
- Regmi and Borji (2018) Regmi, K., Borji, A., 2018. Cross-view image synthesis using conditional gans, in: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
- Regmi and Shah (2019) Regmi, K., Shah, M., 2019. Bridging the domain gap for ground-to-aerial image matching. arXiv preprint arXiv:1904.11045 .
Salakhutdinov and Hinton (2009)
Salakhutdinov, R., Hinton, G.E.,
Deep boltzmann machines., in: International Conference on Artificial Intelligence and Statistics (AISTATS), p. 3.
- Salimans et al. (2016) Salimans, T., Goodfellow, I.J., Zaremba, W., Cheung, V., Radford, A., Chen, X., 2016. Improved techniques for training gans, in: Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 2226–2234.
- Shi et al. (2016) Shi, W., Caballero, J., Huszar, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., Wang, Z., 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 1874–1883. URL: https://doi.org/10.1109/CVPR.2016.207, doi:10.1109/CVPR.2016.207.
- Smolensky (1986) Smolensky, P., 1986. Parallel distributed processing: Explorations in the microstructure of cognition, vol. 1, MIT Press, Cambridge, MA, USA. chapter Information Processing in Dynamical Systems: Foundations of Harmony Theory, pp. 194–281.
- Song et al. (2017) Song, L., Lu, Z., He, R., Sun, Z., Tan, T., 2017. Geometry guided adversarial facial expression synthesis .
- Soran et al. (2014) Soran, B., Farhadi, A., Shapiro, L., 2014. Action recognition in the presence of one egocentric and multiple static cameras, in: Asian Conference on Computer Vision, Springer. pp. 178–193.
- Szegedy et al. (2016) Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z., 2016. Rethinking the inception architecture for computer vision, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 2818–2826. URL: https://doi.org/10.1109/CVPR.2016.308, doi:10.1109/CVPR.2016.308.
- Tatarchenko et al. (2016) Tatarchenko, M., Dosovitskiy, A., Brox, T., 2016. Multi-view 3d models from single images with a convolutional network, in: Leibe, B., Matas, J., Sebe, N., Welling, M. (Eds.), Computer Vision – ECCV 2016, Springer International Publishing, Cham. pp. 322–337.
- Theis et al. (2016) Theis, L., van den Oord, A., Bethge, M., 2016. A note on the evaluation of generative models, in: International Conference on Learning Representations.
- Vo and Hays (2016) Vo, N.N., Hays, J., 2016. Localizing and Orienting Street Views Using Overhead Imagery. Springer International Publishing, Cham. pp. 494–509. URL: http://dx.doi.org/10.1007/978-3-319-46448-0_30, doi:10.1007/978-3-319-46448-0_30.
- Workman et al. (2015) Workman, S., Souvenir, R., Jacobs, N., 2015. Wide-area image geolocalization with aerial reference imagery, in: IEEE International Conference on Computer Vision (ICCV).
- Yang et al. (2017) Yang, C., Lu, X., Lin, Z., Shechtman, E., Wang, O., Li, H., 2017. High-resolution image inpainting using multi-scale neural patch synthesis, in: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
- Yeh et al. (2017) Yeh, R.A., Chen, C., Lim, T.Y., Schwing, A.G., Hasegawa-Johnson, M., Do, M.N., 2017. Semantic image inpainting with deep generative models, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5485–5493.
- Zhai et al. (2017) Zhai, M., Bessinger, Z., Workman, S., Jacobs, N., 2017. Predicting ground-level scene layout from aerial imagery, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
- Zhang et al. (2017) Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., Metaxas, D., 2017. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks, in: ICCV.
- Zhou et al. (2017) Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A., 2017. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence .
- Zhou et al. (2016) Zhou, T., Tulsiani, S., Sun, W., Malik, J., Efros, A.A., 2016. View synthesis by appearance flow, in: Leibe, B., Matas, J., Sebe, N., Welling, M. (Eds.), Computer Vision – ECCV 2016, Springer International Publishing, Cham. pp. 286–301.
- Zhu et al. (2017) Zhu, J.Y., Park, T., Isola, P., Efros, A.A., 2017. Unpaired image-to-image translation using cycle-consistent adversarial networkss, in: Computer Vision (ICCV), 2017 IEEE International Conference on.