DLOW: Domain Flow for Adaptation and Generalization

1 Introduction

The domain shift problem is drawing more and more attention in recent years [19, 58, 50, 48, 13, 6]. In particular, there are two tasks that are of interest in computer vision community. One is the domain adaptation problem, where the goal is to learn a model for a given task from a label-rich data domain (i.e., source domain) that performs well in a label-scarce data domain (i.e., target domain). The other one is the image translation problem, where the goal is to transfer images in the source domain to mimic the image style in a target domain.

Figure 1: Illustration of data flow generation. Traditional image translation methods directly map the image from the source domain to the target domain, while our DLOW model is able to produce a sequence of intermediate domains shifting from the source domain to the target domain.

Generally, most existing works focus on the target domain only. They aim to learn models that well fit the target data distribution, e.g., achieving good classification accuracy in the target domain, or transferring source images into the target style. In this work, we instead are interested in the intermediate domains between the source and target domains. We propose a new domain flow generation (DLOW) approach, which is able to translate images from the source domain into an arbitrary intermediate domain between source and target domains. As shown in Fig 1, by translating a source image along the domain flow from the source domain to the target domain, we obtain a sequence of images that naturally characterize the distribution shift from the source domain to the target domain.

The benefits of our DLOW approach are two-fold. First, those intermediate domains are helpful to bridge the distribution gap between two domains. By translating images into intermediate domains, those translated images can be employed to ease the domain adaptation task. We show that the traditional domain adaptation methods can be boosted to achieve better performances in target domain with intermediate domain images. Moreover, the obtained models also exhibit good generalization ability on new datasets that are unseen in the training phase, benefiting from the diverse intermediate domain images.

Second, our DLOW model can be used for style generalization. Traditional image-to-image translation works

[58, 25, 27, 35]

focus on learning a deterministic one-to-one mapping that transfers a source image into the target style. In contrast, our DLOW model allows to translate a source image into an intermediate domain that are related to multiple target domains. For example, when performing the photo to painting transfer, instead of obtaining a Monet or Van Gogh style, our DLOW model could produce a painting with mixed styles of Van Gogh, Monet, etc. Such mixture can be customized arbitrarily in the inference phase by simply adjusting an input vector that encodes the relatedness to different domains.

We implement our DLOW model based on CycleGAN [58], which is one of the state-of-the-art unpaired image-to-image translation methods. We augment the CycleGAN to include an additional input of domainness variable. On one hand, the domainness variable is injected into the translation network by using a conditional instance normalization layer to affect the style of output images. On the other hand, it is also used as weights on discriminators to balance the relatedness of the output images to source and target domains. For multiple target domains, the domainness variable is extended as a vector containing the relatedness to all target domains.

We evaluate our DLOW model with two tasks, mixed-style image translation, and domain adaptation. For the first task, we show that our learnt model is able to translate a source image into an arbitrary mixture of multiple styles. For the second task, we are able to further improve the state-of-the-art cross-domain semantic segmentation methods by using the translated images in intermediate domains as training data. Extensive results on benchmark dataasets demonstrate the effectiveness of our proposed model.

2 Related Work

Image to Image Translation Our work is related to the image-to-image translation works. The image-to-image translation task aims at translating the image from one domain into another domain. Inspired by the success of Generative Adversarial Networks(GANs) [15], many works have been proposed to address the image-to-image translation based on GANs [25, 52, 58, 35, 36, 18, 59, 24, 1, 6, 30, 53, 34]. The early works [25, 52] assumed that paired images between two domains are available, while the recent works such as CycleGAN [58], DiscoGAN [27] and UNIT [35] employed cycle consistency loss to learn the mapping without using paired images. However, those works focus on learning deterministic image-to-image mappings. Once the model is learnt, a source image can only be transferred to a fixed target style, and v.v. A few recent works [36, 18, 59, 24, 1, 6, 30, 53, 34] concentrates on learning a unified model to translate images into different styles. For instance, the Augmented CycleGAN [1] proposed to inject a noise input into CycleGAN with Conditional Normalization (CN) [23, 8]

, leading to sytle variance of translated images. The StarGAN model

[6] adds the mask vector to the conditional GANs to train a single model which produces styles to mimic multiple datasets. However, they either exploit the style variance within a single domain, or train a unifed model to switch among different target domains. It is unclear for those methods how to generalize to a new unseen domain. In constrast, when multiple target domains are aviable, our work is able to transfer images into an arbitary intermediate domain relatd to those multiple domains. This allows us to translate the image into a new sytle that is unseen in the training data.

Domain Adaptation and Generalization Our work is also related to the domain adaptation and generalization works. Domain adaptation aims to utilize a labeled source domain to learn a model that performs well on a unlabeled target domain [11, 16, 10, 51, 26, 3, 28, 14, 29]. Domain generalization is a similar problem, which aims to learn a model that could be generalized to an unseen target domain by using multiple labeled source domains [39, 13, 42, 38, 41, 31, 33, 32].

Our work is partially inspired by SGF [16] and GFK [14], which have shown that the intermediate domains between source and target domains are useful for addressing the domain adaptation problem. They represented each domain as a subspace, and then connected them on Grassmannian manifold to model intermediate domains. Different from them, we model the intermediate domains by directly translate images on pixel level. This allows us to easily improve the existing deep domain adaptation models by using the translated images as training data. Moreover, our model can also be applied to image-level domain generalization by generating mixed-style images.

Recently, there is an increasing interests to apply domain adaptation techniques for semantic segmentation from synthetic data to the real scenario [20, 19, 5, 61, 37, 22, 9, 43, 47, 49, 21, 44, 56, 50, 40, 46, 48, 60]. Most of those works conduct the domain adaptation by adversarial training on the feature level with different priors. The recent Cycada [19] also show that it is beneficial to perform pixel-level domain adaptation firstly by transferring source image into the target style based on the image-to-image translation methods like CycleGAN [58]. However, those methods address domain shift by focusing on adapting to only the target domain. In contrast, we aim to perform pixel-level adaptation by transferring source images to a flow of intermediate domains, which is proven to be more effective than those that focus on only target domain in our experiments. Moreover, our model can also be used to further improve the existing feature-level adaptation methods.

3 Domain Flow Generation

In this section, we introduce domain flow generation (DLOW) model for translating source images into intermedaite domains that bridge the source and target domains.

3.1 Problem Statement

In the domain shift problem, we are given a source domain $S$ and a target domain $T$ containing samples from two different distributions $P_{S}$ and $P_{T}$ , respectively. Denoting $x^{s} \in S$ as a source domain sample and $x^{t} \in T$ , we have $x^{s} \sim P_{S}$ , $x^{t} \sim P_{T}$ , and $P_{S} \neq P_{T}$ .

Such distribution mismatch usually leads to a significant performance drop when applying the model trained on $S$ to the new target domain $T$ . Many works have been proposed to address the domain shift for different vision applications. A group of recent works aim to reduce the distribution difference on the feature level by learning domain-invariant features [11, 16, 28, 14], while others work on the image level to transfer source images to mimic the target domain style [58, 35, 59, 24, 1, 6].

In this work, we also propose to address the domain shift problem on image level. However, different from existing works that focus on transferring source images into only the target domain, we instead transfer them into all intermediate domains that connect source and target domains. This is partially motivated by the previous works [16, 14], which have shown that the intermediate domains between source and target domains are useful for addressing the domain adaptation problem.

In the follows, we first briefly review the conventional image-to-image translation model CycleGAN. Then, we formulate the intermediate domain adaptation problem based on the data distribution distance. Next, we develop our DLOW model based on the CycleGAN model. We then show the benefits of our DLOW model for two applications: 1) how to improve existing domain adaptation models with the images generated from DLOW model, and 2) how to transfer images into arbitrarily mixed styles when there are multiple target domains.

3.2 The CycleGAN Model

We build our model upon the state-of-the-art CycleGAN model [58] which was proposed for unpaired image-to-image translation. Formally, the CycleGAN model learns two mappings between $S$ and $T$ , i.e., $G_{S T} : S \to T$ which transfers the images in $S$ into the style of $T$ , and $G_{T S} : T \to S$ which acts in the inverse direction. We take the $S \to T$ direction as an example to explain CycleGAN.

To transfer source images into the target style and also preserve the semantics, the CycleGAN employed an adversarial training module and a reconstruction module, respectively. In particular, the adversarial training module is to align the image distributions for two domains, such that the style of mapped images matches the target domain. Let us denote $D_{T}$ as the discriminator, which attempts to distinguish the translated images and the target images. Then the objective function of the adversarial training module can be written as,

	$min G_{S T} max D_{T}$		$E_{x^{t} \sim P_{T}} [log (D_{T} (x^{t}))]$
		$+$	$E_{x^{s} \sim P_{S}} [log (1 - D_{T} (G_{S T} (x^{s})))] .$

Moreover, the reconstruction module is to ensure the mapped image $G_{S T} (x^{s})$ to preserve the semantic content of the original image $x^{s}$ . This is realized by enforcing a cyclic consistency loss such that $G_{S T} (x^{s})$ is able to recover $x^{s}$ when being mapped back to the source style, i.e.,

min G_{S T} E_{x^{s} \sim P_{S}} [∥ G_{T S} (G_{S T} (x^{s})) - x^{s} ∥_{1}] .

(2)

Similar modules are also applied to the $T \to S$ direction. By jointly optimizing all modules, the CycleGAN model is able to transfer source images into the target style and v.v.

Figure 2: Illustration of domain flow. Many possible paths (the green dash lines) connect source and target domains, while the domain flow is the shortest one (the red line). An intermediate domain (the blue dot) is the point at the domain flow that keeps the right distances to two domains.

Figure 3: Our DLOW model consists of three modules: (a) the generator $G_{S T}$ takes domainness $z$ as additional input to control the image translation; (b) $G_{T S}$ takes $z$ to reconstruct the source image; (c) $z$ is used as target values for learning regressor. Three modules are trained jointly.

3.3 Modeling Intermediate Domains

In our task, we aim to translate the source images not only into the target domain, but also into all intermediate domains that connect the source and target domains. In particular, let us denote an intermediate domain as $M^{(z)}$ , where $z \in [0, 1]$ is a continous variable which models the relatedness to source and target domains. We refer to $z$ as the domainness of intermediate domain. When $z = 0$ , the intermediate domain $M^{(z)}$ is identical to the source domain $S$ ; and when $z = 1$ , it is identical to the target domain $T$ . By varying $z$ in the range of $[0, 1]$ , we thus obtain a sequence of intermediate domains that flows from $S$ to $T$ .

There are many possible paths to connect the source and target domains. As shown in Fig 2, assuming there is a manifold of domains, where a domain with given data distribution can be seen as a point residing at the manifold. We expect the domain flow $M^{(z)}$ to be the shortest geodesic path connecting $S$ and $T$ . Moreover, given any $z$ , the distance from $S$ to $M^{(z)}$ should also be proportional to the distance between $S$ to $T$ by the value of $z$ . Denoting the data distribution of $M^{(z)}$ as $P_{M}^{(z)}$ , we expect that

\frac{d i s t (P_{S}, P_{M}^{(z)})}{d i s t (P_{T}, P_{M}^{(z)})} = \frac{z}{1 - z},

(3)

where $d i s t (\cdot, \cdot)$ is a valid distance measurement over two distributions.Thus, generating an intermediate domain $M^{(z)}$ for a given $z$ becomes to find the point satisfying Eq. (3) that is closet to $S$ and $T$ , which leads to minimize the following loss,

L = (1 - z) \cdot d i s t (P_{S}, P_{M}^{(z)}) + z \cdot d i s t (P_{T}, P_{M}^{(z)}) .

(4)

As shown in [2], many types of distance have been exploited for image generation and image translation. The adversarial loss in Eq. (3.2) can be seen as a lower bound of the Jessen-Shannon divergence. We also use it for measuring distribution distance in this work.

3.4 The DLOW Model

We now develop our DLOW model to generate intermediate domains. Given a source image $x^{s} \sim P_{s}$ , and a domainness parameter $z \in [0, 1]$ , our task is to transfer $x^{s}$ into the intermediate domain $M^{(z)}$ with the distribution $P_{M}^{(z)}$ that minimizes the objective in Eq. (4). We take the $S \to T$ direction as an example, and the other direction can be similarly applied.

In our DLOW model, a generator $G_{S T}$ is no longer to directly transfer $x^{s}$ to the target domain $T$ , but to move $x^{s}$ towards it. The interval of such moving is controlled by the domainness variable $z$ . Let us denote $Z = [0, 1]$ as the domain of $z$ , then the generator in our DLOW model can be represented as $G_{S T} (x^{s}, z) : S \times Z \to M^{(z)}$ where the input is a joint space of $S$ and $Z$ .

Adversarial Loss: As discussed in Section 3.3, We employ the adversarial loss as the distribution distance measurement to control the relatedness of an intermediate domain to the source and target domains. Specifically, we introduce two discriminators, $D_{S} (x)$ to distinguish $M^{(z)}$ and $S$ , and $D_{T} (x)$ to distinguish $M^{(z)}$ and $T$ , respectively. Then, the adversarial losses between $M^{(z)}$ and $S$ and $T$ can be written respectively as,

$L_{a d v} ($	$G_{S T}$	$, D_{S}) = E_{x^{s} \sim P_{S}} [log (D_{S} (x^{s}))]$
	$+$	$E_{x^{s} \sim P_{S}} [log (1 - D_{S} (G_{S T} (x^{s}, z)))]$
$L_{a d v} ($	$G_{S T}$	$, D_{T}) = E_{x^{t} \sim P_{T}} [log (D_{T} (x^{t}))]$
	$+$	$E_{x^{s} \sim P_{S}} [log (1 - D_{T} (G_{S T} (x^{s}, z)))] .$

By using the above losses to model $d i s t (P_{S}, P_{M}^{(z)})$ and $d i s t (P_{T}, P_{M}^{(z)})$ in Eq. (4), we arrive at the following loss,

L_{a d v} = (1 - z) L_{a d v} (G_{S T}, D_{S}) + z L_{a d v} (G_{S T}, D_{T}) .

(7)

Image Cycle Consistency Loss: Similarly as in CylceGAN, we also apply a cyclic consistency loss to ensure the semantic content is well-preserved in the translated images. Let us denote $G_{T S} (x^{t}, z) : T \times Z \to M^{(1 - z)}$ as the generator on the other direction, which transfers a sample $x^{t}$ from the target domain towards the source domain by a interval of $z$ . Since $G_{T S}$ acts in an inverse way to $G_{S T}$ , we can use it to recover $x^{s}$ from the translated version $G_{S T} (x^{s}, z)$ , which gives the following loss,

L_{c y c} =

E_{x^{s} \sim P_{s}} [∥ G_{T S} (G_{S T} (x^{s}, z), z) - x^{s} ∥_{1}] .

(8)

Domainness Cycle Consistency Loss: To guarantee that the translated image $G_{S T} (x^{s}, z)$ correctly encodes the information of the domainness parameter $z$ , we introduce a regressor $R_{S} : M^{(z)} \to z$ to reconstruct the domainness parameter $z$ . In particular, $R_{S}$ is expected to output $0$ for source images, $1$ for target images, and $z$ for images in $M^{(z)}$ . We use the cross entropy loss for source and target images, and the square loss for $z$ , and arrive at the domainness cycle consistency loss as follows,

$L_{d n s} = - E_{x^{t} \sim P_{T}}$	$[log (R_{S} (x^{t}))]$	(9)
$- E_{x^{s} \sim P_{S}}$	$[log (1 - R_{S} (x^{s})]$
$+ E_{x^{s} \sim P_{S}}$	$[∥ R_{S} (G_{S T} (x^{s}, z)) - z ∥_{2}] .$

Full Objective: We integrate the losses defined above, the full objective can be defined as:

L = L_{a d v} + λ_{1} L_{c y c} + λ_{2} L_{d n s},

(10)

where $λ_{1}$ and $λ_{2}$ are hyper-parameters used to balance the adversarial loss, the image cycle consistency loss and the domainness cycle consistency loss in the training process.

Similar loss can be defined for the other direction $T \to S$ . Due to the usage of adversarial loss $L_{a d v}$ , the training is performed in an alternating manner. We first minimize the full objective with regard to the generators and regressors, and then maximize it with regard to the discriminators.

Implementation: We illustrate the network structure of $S \to T$ direction of our DLOW model in Fig 3. A figure of the complete model is provided in the Appendix. First, the domainness parameter $z$ is taken as the input of the generator $G_{S T}$ . This is implemented with the Conditional Instance Normalization (CN) layer [1, 23]. We first use one deconvolution layer to map the domainness parameter $z$ to the vector with dimension $(1, 16, 1, 1)$ , and then use this vector as the input for the CN layer. Moreover, the domainness parameter also plays the role of weighting discriminators to balance the relatedness of the generated images to different domains. It is also used as input in the image cycle consistency module, as well as label for the domainness cycle consistency module. During the training phase, we randomly generate the domainess parameter $z$ for each input image. As inspired by [x], we forces the domainness parameter $z$

to obey the beta distribution, i.e.

f (z, α, β) = \frac{1}{B (α, β)} z^{α - 1} (1 - z)^{β - 1}

, where

β

is fixed as

1

, and

α

is a function of the training step

α = e^{\frac{t - 0.5 T}{0.25 T}}

with

t

being the current iteration and

T

being the total number of iterations. In this way,

z

tends to be sampled more likely as small values at the beginning, and gradually shift to larger values at the end, which gives slightly more stable training than uniform sampling.

3.5 Boosting Domain Adaptation Models

With the DLOW model, we are able to translate each source image $x^{s}$ into an arbitary intermediate domain $M^{(z)}$ . Let us denote the source dataset as $S = {(x_{i}^{s}, y_{i}) |_{i = 1}^{n}}$ where $y_{i}$ is the label of $x_{i}^{s}$ . By inputting each of the image $x_{i}^{s}$ combined with $z_{i}$

randomly sampled from the uniform distribution

U (0, 1)

, we then obtain a translated dataset

~ S = {({~ x}_{i}^{s}, y_{i}) |_{i = 1}^{n}}

where

{~ x}_{i}^{s} = G_{S T} (x_{i}^{s}, z_{i})

is the translated version of

x_{i}^{s}

. The images in

~ S

spread along the domain flow from source to target domain, therefore, becomes much more diverse. Using

~ S

as the training data is helpful to learn a domain-invariant models for computer vision tasks. In Section 4.1, we demonstrate that model trained on

~ S

achieves good performance for the cross-domain semantic segmentation problem.

Moreover, the translated dataset $~ S$ can also be used to boost the existing adversarial training based domain adaptation approaches. Images in $~ S$ fill the gap between the source and target domains, and thus ease the domain adaptation task. Taking semantic segmentation as an example, a typical way is to append a discriminator to the segmentation model, which is used to distinguish the source and target samples. Using the adversarial training strategy to optimize the discriminator and the segmentation model, the segmentation model is trained to be more domain-invariant.

As shown in Fig 4, we replace the source dataset $S$ with the translated version $~ S$ , and apply a weight $\sqrt{1 - z_{i}}$ to the adversarial loss. The motivation is as follows, for each sample ${~ x}_{i}^{s}$ , if the domainness $z_{i}$ is higher, it is closer to the target domain, then the weight of adversarial loss can be reduced. Otherwise, we should enhance the loss weight.

Figure 4: Intermediate domain images are used as source dataset, and the adversarial loss is weighted by domainness.

3.6 Style Generalization

Most existing image-to-image translation works learn a deterministic mapping between two domains. After learning the model, source images can only be translated to a fixed style. In contrast, our DLOW model take an random $z$ to translate images into various styles. When multiple target domains are provided, it is also able to transfer the source image into a mixture of different target styles. In other words, we are able to generalize to an unseen intermediate domain that are related to existing domains.

In particular, suppose we have $K$ target domains, denoted as $T_{1}, \dots, T_{K}$ . Accordingly, the domainness variable $z$ is expanded as a $K$ -dim vector $z = [z_{1}, \dots, z_{K}]^{'}$ with $\sum_{k = 1}^{K} z_{k} = 1$ . Each elelment $z_{k}$ represents the relatedness to the $k$ -th target domain. To map an image from the source domain to the intermediate domain defined by $z$ , we need to optimize the following objective,

L = K \sum k = 1 z_{k} \cdot d i s t (P_{M}, P_{T_{k}}), s.t. K \sum 1 z_{k} = 1

(11)

where $P_{M}$ is the distribution of the intermediate domain, $P_{T_{K}}$ is the distribution of $T_{k}$ . The network structure can be easily adjusted from our DLOW model to optimize the above objective. We leave the details in the Appendix due to the space limitation.

GTA5 $\to$ Cityscapes
Method	road	sidewalk	building	wall	fence	pole	traffic light	traffic sign	vegetation	terrian	sky	person	rider	car	truck	bus	train	motorbike	bicycle	mIoU
NonAdapt[50]	75.8	16.8	77.2	12.5	21.0	25.5	30.1	20.1	81.3	24.6	70.3	53.8	26.4	49.9	17.2	25.9	6.5	25.3	36.0	36.6
CycleGAN[19]	81.7	27.0	81.7	30.3	12.2	28.2	25.5	27.4	82.2	27.0	77.0	55.9	20.5	82.8	30.8	38.4	0.0	18.8	32.3	41.0
DLOW( $z = 1$ )	88.5	33.7	80.7	26.9	15.7	27.3	27.7	28.3	80.9	26.6	74.1	52.6	25.1	76.8	30.5	27.2	0.0	15.7	36.0	40.7
DLOW	87.1	33.5	80.5	24.5	13.2	29.8	29.5	26.6	82.6	26.7	81.8	55.9	25.3	78.0	33.5	38.7	0.0	22.9	34.5	42.3

Table 1: Results of segmentation segmentation on the CityScapes dataset based on DeepLab-v2 model with ResNet-101 backbone using the images translated with different models.

	Cityscapes	KITTI	WildDash	BDD100K
Original [50]	42.4	30.7	18.9	37.0
DLOW	44.8	36.6	24.9	39.1

Table 2: Comparison of the performance of AdaptSegNet [50] when using original source images and intermediate domain images translated with our DLOW model for semantic segmention under domain adaptation (1st column) and domain generalization (2nd to 4th columns) scenarios

4 Experiments

In this section, we demonstrate the benefits of our DLOW model with two tasks. In the first task, we address the domain adaptation problem, and train our DLOW model to generate the intermediate domain sample to boost the domain adaptation performance. In the second task, we consider the style generalization problem, and train our DLOW model to transfer images into new styles that are unseen in the training data.

4.1 Domain Adaptation and Generalization

4.1.1 Experiments Setup

For the domain adaptation problem, we follow [20, 19, 5, 61] to conduct experiments on the urban scene semantic segmentation by learning from synthetic data to real scenario. The GTA5 dataset [45] is used as the source domain while the Cityscapes dataset [7] as the target domain. Moreover, we also evaluate the generalization ability of learnt segmentation models to unseen domains, for which we take the KITTI [12], WildDash [55] and BDD100K [54] datasets as additional unseen datasets for evaluation.

Cityscapes is a dataset consisting of urban scene images taken from some European cities. We use the $2, 993$ training images without annotation as unlabeled target samples in training phase, and 500 validation images with annotation for evaluation, which are densely labelled with 19 classes.

GTA5 is a dataset consisting of $24, 966$ densely labelled synthetic frames generated from the computer game whose scenes are based on the city of Los Angeles. The annotations of the images are compatible with the Cityscaps.

KITTI is a dataset consisting of images taken from mid-size city of Karlsruhe. We use 200 validation images densely labeled and compatible with Cityscapes.

WildDash is a dataset covers images from different sources, different environments(place, weather, time and so on) and different camera characteristics. We use 70 labeled and Cityscapes annotation compatible validation images.

BDD100K is a driving dataset covering diverse images taken from US whose label maps are with training indices specified in Cityscapes. We use $1, 000$ densely labeled images for validation in our experiment.

In this task, we first train our proposed DLOW model using the GTA5 dataset as the source domain, and Cityscapes as the target domain. Then, we generate a translated GTA5 dataset with the learnt DLOW model. Each source image is inputted into DLOW with a random domainness variable $z$ . The new translated GTA5 dataset contains exactly the same number images as the original one, but the styles of images randomly drift from the synthetic style to the real style. We then use the translated GTA dataset as the new source domain for training segmentation models.

We implement our DLOW model based on Augmented CycleGAN [1] and CyCADA [19]. Following their setup, all images are resized to $1024 \times 1024$ and the crop size is set as $400 \times 400$ . When training the DLOW model, the image cycle loss weight is 10 and the domain cycle loss weight is 1. The learning rate is fixed as 0.0002. For the segmentation network, we use the AdaptSegNet [50] model, which is based on DeepLab-v2 [4] with ResNnet-101 [17] as the backbone network. The training images are resized to $1280 \times 720$ . We follow the exact the same training policy as in the AdaptSegNet.

4.1.2 Experimental Results

Intermediate Domain Images: To verify the ability of our DLOW model to generate intermediate domain images, after learning the model, we fix the input source image, and vary the domainness parameter from 0 to 1. A few examples are shown in Fig 5. It can be observed that the styles of translated images gradually shift from the synthetic style of GTA5 to the real style of Cityscapes, which demonstrates the DLOW model is capable of modeling the domain flow to bridge the source and target domains as expected.

Figure 6: Examples of style generalization results. The vectors above each images are the domainness.

Cross-Domain Semantic Segmentation: We further evaluate the usefulness of intermediate domain images in two settings. In the first setting, we compare with the CycleGAN model [58], which was used in the CycADA approach [19] for performing pixel-level domain adaptation. The difference between CycleGAN and our DLOW model is that CycleGAN transfers source images to mimic only the target style, while our DLOW model transfers source images into random styles flowing from the source domain to the target domain. We first obtain a translated version of the GTA5 dataset with each model. Then, we respectively use the two transalated GTA5 datasets to train DeepLab-v2 models, which are evaluated on the Cityscapes dataset for semantic segmentation. We also include the “NonAdapt” baseline which uses the original GTA5 images as training data, as well as a special case of our approach, “DLOW( $z = 1$ )”, where we set $z = 1$ for all source images when making image translation using the learnt DLOW model.

The results are shown in Table 1. We observe that all pixel-level adaptation methods outperform the “NonAdapt” baseline, which verifies that image translation is helpful for training models for cross-domain semantic segmentation. Moreover, ‘DLOW( $z = 1$ )” is a special case of our model that directly translates source images into the target domain, which non-surprisingly gives comparable results as the CycADA-pixel method ( $40.7 %$ v.s. $41.0 %$ ). By further using intermedaite domain images, our DLOW model is able to improve the segmentation results from $40.7 %$ to $42.3 %$ , which demonstrates that intermediate domain images are helpful for learning a more robust domain-invariant segmentation model.

In the second setting, we further use intermediate domain images to improve the feature-level domain adpatation model. We conduct experiments based on the AdaptSegNet method [50], which is open source and has reported the state-of-the-art result for GTA5 $\to$ CityScapes. It consists of multiple levels of adversarial training, and we augment each level with the loss weight discussed in Section 3.5. The results are reported in Table 2. The “Original” method denotes the AdaptSegNet model that trained using GTA5 as the source domain, for which the results are obtained using their released pretrained model. The “DLOW” method is AdaptSegNet trained using translated dataset with our DLOW model. From the first column, we observe that the intermediate domain images are able to improve the AdaptSegNet model by $2.5 %$ from $42.3$ to $44.8$ . More interestingly, we show that the AdaptSegNet model with DLOW translated images also exhibits excellent domain generalization ability when being applied to unseen domains, which achieves significantly better results than the original AdaptSegNet model on the KITTI, WildDash and BDD100K datasets as reported in the second to the fourth columns, respectively. This shows that intermediate domain images are useful to improve the model’s cross-domain generalization ability.

4.2 Style Generalization

We conduct the style generalization experiment on the Photo to Artworks dataset[58], which consists of real photographs ( $6, 853$ images) and artworks from Monet( $1, 074$ images), Cezanne( $584$ images), Van Gogh( $401$ images) and Ukiyo-e( $1, 433$ images). We use the real photographs as the source domain, and the remaining as four target domains. As discussed in Section 3.6, The domainness variable in this experiment is expanded as a $1 \times 4$ vector $[z_{1}, z_{2}, z_{3}, z_{4}]$ meeting the condition $\sum_{i = 1}^{4} z_{i} = 1$ . Also, $z_{1}, z_{2}, z_{3}$ and $z_{4}$ are corresponding to Monet, Van Gogh, Ukiyo-e and Cezanne, respectively. Each element $z_{i}$ can be seen as how much each style contributes to the final mixture style. For example, the domainness parameter $[1, 0, 0, 0]$ , $[0.5, 0.5, 0, 0]$ and $[0.25, 0.25, 0.25, 0.25]$ represents the pure Monet style, Monet and Van Gogh equally mixture style and Four styles equally mixture style, respectively. In every 5 steps of the training, we set the domainness parameter $z$ as $[1, 0, 0, 0]$ , $[0, 1, 0, 0]$ , $[0, 0, 1, 0]$ , $[0, 0, 0, 1]$

and uniformly distributed random variable. The weight of the cycle loss in training is 10 and the learning rate is set as 0.002. And we train the model for 54 epochs. The qualitative results of the style generalization are shown in Fig

6. From the qualitative results, it is shown that our DLOW model can translate the photo image to corresponding artworks with different styles. When varying the values of domainness vector, we could also successfully produce new styles related to different painting styles, which demonstrates the good generalization ability of our model to unseen domains. Note, different from [57, 23] works, we do not need any reference image in the test phase, and the domainness vector can be changed instantly to generate different new styles of images. We provide more examples in Appendix.

5 Conclusion

In this paper, we have presented the DLOW model to generate intermediate domains for bridging different domains. The model takes a domainness variable $z$ (or domainness vector $z$ ) as the conditional input, and transfers images into the intermediate domain controlled by $z$ or $z$ . We demonstrate the benefits of our DLOW model in two scenarios. Firstly, for the cross-domain semantic segmentation task, our DLOW model could improve the performance of the pixel-level domain adaptation by taking the translated images in intermediate domains as training data. Secondly, our DLOW model also exhibits excellent style generalization ability for image translation and we are able to transfer images into a new style that is unseen in the training data. Extensive experiments on benchmark datasets have verified the effectiveness of our proposed model.

References

[1] A. Almahairi, S. Rajeswar, A. Sordoni, P. Bachman, and A. Courville. Augmented cyclegan: Learning many-to-many mappings from unpaired data. arXiv preprint arXiv:1802.10151, 2018.
[2] M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein gan. arXiv preprint arXiv:1701.07875, 2017.
[3] M. Baktashmotlagh, M. T. Harandi, B. C. Lovell, and M. Salzmann. Unsupervised domain adaptation by domain invariant projection. In Proceedings of the IEEE International Conference on Computer Vision, pages 769–776, 2013.
[4] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834–848, 2018.
[5] Y. Chen, W. Li, and L. Van Gool. Road: Reality oriented adaptation for semantic segmentation of urban scenes. In
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
, pages 7892–7901, 2018.
[6] Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[7] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele.
The cityscapes dataset for semantic urban scene understanding.
In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[8] V. Dumoulin, J. Shlens, and M. Kudlur. A learned representation for artistic style. Proc. of ICLR, 2017.
[9] A. Dundar, M.-Y. Liu, T.-C. Wang, J. Zedlewski, and J. Kautz. Domain stylization: A strong, simple baseline for synthetic to real image domain adaptation. arXiv preprint arXiv:1807.09384, 2018.
[10] B. Fernando, A. Habrard, M. Sebban, and T. Tuytelaars. Unsupervised visual domain adaptation using subspace alignment. In Proceedings of the IEEE international conference on computer vision, pages 2960–2967, 2013.
[11] Y. Ganin and V. S. Lempitsky.
Unsupervised domain adaptation by backpropagation.
In ICML, 2015.
[12] A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
[13] M. Ghifary, W. Bastiaan Kleijn, M. Zhang, and D. Balduzzi.
Domain generalization for object recognition with multi-task autoencoders.
In Proceedings of the IEEE international conference on computer vision, pages 2551–2559, 2015.
[14] B. Gong, Y. Shi, F. Sha, and K. Grauman. Geodesic flow kernel for unsupervised domain adaptation. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 2066–2073. IEEE, 2012.
[15] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672–2680, 2014.
[16] R. Gopalan, R. Li, and R. Chellappa. Domain adaptation for object recognition: An unsupervised approach. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 999–1006. IEEE, 2011.
[17] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
[18] Z. He, W. Zuo, M. Kan, S. Shan, and X. Chen. Arbitrary facial attribute editing: Only change what you want. arXiv preprint arXiv:1711.10678, 2017.
[19] J. Hoffman, E. Tzeng, T. Park, J.-Y. Zhu, P. Isola, K. Saenko, A. Efros, and T. Darrell. CyCADA: Cycle-consistent adversarial domain adaptation. In J. Dy and A. Krause, editors,
Proceedings of the 35th International Conference on Machine Learning
, volume 80 of Proceedings of Machine Learning Research, pages 1989–1998, Stockholmsmässan, Stockholm Sweden, 10–15 Jul 2018. PMLR.
[20] J. Hoffman, D. Wang, F. Yu, and T. Darrell. Fcns in the wild: Pixel-level adversarial and constraint-based adaptation. arXiv preprint arXiv:1612.02649, 2016.
[21] W. Hong, Z. Wang, M. Yang, and J. Yuan. Conditional generative adversarial network for structured domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1335–1344, 2018.
[22] H. Huang, Q. Huang, and P. Krähenbühl. Domain transfer through deep activation matching. In European Conference on Computer Vision, pages 611–626. Springer, 2018.
[23] X. Huang and S. Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV, 2017.
[24] X. Huang, M.-Y. Liu, S. Belongie, and J. Kautz. Multimodal unsupervised image-to-image translation. In ECCV, 2018.
[25] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros.
Image-to-image translation with conditional adversarial networks.
CVPR, 2017.
[26] I.-H. Jhuo, D. Liu, D. Lee, and S.-F. Chang. Robust visual domain adaptation with low-rank reconstruction. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 2168–2175. IEEE, 2012.
[27] T. Kim, M. Cha, H. Kim, J. K. Lee, and J. Kim. Learning to discover cross-domain relations with generative adversarial networks. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1857–1865, International Convention Centre, Sydney, Australia, 06–11 Aug 2017. PMLR.
[28] E. Kodirov, T. Xiang, Z. Fu, and S. Gong. Unsupervised domain adaptation for zero-shot learning. In Proceedings of the IEEE International Conference on Computer Vision, pages 2452–2460, 2015.
[29] B. Kulis, K. Saenko, and T. Darrell. What you saw is not what you get: Domain adaptation using asymmetric kernel transforms. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages 1785–1792. IEEE, 2011.
[30] H.-Y. Lee, H.-Y. Tseng, J.-B. Huang, M. K. Singh, and M.-H. Yang. Diverse image-to-image translation via disentangled representations. In European Conference on Computer Vision, 2018.
[31] H. Li, S. J. Pan, S. Wang, and A. C. Kot. Domain generalization with adversarial feature learning. In CVPR, 2018.
[32] W. Li, Z. Xu, D. Xu, D. Dai, and L. Van Gool. Domain generalization and adaptation using low rank exemplar svms. IEEE transactions on pattern analysis and machine intelligence, 40(5):1114–1127, 2018.
[33] Y. Li, X. Tian, M. Gong, Y. Liu, T. Liu, K. Zhang, and D. Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 624–639, 2018.
[34] J. Lin, Y. Xia, T. Qin, Z. Chen, and T.-Y. Liu. Conditional image-to-image translation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)(July 2018), 2018.
[35] M.-Y. Liu, T. Breuel, and J. Kautz. Unsupervised image-to-image translation networks. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 700–708. Curran Associates, Inc., 2017.
[36] Y. Lu, Y.-W. Tai, and C.-K. Tang. Conditional cyclegan for attribute guided face image generation. arXiv preprint arXiv:1705.09966, 2017.
[37] Y. Luo, L. Zheng, T. Guan, J. Yu, and Y. Yang. Taking a closer look at domain shift: Category-level adversaries for semantics consistent domain adaptation. arXiv preprint arXiv:1809.09478, 2018.
[38] S. Motiian, M. Piccirilli, D. A. Adjeroh, and G. Doretto. Unified deep supervised domain adaptation and generalization. In The IEEE International Conference on Computer Vision (ICCV), volume 2, page 3, 2017.
[39] K. Muandet, D. Balduzzi, and B. Schölkopf. Domain generalization via invariant feature representation. In S. Dasgupta and D. McAllester, editors, Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 10–18, Atlanta, Georgia, USA, 17–19 Jun 2013. PMLR.
[40] Z. Murez, S. Kolouri, D. Kriegman, R. Ramamoorthi, and K. Kim. Image to image translation for domain adaptation. arXiv preprint arXiv:1712.00479, 13, 2017.
[41] L. Niu, W. Li, and D. Xu. Multi-view domain generalization for visual recognition. In Proceedings of the IEEE International Conference on Computer Vision, pages 4193–4201, 2015.
[42] L. Niu, W. Li, and D. Xu. Visual recognition by learning from web data: A weakly supervised domain generalization approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2774–2783, 2015.
[43] X. Pan, P. Luo, J. Shi, and X. Tang. Two at once: Enhancing learning and generalization capacities via ibn-net. In ECCV, 2018.
[44] X. Peng, B. Usman, N. Kaushik, D. Wang, J. Hoffman, K. Saenko, X. Roynard, J.-E. Deschaud, F. Goulette, T. L. Hayes, et al. Visda: A synthetic-to-real benchmark for visual domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 2021–2026, 2018.
[45] S. R. Richter, V. Vineet, S. Roth, and V. Koltun. Playing for data: Ground truth from computer games. In B. Leibe, J. Matas, N. Sebe, and M. Welling, editors, European Conference on Computer Vision (ECCV), volume 9906 of LNCS, pages 102–118. Springer International Publishing, 2016.
[46] K. Saito, K. Watanabe, Y. Ushiku, and T. Harada. Maximum classifier discrepancy for unsupervised domain adaptation. arXiv preprint arXiv:1712.02560, 3, 2017.
[47] F. S. Saleh, M. S. Aliakbarian, M. Salzmann, L. Petersson, and J. M. Alvarez. Effective use of synthetic data for urban scene semantic segmentation. In European Conference on Computer Vision, pages 86–103. Springer, Cham, 2018.
[48] S. Sankaranarayanan, Y. Balaji, A. Jain, S. N. Lim, and R. Chellappa. Unsupervised domain adaptation for semantic segmentation with gans. arXiv preprint arXiv:1711.06969, 2017.
[49] S. Sankaranarayanan, Y. Balaji, A. Jain, S. N. Lim, and R. Chellappa. Learning from synthetic data: Addressing domain shift for semantic segmentation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[50] Y.-H. Tsai, W.-C. Hung, S. Schulter, K. Sohn, M.-H. Yang, and M. Chandraker. Learning to adapt structured output space for semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[51] E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell. Adversarial discriminative domain adaptation. In Computer Vision and Pattern Recognition (CVPR), volume 1, page 4, 2017.
[52] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[53] Z. Yi, H. R. Zhang, P. Tan, and M. Gong. Dualgan: Unsupervised dual learning for image-to-image translation. In ICCV, pages 2868–2876, 2017.
[54] F. Yu, W. Xian, Y. Chen, F. Liu, M. Liao, V. Madhavan, and T. Darrell. Bdd100k: A diverse driving video database with scalable annotation tooling. arXiv preprint arXiv:1805.04687, 2018.
[55] O. Zendel, K. Honauer, M. Murschitz, D. Steininger, and G. Fernandez Dominguez. Wilddash - creating hazard-aware benchmarks. In The European Conference on Computer Vision (ECCV), September 2018.
[56] Y. Zhang, Z. Qiu, T. Yao, D. Liu, and T. Mei. Fully convolutional adaptation networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6810–6818, 2018.
[57] Y. Zhang, Y. Zhang, and W. Cai. A unified framework for generalizable style transfer: Style and content separation. arXiv preprint arXiv:1806.05173, 2018.
[58] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv preprint, 2017.
[59] J.-Y. Zhu, R. Zhang, D. Pathak, T. Darrell, A. A. Efros, O. Wang, and E. Shechtman. Toward multimodal image-to-image translation. In Advances in Neural Information Processing Systems, pages 465–476, 2017.
[60] X. Zhu, H. Zhou, C. Yang, J. Shi, and D. Lin. Penalizing top performers: Conservative loss for semantic segmentation adaptation. arXiv preprint arXiv:1809.00903, 2018.
[61] Y. Zou, Z. Yu, B. V. Kumar, and J. Wang. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In Proceedings of the European Conference on Computer Vision (ECCV), pages 289–305, 2018.

6 Appendix

In this Appendix, we provide additional information for,

the complete pipeline of our proposed DLOW model,
the detailed network structure of our DLOW model for style generalization with four target domains,
more examples for style generalization.

6.1 Pipeline of the DLOW model

In Section 3.4 of the main paper, we introduced three modules of the DLOW model by taking the direction $S \to T$ as an example. Those three modules were illustrated separately for clarity. In this Appendix, we further present the complete pipeline for a better illustration of our DLOW model. By combining the three modules of the direction $S \to T$ and supplementing the structure of the direction $T \to S$ , the complete model is shown in Fig 7. Taking the direction $S \to T$ for an example (c.f. Fig 6(a)), the orange dotted box shows the adversarial loss module which was illustrated in Fig 3a of the main paper. Correspondingly, the green dotted box and the purple dotted box are the image reconstruction module and the domainness reconstruction module which were illustrated in Fig 3b and Fig 3c of the main paper, respectively. Also, the structure of the other direction $T \to S$ is presented in Fig 6(b), which is symmetric to the direction $S \to T$ .

6.2 Network Structure for Style Generalization

In Section 3.6 of the main paper, we introduced that our DLOW model can be adapted for style generalization when there are multiple target domains available. We present the details in this section. The network structure of our DLOW model for style generalization is shown in Fig 8, where we have four target domains, each of which represents an image style. For the direction of $S \to T$ , shown in Fig 7(a), the style generalization model consists of three modules, the adversarial module, the image reconstruction module and the domainness reconstruction module. For each target domain $T_{i}$ , there is one corresponding discriminator $D_{T_{i}}$ measuring the distribution distance between the source domain $S$ and the target domain $T_{i}$ . Accordingly, the domainness variable $z$ is expanded as a $4$ -dim vector $z = [z_{1}, \dots, z_{4}]^{'}$ . Also, the output of the regressor is expanded to be multiple dimensions to reconstruct the domainness vector. For the other direction $T \to S$ , shown in Fig 7(b), the adversarial module is similar to that of the direction $S \to T$ . However, the image reconstruction module is slightly different, since the image reconstruction loss should be weighted by the domainness vector $z$ .

6.3 Additional Results for Style Generalization

We provided two examples for style generalization in Fig 6 of the main paper. Here we provide more experimental results in Fig 9, Fig 10 and Fig 11. The images with red bounding boxes are translated images in four target domains, i.e., Monet, Van Gogh, Cezanne, and Ukiyo-e. Those can be considered as the “seen” styles. Our model gives similar translation results to CycleGAN model for each target domain. But the difference is that we only need one unified model for the four target domains while the CycleGAN should train four models. Moreover, the images with green bounding boxes are the mixed style images of their neighboring target styles and the image in the center is the mixed style image of all the four target styles, which are new styles that are never seen in the training data. We can observe that our DLOW model could generalize well across different styles, which proves the good domain generalization ability of our model.

DLOW: Domain Flow for Adaptation and Generalization

Cluster, Split, Fuse, and Update: Meta-Learning for Open Compound Domain Adaptive Semantic Segmentation

Test-time Fourier Style Calibration for Domain Generalization

Generalizing Nucleus Recognition Model in Multi-source Images via Pruning

Source-Free Open Compound Domain Adaptation in Semantic Segmentation

Cross-Domain Visual Recognition via Domain Adaptive Dictionary Learning

Multi-Domain Level Generation and Blending with Sketches via Example-Driven BSP and Variational Autoencoders

Bridging the Source-to-target Gap for Cross-domain Person Re-Identification with Intermediate Domains

Code Repositories

DLOW

1 Introduction

2 Related Work