Style transfer methods apply the “style” from one example image to the “content” of another; for instance, one might render a camera image (the content) as a watercolor painting (the style). Recent work has shown that highly effective style transfer can be achieved by searching for an image such that early layers of CNN representation match the early layers of the style image and later layers match the later layers of a content image . Content matching is by comparing unit outputs at each location of feature map. But style matching is achieved by comparing summary statistics – in particular, the gram matrix – of the layers individually. Comparing gram matrices of individual layers ensures that small, medium and large patterns that are common in the style image appear with about the same frequency in the synthesized image, and that spatial co-occurences between these patterns are about the same in synthesized and style image.
Novak and Nikulin noticed that across-layer gram matrices reliably produce improvement on style transfer. (). However, their work was an exploration of variants of style transfer rather than a thorough study to gain insights on style summary statistics. There are reasons cross-layer terms produce improvements. In some styles, very long scale patterns are formed out of small components. For instance, in Figure 3, small white spots are organized into long curves. Within-layer gram matrices are not well adapted to represent this phenomenon, as Figure 3 shows. Generally, such hard styles occur where effects at short spatial scales are organized into longer scale structures. Such hard styles are strongly associated with physical materials (for instance, relief painting in Figure 2). In this paper, we show that comparing cross-layer gram matrices – which encode co-occurrences between (say) small and medium scale patterns — produces improvements in style transfer for such styles. Furthermore, controlling cross-layer gram matrices also effectively controls pattern frequencies.
We show that controlling cross-layer, rather than within-layer, gram matrices produces visible improvements in style transfer for many styles even though the cross-layer has less constraints than within-layer. This observation differs from the main claim of Novak and Nikulin, which suggests more layers(16layers) are needed for cross-layer gram matrix to improve within layer terms . Furthermore, they found reliable small improvements from cross-layer gram-matrices; in contrast, we argue that the method produces large, principled improvements, particularly for styles where inter-scale relations are important (Figure 1, 2).
Figure 4: Left: styles to transfer; center: results using within-layer loss; right: results using cross-layer loss. There are visible advantages to using the cross-layer loss. Note how cross-layer preserves the shape of the abstract color blocks (top row); avoids smearing large paint strokes (second row); preserves the overall structure of the curves as much as possible (third row); and produces color blocks with thin boundaries (fourth row).
We show that universal style transfer (UST) method can adapt to cross-layer gram matrices, consequently improving style transfer.
We demonstrate that loss multiplication method often produce better looking style transfers. We claim that multiplicative loss has stronger capability to encourage the stylized image to preserve prominent boundaries in content image geometry than additive loss does.
2 Related work
Bilinear models are capable of simple image style transfer  by factorizing style and content representations, but non-parametric methods like patch-based texture synthesis can deal with much more complex texture fields . Image analogies use a rendering of one image in two styles to infer a mapping from a content image to a stylized image . Researchers have been looking for versatile parametric methods to control style patterns at different scales to be transferred. Adjusting filter statistics is known to yield texture synthesis [1, 20]
. Gatys et al. demonstrated that producing neural network layers with particular summary statistics (i.e Gram matrices) yielded effective texture synthesis. In a following paper, Gatys et al. achieved style transfer by searching for an image that satisfies both style texture summary statistics and content constraints . This work has been much elaborated. The search can be replaced with a regression (at one scale ; at multiple scales ; with cached  or learned  style representations) or a decoding process that allows efficient adjusting of statistics . Search can be sped up with local matching methods . Methods that produce local maps (rather than pixels) result in photorealistic style transfer [19, 17]. Style transfer can be localized to masked regions . The criterion of matching summary statistics is a Maximum Mean Discrepancy condition . Style transfer can be used to enhance sketches .
Novak and Nikulin search a range of variant style transfer methods, including cross-layer gram matrices. However, their primary suggestions are adding more layers for more features, and shifting activations such that the number of zero entries in gram matrix is reduced. They don’t pursue on cross-layer gram matrices nor explain its results. They experiment on a long chain of cross-layer gram matrices but do not identify what the improvements are or extend the method to fast style transfer . There is a comprehensive review in .
3 Within layer gram matrix for style transfer
Gatys et al.  finds an image where early layers of a CNN representation match the lower layers of the style image and higher layers match the higher layers of a content image. We review the original work of Gatys et al. in detail. Write (resp. , ) for the style (resp. content, new) image, and for some parameter balancing style and content losses ( and respectively). We obtain by optimizing
Losses are computed on a network representation, with convolutional layers, where the ’th layer produces a feature map of size (resp. height, width, and channel number). We partition the layers into three groups (style, content and irrelevant). Then we reindex the spatial variables (height and width) and write for the response of the ’th channel at the ’th location in the ’th convolutional layer. The content loss is
(where ranges over content layers). The style loss is depends on within-layer gram matrices. Write
and for the weight applied to the ’th layer. Then
where ranges over style layers. Gatys et al. use Relu1_1, Relu2_1, Relu3_1, Relu4_1, and Relu5_1 as style layers, and layer Relu4_2 for the content loss, and search for using L-BFGS . Notation: From now on, we write R51 for Relu5_1, etc.
4 The cross layer gram matrix
Now consider layer and , both style layers, with decreasing spatial resolution. Write for an upsampling of to , and consider
as the cross-layer gram matrix, We can form a style loss
(where is a set of pairs of style layers). We can substitute this loss into the original style loss, and minimize as before. This construction has a variety of interesting properties which we will investigate later.
Style layer pairs: In principle, any set of pairs can be used. We have investigated a pairwise descending strategy, where one constrains each layer and its successor (i.e. (R51, R41); (R41, R31); etc) and an all distinct pairs strategy, where one constrains all pairs of distinct layers.
Pattern management across scales: Controlling within-layer gram matrices by proper weighting ensures that the statistics of patterns at a particular scale are “appropriate”. However, we speculate – and our experimental results seem to confirm – that one can get these statistics right without having desirable weighting relations across scales. Inter-layer gram matrices require that phenomena at one scale are correlated to those at the next scale appropriately. In other words, carefully controlling weights for each layer’s style loss is not necessary in cross-layer gram matrix scenario.
Number of constraints: Cross-layer gram matrices control considerably fewer parameters than within layer gram matrices. For a pairwise descending strategy, we have four cross-layer gram matrices, leading to control of parameters; compare within layer gram matrices, which control parameters. It may seem that there is less constraint on style. Experiment suggests our method produces visible improved results, meaning that many of the parameters controlled by within-layer gram matrices have no particular effect on the outcome.
4.1 Fast Universal Cross-layer Transfers
Li et al. use signal whitening and coloring to implement a fast version of style transfer using a VGG encoder . Their procedure takes the R51 layer from the content image, then applies an affine transformation (by whitening, coloring, and matching means) to match the gram matrix of the corresponding layer computed for the style image. The resulting layer is decoded to an image through one of five pre-trained image reconstruction decoder networks. The R41 layer produced by this image is again affine transformed to match the gram matrix of the corresponding layer computed for the style image. This layer is then again decoded to an image. The process continues until the affine transformed R11 layer is decoded to an image, which is retained.
This procedure is easily extended to cross-layer gram matrices (Figure 5). We start by choosing sequence of sets of layer covariances to control. The simple scheme is individual, controlling (R51), (R41), (R31), (R21), (R11). An alternative scheme is pairwise descending, where one controls (R51, R41); (R41, R31); (R31, R21); and (R21, R11). Another scheme is descending, where one controls (R51, R41, R31, R21, R11); (R41, R31, R21, R11); (R31, R21, R11); etc. we then start the first set of relevant layers from the content image (i.e. (R51, R41) for pairwise; (R51, … R11) for descending). Construct a gram matrix from this set of layers, upsampling as required. Apply an affine transformation to match the gram matrix of the corresponding set of layers for the style image, then decode the resulting layers to an image. Pass this image through VGG, recover the next set of control layers from the result, apply an affine transformation to match the gram matrix of the corresponding set of layers for the style image, then decode the resulting layers to an image. Proceed until the R11 layer is decoded to an image, and use that image. Note that this approach controls both within layer and between layer statistics, as the relevant gram matrices have within layer gram matrices as diagonal blocks, and between layer gram matrices as off-diagonal blocks.
4.2 Loss multiplication
Style transfer methods require a choice of parameter, , to balance the style and content losses. The value is typically chosen by eye, which is unsatisfying. A natural alternative to adding the losses is to multiply them; in this case, no parameter is needed, and we can form
Multiplicative loss tends to emphasize strong boundaries in the content image( Figure 6). We believe this is because style loss is always large, so that minimization will force down large differences (which are large difference in values between stylized image and content image) in the content layer. Our experimental results suggest that this approach is successful (section 5.3). The effect is quite prominent (Figure 6, Figure 11, Figure 12 ), multiplicative loss has significant advantage of reducing the number of parameters that need to be searched over to produce useful results. Figure 2 shows style transfer results using cross-layer gram matrices and multiplicative loss, we observe distinguishable improvement over Gatys’ method in preserving content boundaries.
We find one trick to improve transfer results using multiplicative loss by shifting the mean when creating the new image to optimized, we recommend this shift should be the channel mean of style image.
5.1 Experimental details
We use VGG-19 for both style transfer and texture synthesis. We use R11, R21, R31, R41, and R51 for style(texture) loss, and R42 for the content loss for style transfer. In loss optimization, if it not specified, all stylized images start from Gaussian noise image and optimized with LBFGS.
5.2 Texture synthesis
Cross-layer gram matrix control applies to texture synthesis since style loss  was first introduced as ”texture” loss in . We now omit the content loss, and seeks a minimum of style loss alone. We show texture synthesis results, which highlights the method’s ability to manage long spatial correlations. We controlled R51, R41, R31, R21, R11 for comparison with our style transfer results. Our synthesis starts from an image which has the mean color of the texture image. As Figure 7 shows, synthesized textures have better long scale coherence. For the universal texture synthesis, we followed Li et al. as starting from zero-mean Gaussian noise, run the multi level pipeline 3 times for better results.
5.3 Style transfer
Cross-layer vs. within-layer style loss: Figures 3, 4 compare style transfers using within-layer gram matrices and cross-layer gram matrices with a pairwise descending strategy. Cross-layer gram matrices are particularly good at preserving relations between effects, as the detail in figure 4 shows.
Multiplicative loss: The multiplicative loss often produces visual pleasing style transfer results by showing better style pattern arrangement at same time keeping the outline of content relatively intact, so that the generated image preserves the perceptual meaning of the content while showing coherent style patterns(Figure 6); More examples are present in supplementary materials.
Pairwise descending vs all pairs distinct: All pairs distinct cross-layer style transfer seems to produce improvements over pairwise descending (Figure 9). This is in some contrast to Novak and Nikulin’s findings (, p5), which suggest “tying distant … layers produces poor results”.
FCT vs WCT: Fast universal cross-layer transfer (FCT) works visually better than the original WCT method of Li et al. , as Figure 8 shows. However, FCT has some of the same difficulties that WCT has. Both methods have difficulty reproducing crisp subshapes in styles.
Individual style loss control: When one controls style loss using a single layer (or a single pair of layers). We can clearly see how they effect stylized images (Figure 10). We observe higher level style loss shows stronger control over long scale patterns from style images, this is in agreement with similar observations in . We also found that cross-layer gram matrices have stronger ability in preserving prominent boundaries of content images while display equal or better control over long scale style patterns compared to the same level within-layer gram matrices.
Scales: A crop of the style image will effectively result in transferring larger style elements. We expect that, when style elements are large compared to the content, cross-layer methods will have a strong advantage because they will be better able to preserve structural relations that make up style elements. Qualitative evidence supports this view (Figure 11 and Figure 12).
Cross-layer gram matrix creates summary statistics that captures the correlation between different layers; higher layers can guide lower layers the most likely location for feature activations through the spacial product of forming cross-layer gram matrix. Therefore, we expect cross-layer gram matrices performs better especially on long scale patterns. Our experiments prove this point. The cross-layer gram matrix has less constraint but better style control than within-layer gram matrix.
Fast Universal Cross-layer Style Transfer successfully unifies the Universal style transfer with our inter-layer statistics, and indeed shows some intrinsic difference in both style transfer and texture synthesis.
Multiplicative style loss
not only simplifies the style weight searching by eliminating one hyperparameter, but also emphasizes the boundary of content object even when strong boundaries information is present in style summary statistics. It provides better style quality in terms of preserving content shape and keeping long style coherence. More examples are present in the supplementary materials.
We give our deep thanks to Mao-Chuang’s supervisor, Professor David Forsyth who provides many helpful suggestions in research and helps us writing the paper. We also thank Jason Rock for his help on setting up faster style transfer and other related recommendations. Besides, Anand Bhattad also gives us a lot of useful suggestions about style transfer. We really thank his kind help.
-  J. D. Bonet. Multiresolution sampling procedure for analysis and synthesis of texture images. SIGGRAPH, 1997.
-  A. J. Champandard. Semantic style transfer and turning two-bit doodles into fine artworks. arXiv preprint arXiv:1603.01768, 2016.
-  D. Chen, L. Yuan, J. Liao, N. Yu, and G. Hua. Stylebank: An explicit representation for neural image style transfer. , July 2017.
-  T. Q. Chen and M. Schmidt. Fast patch-based style transfer of arbitrary style. arXiv preprint arXiv:1612.04337, 2016.
-  V. Dumoulin, J. Shlens, and M. Kudlur. A learned representation for artistic style. ICLR, 2017.
-  A. A. Efros and W. T. Freeman. Image quilting for texture synthesis and transfer. Proceedings of the 28th annual conference on Computer graphics and interactive techniques - SIGGRAPH ’01, pages 341–346, 2001.
L. Gatys, A. S. Ecker, and M. Bethge.
Texture synthesis using convolutional neural networks.In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 262–270. Curran Associates, Inc., 2015.
-  L. A. Gatys, A. S. Ecker, and M. Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2414–2423, 2016.
-  L. A. Gatys, A. S. Ecker, M. Bethge, A. Hertzmann, and E. Shechtman. Controlling perceptual factors in neural style transfer. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
-  A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, and D. H. Salesin. Image analogies. Proceedings of the 28th annual conference on Computer graphics and interactive techniques - SIGGRAPH ’01, (August):327–340, 2001.
-  X. Huang and S. Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. arXiv preprint arXiv:1703.06868, 2017.
-  Y. Jing, Y. Yang, Z. Feng, J. Ye, and M. Song. Neural style transfer: A review. arXiv preprint arXiv:1705.04058, 2017.
J. Johnson, A. Alahi, and L. Fei-Fei.
Perceptual losses for real-time style transfer and super-resolution.In European Conference on Computer Vision, 2016.
-  Y. Li, C. Fang, J. Yang, Z. Wang, X. Lu, and M.-H. Yang. Universal style transfer via feature transforms. arXiv preprint arXiv:1705.08086.
-  Y. Li, N. Wang, J. Liu, and X. Hou. Demystifying neural style transfer. arXiv preprint arXiv:1701.01036, 2017.
-  D. C. Liu and J. Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical programming, 45(1):503–528, 1989.
-  F. Luan, S. Paris, E. Shechtman, and K. Bala. Deep Photo Style Transfer. 2017.
-  R. Novak and Y. Nikulin. Improving the neural algorithm of artistic style. arXiv preprint arXiv:1605.04603, 2016.
-  Y. Shih, S. Paris, C. Barnes, W. T. Freeman, and F. Durand. Style transfer for headshot portraits. ACM Transactions on Graphics, 33(4):1–14, 2014.
-  E. P. Simoncelli and J. Portilla. Texture characterization via joint statistics of wavelet coefficient magnitudes. In ICIP, 1998.
-  J. B. Tenenbaum and W. T. Freeman. Separating Style and Content with Bilinear Models. Neural Computation, 12(6):1247–1283, 2000.
-  X. Wang, G. Oxholm, D. Zhang, and Y.-F. Wang. Multimodal transfer: A hierarchical deep convolutional neural network for fast artistic style transfer. arXiv preprint arXiv:1612.01895, 2016.