Imagine being able to easily create a matte — the per-pixel color and alpha — of a person by taking photos or videos in an everyday setting with just a handheld smartphone. Today, the best methods for extracting (“pulling”) a good quality matte require either a green screen studio, or the manual creation of a trimap (foreground/background/unknown segmentation), a painstaking process that often requires careful painting around strands of hair. Methods that require neither of these are beginning to appear, but they are not of comparable quality. Instead, we propose taking an additional photo of the (static) background just before or after the subject is in frame, and using this photo to perform background matting
. Taking one extra photo in the moment requires a small amount of foresight, but the effort is tiny compared to creating a trimap after the fact. This advantage is even greater for video input. Now, the world is your green screen.
We focus on a method that is tuned to human subjects. Still, even in this setting — pulling the matte of a person given a photo of the background — the problem is ill-posed and requires novel solutions.
Consider the compositing equation for image given foreground , background , and mixing coefficient : For color images and scalar , and given , we have four unknowns ( and ), but only three observations per pixel (). Thus, the background matting problem is underconstrained. Background/foreground differences provide a signal, but the signal is poor when parts of the person are similar in color to the background. Furthermore, we do not generally have an image of the ideal background: the subject can cast shadows and cause reflections not seen in the photo taken without the subject, and exact, pixel-level alignment with no resampling artifacts between handheld capture of two photos is generally not attainable. In effect, rather than the true that produced , we have some perturbed version of it, . Finally, we can build on person segmentation algorithms to make the problem more tractable to identify what is semantically the foreground. However current methods, exhibit failures for complex body poses and fine features like hair and fingers.
Given these challenges and recently published successes in solving matting problems, a deep learning approach is a natural solution. We propose a deep network that estimates the foreground and alpha from input comprised of the original image, the background photo, and an automatically computed soft segmentation of the person in frame. The network can also utilize several frames of video, useful for bursts or performance capture, when available. However, the majority of our results, including all comparisons to single-image methods, do not use any temporal cues.
We initially train our network on the Adobe Matting dataset , comprised of ground truth mattes that can be synthetically composited over a variety of backgrounds. In practice, we found the domain gap between these synthetic composites and real-world images did not lead to good results using standard networks. We partially close this gap in two ways: by augmenting the dataset and by devising a new network — a “Context Switching Block” — that more effectively selects among the input cues. The resulting mattes for real images can still have significant artifacts, particularly evident when compositing onto a new background. We thus additionally train the network in a self-supervised manner on real unlabelled input images using an adversarial loss to judge newly created composites and ultimately improve the matting process.
Our method has some limitations. First, we do require two images. Trimap-based methods arguably require two images as well for best results – the trimap itself is a hand-made second image – though they can be applied to any input photo. Second, we require a static background and small camera motion; our method would not perform well on backgrounds with people walking through or with a camera that moves far from the background capture position. Finally, our approach is specialized to foregrounds of (one or more) people. That said, person matting without big camera movement in front of a static background is, we argue, a very useful and not uncommon scenario, and we deliver state-of-the-art results under these circumstances.
Our contributions include: The first trimap-free automatic matting algorithm that utilizes a casually captured background. A novel matting architecture (Context Switching Block) to select among input cues. A self-supervised adversarial training to improve mattes on real images. Experimental comparisons to a variety of competing methods on wide range of inputs (handheld, fixed-camera, indoor, outdoor), demonstrating the relative success of our approach. Our code and data is available at http://github.com/senguptaumd/Background-Matting.
2 Related Work
Matting is a standard technique used in photo editing and visual effects. In an uncontrolled setting, this is known as the “natural image matting” problem; pulling the matte requires solving for seven unknowns per pixel () and is typically solved with the aid of a trimap. In a studio, the subject is photographed in front of a uniformly lit, constant-colored background (e.g., a green screen); reasonable results are attainable if the subject avoids wearing colors that are similar to the background. We take a middle ground in our work: we casually shoot the subject in a natural (non-studio) setting, but include an image of the background without the subject to make the matting problem more tractable. In this section, we discuss related work on natural image matting, captured without unusual hardware.
Traditional approaches. Traditional (non-learning based) matting approaches generally require a trimap as input. They can be roughly categorized into sampling-based techniques and propagation-based techniques. Sampling-based methods [11, 9, 14, 29, 33, 34, 2] use sampling to build the color statistics of the known foreground and background, and then solve for the matte in the ‘unknown’ region. Propagation-based approaches [6, 18, 20, 21, 31, 13, 15] aim to propagate the alpha matte from the foreground and the background region into the ‘unknown’ region to solve the matting equation. Wang and Cohen  presents a nice survey of many different matting techniques.
Deep learning approaches showed renewed success in natural image matting, especially in presence of user-generated trimaps. Some methods combine learning-based approaches with traditional techniques, e.g., KNN-matting[30, 7]. Xu et al.  created a matting dataset with real mattes and composited over a variety of backgrounds and trained a deep network to predict the alpha matte; these results were further improved by Lutz et al.  using an adversarial loss. Recently Tang et al.  proposed a hybrid of a sampling-based approach and learning to predict the alpha matte. Lu et.al  proposed a new index-guided upsampling and unpooling operation that helps the network predict better alpha mattes. Cai et al.  showed robustness to faulty user-defined trimaps. All of these methods only predict the alpha matte and not the foreground, leaving open the (non-trivial) problem of recovering foreground color needed for composites. Recently Hou et al.  introduced Context-Aware Matting (CAM) which simultaneously predicts the alpha and the foreground, thus solving the complete matting problem, but is not robust to faulty trimaps. In contrast to these methods (and the traditional approaches), our work jointly predicts alpha and foreground using an image of the background instead of a trimap.
Recently, researchers have developed algorithms that perform matting without a trimap, focusing mostly on humans (as we do). Aksoy et.al.  introduced fully automatic semantic soft segmentation for natural images. In [38, 30] the authors perform portrait matting without trimap, utilizing segmentation cues. Trimap-free matting has also been extended to handle whole bodies in [37, 5]. These methods aim to perform trimap prediction, followed by alpha prediction. Our work is also human-focused; we compare our approach with the recent state-of-the-art automatic human matting algorithm  and obtain significantly better performance with the aid of the background image.
Matting with known natural background. Difference matting proposed by Qian and Sezan  attempts to solve matting with a natural background by simple background subtraction and thresholding but is very sensitive to the threshold and produces binary mattes. Similarly, change detection via background subtraction [25, 10] generally does not produce alpha mattes with foreground and considers shadows to be part of the foreground. Some traditional approaches like Bayesian matting  and Poisson matting [31, 12] can handle known background in their framework, but additionally require trimaps.
Video Matting. Researchers have also focused on video-specific methods. Chuang et.al.  extended Bayesian Matting to videos by utilizing the known background and optical flow, requiring trimaps for keyframes. Flow-based temporal smoothing can be used [19, 28] (again with trimaps) to encourage temporal coherence.
3 Our Approach
The input to our system is an image or video of a person in front of a static, natural background, plus an image of just the background. The imaging process is easy, just requiring the user to step out of the frame after the shot to capture the background, and works with any camera with a setting to lock the exposure and focus (e.g., a smartphone camera). For handheld capture, we assume camera motion is small and align the background to a given input image with a homography. From the input, we also extract a soft segmentation of the subject. For video input, we can additionally utilize nearby frames to aid in matting.
At the core of our approach is a deep matting network that extracts foreground color and alpha for a given input frame, augmented with background, soft segmentation, and (optionally nearby video frames), and a discriminator network that guides the training to generate realistic results. In Section 3.1, we describe the matting network, which contains a novel architecture – a “Context-switching block” – that can combine different input cues selectively. We first train a copy of this network with supervision using the Adobe Matting Dataset . We use known foreground and alpha mattes of non-transparent objects, which are then composited over a variety of backgrounds (i.e., real source images, but synthetic composites). Our matting network, along with some data augmentation, help overcome some of the domain gap between the synthetically composited imagery and real data that we later capture with a consumer camera (e.g., a smartphone).
In Section 3.2, we describe a self-supervised scheme to bridge the domain gap further and to generally improve the matting quality. The method employs an adversarial network comprised of a separate copy of the deep matting network, , that tries to produce a matte similar to the output of and a discriminator network that scores the result of compositing onto a novel background as real or fake. We train and jointly on real inputs, with supervision provided by (the now fixed) network applied to the same data.
3.1 Supervised Training on the Adobe Dataset
Here we describe our deep matting network, which we first train on the Adobe Matting Dataset, restricted to the subset of non-transparent objects. The network takes as input an image with a person in the foreground, an image of the background registered to (as noted earlier, is not the same as the true with subject present), a soft segmentation of the person , and (optionally for video) a stack of temporally nearby frames , and produces as output a foreground image and alpha matte . To generate , we apply person segmentation  and then erode (5 steps), dilate (10 steps), and apply a Gaussian blur (). When video is available, we set to be the concatenation of the two frames before and after , i.e., for frame interval ; these images are converted to grayscale to ignore color cues and focus more on motion cues. In the absence of video, we simply set to , also converted to grayscale. We denote the input set as . The network with weight parameters thus computes:
In designing and training the network, the domain gap between the Adobe dataset and our real data has proven to be a significant driver in our choices as we describe below.
A natural choice for would be a residual-block-based encoder-decoder  operating on a concatenation of the inputs . Though we would expect such a network to learn which cues to trust at each pixel when recovering the matte, we found that such a network did not perform well. When training on the Adobe synthetic-composite data and then testing on real data, the resulting network tended to make errors like trusting the background too much and generating holes whenever was too close in color; the network was not able to bridge the domain gap.
Instead, we propose a new Context Switching block (CS block) network (Figure 2) to combine features more effectively from all cues, conditioned on the input image. When, e.g., a portion of the person matches the background, the network should focus more on segmentation cue in that region. The network has four different encoders for , , , and that separately produce 256 channels of feature maps for each. It then combines the image features from with each of , and
separately by applying 1x1 convolution, BatchNorm, and ReLU (‘Selector’ block in Fig.2), producing 64-channel features for each of the three pairs. Finally, these three 64-channel features are combined with the original 256-channel image features with 1x1 convolution, BatchNorm, and ReLU (the ‘Combinator’ block in Fig. 2) to produce encoded features which are passed on to the rest of the network, consisting of residual blocks and decoders. We observe that the CS Block architecture helps to generalize from the synthetic-composite Adobe dataset to real data (Figure 4). More network architecture details are provided in the supplementary material.
We train the network with the Adobe Matting dataset  which provides 450 ground truth foreground image and alpha matte (manually extracted from natural images). We select the subset of 280 images corresponding to non-transparent objects (omitting, e.g., objects made of glass). As in , we can compose these foregrounds over known backgrounds drawn from the MS-COCO dataset, augmented with random crops of varying resolutions, re-scalings, and horizontal flips. These known backgrounds would not be the same as captured backgrounds in a real setting. Rather than carefully simulate how and might differ, we simply perturbed to avoid training the network to rely too much on its exact values. In particular, we generated each by randomly applying either a small gamma correction to or adding gaussian noise around the foreground region. Further, to simulate imperfect segmentation guidance we threshold the alpha matte and then erode (10-20 steps), dilate (15-30 steps) and blur () the result. For the motion cue , we applied random affine transformations to foreground+alpha before compositing onto the background, followed by conversion to grayscale. To compute and we used the compositing equation with as the background, but we provided as the input background to the network.
Finally, we train our network on the Adobe dataset with supervised loss:
where , and the gradient term on encourages sharper alpha mattes .
3.2 Adversarial Training on Unlabelled Real data
Although our proposed Context Switch block (CS block) combined with data augmentation significantly helps in bridging the gap between real images and synthetic composites created with the Adobe dataset, it still fails to handle all difficulties present in real data. Theses difficulties include (1) traces of background around fingers, arms, and hairs being copied into the matte; (2) segmentation failing; (3) significant parts of the foreground color matching the background color; (4) misalignment between the image and the background (we assume only small misalignment). To handle these cases, we aim to learn from unlabelled, real data (real images + backgrounds) with self-supervision.
The key insight is that significant errors in the estimated matte typically result in unrealistic composites over novel backgrounds. For example, a bad matte might contain a chunk of the source background, which, when composited over a new background, will have a piece of the original background copied over the new background, a major visual artifact. Thus, we can train an adversarial discriminator to distinguish between fake composites and (already captured) real images to improve the matting network.
The matting network () and discriminator network can be trained end-to-end based on just a standard discriminator loss. However, could settle on setting everywhere, which would result in simply copying the entire input image into the composite passed to . This solution is “optimal” for , since the input image is indeed real and should fool . Initializing with and fine-tuning with a low learning rate (was necessary for stable training with a discriminator) is not very effective. It does not allow significant changes to network weights needed to generate good mattes on real data.
Instead, we use for teacher-student learning. In particular, for a real training image and associated inputs comprising , we obtain to serve as “pseudo ground-truth”. We can now train with an adversarial loss and a loss on the output of the matting network when compared to “pseudo ground-truth”, following 
; this second loss is given small weight which is reduced between epochs during training. Though we initializein the standard randomized way, the network is still encouraged to stay similar to the behavior of while having the flexibility to make significant changes that improve the quality of the mattes. We hypothesize that this formulation helps the network to avoid getting stuck in the local minimum of , instead finding a better minimum nearby for real data.
We use the LS-GAN  framework to train our generator and discriminator . For the generator update we minimize:
where , is a given background for generating a composite seen by , and we set to 0.05 and reduce by every two epochs during training to allow the discriminator to play a significant role. We use a higher weight on the alpha losses (relative to Equation 2), especially the gradient term to encourage sharpness.
For the discriminator, we minimize:
where represents the weights of the discriminator network and again .
As a post-process, we threshold the matte at , extract the largest connected components, and set for pixels not in those components, where is the number of disjoint person segmentations in the image.
4 Experimental Evaluation
We compared our approach with a variety of alternative methods, esp. recent deep matting algorithms that have performed well on benchmarks: BM: Bayesian Matting  - traditional, trimap-based method that can accept a known background . (An alternative, Poisson Matting [31, 12] with known background, performed much worse.). CAM: Context-Aware Matting  - trimap-based deep matting technique that predicts both alpha and foreground. IM: Index Matting  - trimap-based deep matting technique that predicts only alpha. LFM: Late Fusion Matting  - trimap-free deep matting algorithm that predicts only alpha.
4.1 Results on Synthetic-Composite Adobe Dataset
We train on 26.9k exemplars: 269 objects composited over 100 random backgrounds, plus perturbed versions of the backgrounds as input to the network. We train with batch-size 4, learning rate with Adam optimizer.
We compare results across 220 synthetic composites from the Adobe Dataset : 11 held-out mattes of human subjects composed over 20 random backgrounds, in Table 1. We computed a trimap for each matte through a process of alpha matte thresholding and dilation as described in . We dilated by 10 and 20 steps to generate two different trimaps (more steps gives wider unknown region). We additionally computed a perturbed background by applying small random affine transformation (translate , rotate and small scaling and shear) followed by gamma correction and gaussian noise . For our approach, we only evaluated the result of applying the network (‘Ours-Adobe’), since it was trained only on the Adobe data, as were the other learning-based approaches we compare to. We rescaled all images to and measure the SAD and MSE error between the estimated and ground truth (GT) alpha mattes, supplying algorithms with the two different trimaps and with backgrounds and as needed. We omitted LFM from this comparison, as the released model was trained on all of the Adobe data, including the test data used here (confirmed by the authors). That said, it produces a SAD and MSE of 2.00, 1.08, resp., while our method achieves (true test) error of 1.72, 0.97.
We observe that our approach is more robust to background perturbation when compared to BM, and it improves on all other trimap-based matting algorithms (BM, CAM, IM). As trimaps get tighter, the trimap-based matting algorithms get better, but tight trimaps are time-consuming to create in practice. The goal of our work is to fully eliminate the need for manually created trimaps.
4.2 Results on Real Data
We captured a mix of handheld and fixed-camera videos, taken indoors and outside using a smartphone (iPhone 8). The fixed-camera setup consisted of an inexpensive selfie stick tripod. In each case, we took a video with the subject moving around, plus a shot of the background (single video frame) with no subject. All frames were captured in HD (19201080), after which they were cropped to 512512 (input resolution to our network) around the segmentation mask for one person or multiple. We retrain on 280k composites consisting of 280 objects from Adobe Dataset . We then train separate copies of , one each on handheld videos and fixed camera videos, to allow the networks to focus better on the input style. For handheld videos we account for small camera shake by aligning the captured background to individuals frames through homography. In total, we trained on 18k frames for hand-held camera and 19k frames for fixed camera. We captured 3390 additional background frames for . We use a batch-size of 8, learning rate of for and for and update with Adam optimizer. We also update the weights of after 5 successive updates of .
|Ours vs.||much better||better||similar||worse||much worse|
|Ours vs.||much better||better||similar||worse||much worse|
To compare algorithms on real data, we used 10 handheld videos and 10 fixed-camera videos as our (held-out) test data. The BM, CAM, and IM methods each require trimaps. We did not manually create trimaps (esp. for video sequences which is infeasible). Instead, we applied segmentation 
, and labeled each pixel with person-class probabilityas foreground, as background, and the rest as unknown. We tried alternative methods, including background subtraction, but they did not work as well.
To evaluate results, we could not compare numerically to ground truth mattes, as none were available for our data. Instead, we composited the mattes over a green background and performed a user study on the resulting videos. Since IM and LFM do not estimate (needed for compositing), we set for these methods. We also tried estimating directly from the matting equation (given and ), but the results were worse (see supplementary material). We do not use any temporal information and set for all comparisons to prior methods.
In the user study, we compared the composite videos produced by network (‘Ours-Real’) head-to-head with each of the competing algorithms. Each user was presented with a web page showing the original video, our composite, and a competing composite; the order of the last two was random. The user was then asked to rate composite A relative to B on a scale of 1-5 (1 being ‘much worse’, 5 ‘much better’). Each video pair was rated 10 users.
The results of the user study, with scores aggregated over all test videos, are shown in Tables 2 and 3. Overall, our method significantly outperformed the alternatives. The gains of our method are somewhat higher for fixed-camera results; with handheld results, registration errors can still lead to matting errors due to, e.g., parallax in non-planar background scenes (see Fig 6(f)).
Single image results are shown in Figure 6, again demonstrating improvement of our method over alternatives. We note that LFM in particular has difficulty zeroing in on the person. More results generated by our approach with handheld camera in natural backgrounds are shown in Figure 3. In (c), (d) we show examples of multiple people interacting in a single image, and in (e) we show a failure case with a dynamic background, the fountain. Please see supplementary material for video results and more image results.
5 Ablation Studies
Role of motion cues. As shown in Figure 5, video motion cues can help in predicting a cleaner matte when foreground color matches the background. (Note: we did not use motion cues when comparing to other methods, regardless of input source.)
|much better||better||similar||worse||much worse|
‘Ours-Real’ vs ‘Ours-Adobe’. As expected, ‘Ours-Adobe’ outperformed ‘Ours-Real’ on the synthetic-composite Adobe dataset on which ‘Ours-Adobe’ was trained. ‘Ours-Real’ achieved a SAD score of 3.50 in comparison to 1.73 of ‘Ours-Adobe’. However ‘Ours-Real’ significantly outperformed ‘Ours-Adobe’ on real data as shown by qualitative examples in Figure 6 and by an additional user study (Table 4). The gain of ‘Ours-Real’ in the user study ( 10 users per pair-wise comparison) was larger for handheld captures; we suspect this is because it was trained with examples having alignment errors. (We did try training ‘Ours-Adobe’ with alignment errors introduced into but found the results degraded overall.)
Role of Context Switching Block (CS Block). We compare our CS Block architecture to a standard residual-block-based encoder-decoder  scheme that was run on a naive concatenation of , , , and . We find that the concatenation-based network learns to focus too much on color difference between and and generates holes when their colors are similar. The CS Block architecture effectively utilizes both segmentation and color difference cues, along with motion cues when present, to produce better matte, as shown in Figure 4 (more in supplementary). Empirically, we observe that the CS block helps significantly in 9 out of 50 real videos, especially when foreground color is similar to the background.
We have proposed a background matting technique that enables casual capture of high quality foreground+alpha mattes in natural settings. Our method requires the photographer to take a shot with a (human) subject and without, not moving much between shots. This approach avoids using a green screen or painstakingly constructing a detailed trimap as typically needed for high matting quality. A key challenge is the absence of real ground truth data for the background matting problem. We have developed a deep learning framework trained on synthetic-composite data and then adapted to real data using an adversarial network.
Acknowledgements. This work was supported by NSF/Intel Visual and Experimental Computing Award #1538618 and the UW Reality Lab.
-  Yağiz Aksoy, Tae-Hyun Oh, Sylvain Paris, Marc Pollefeys, and Wojciech Matusik. Semantic soft segmentation. ACM Transactions on Graphics (TOG), 37(4):72, 2018.
-  Yagiz Aksoy, Tunc Ozan Aydin, and Marc Pollefeys. Designing effective inter-pixel information flow for natural image matting. In , pages 29–37, 2017.
-  Shaofan Cai, Xiaoshuai Zhang, Haoqiang Fan, Haibin Huang, Jiangyu Liu, Jiaming Liu, Jiaying Liu, Jue Wang, and Jian Sun. Disentangled image matting. International Conference on Computer Vision (ICCV), 2019.
-  Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018.
-  Quan Chen, Tiezheng Ge, Yanyu Xu, Zhiqiang Zhang, Xinxin Yang, and Kun Gai. Semantic human matting. In 2018 ACM Multimedia Conference on Multimedia Conference, pages 618–626. ACM, 2018.
-  Qifeng Chen, Dingzeyu Li, and Chi-Keung Tang. Knn matting. IEEE transactions on pattern analysis and machine intelligence, 35(9):2175–2188, 2013.
Donghyeon Cho, Yu-Wing Tai, and Inso Kweon.
Natural image matting using deep convolutional neural networks.In European Conference on Computer Vision, pages 626–643. Springer, 2016.
-  Yung-Yu Chuang, Aseem Agarwala, Brian Curless, David H Salesin, and Richard Szeliski. Video matting of complex scenes. In ACM Transactions on Graphics (ToG), volume 21, pages 243–248. ACM, 2002.
-  Yung-Yu Chuang, Brian Curless, David H Salesin, and Richard Szeliski. A bayesian approach to digital matting. In CVPR (2), pages 264–271, 2001.
-  Ahmed Elgammal, David Harwood, and Larry Davis. Non-parametric model for background subtraction. In European conference on computer vision, pages 751–767. Springer, 2000.
-  Eduardo SL Gastal and Manuel M Oliveira. Shared sampling for real-time alpha matting. In Computer Graphics Forum, volume 29, pages 575–584. Wiley Online Library, 2010.
-  Minglun Gong and Yee-Hong Yang. Near-real-time image matting with known background. In 2009 Canadian Conference on Computer and Robot Vision, pages 81–87. IEEE, 2009.
-  Leo Grady, Thomas Schiwietz, Shmuel Aharon, and Rüdiger Westermann. Random walks for interactive alpha-matting. In Proceedings of VIIP, volume 2005, pages 423–429, 2005.
-  Kaiming He, Christoph Rhemann, Carsten Rother, Xiaoou Tang, and Jian Sun. A global sampling method for alpha matting. In CVPR 2011, pages 2049–2056. IEEE, 2011.
-  Kaiming He, Jian Sun, and Xiaoou Tang. Fast matting using large kernel matting laplacian matrices. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 2165–2172. IEEE, 2010.
-  Qiqi Hou and Feng Liu. Context-aware image matting for simultaneous foreground and alpha estimation. International Conference on Computer Vision (ICCV), 2019.
-  Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017.
-  Philip Lee and Ying Wu. Nonlocal matting. In CVPR 2011, pages 2193–2200. IEEE, 2011.
-  Sun-Young Lee, Jong-Chul Yoon, and In-Kwon Lee. Temporally coherent video matting. Graphical Models, 72(3):25–33, 2010.
-  Anat Levin, Dani Lischinski, and Yair Weiss. A closed-form solution to natural image matting. IEEE transactions on pattern analysis and machine intelligence, 30(2):228–242, 2007.
-  Anat Levin, Alex Rav-Acha, and Dani Lischinski. Spectral matting. IEEE transactions on pattern analysis and machine intelligence, 30(10):1699–1712, 2008.
-  Hao Lu, Yutong Dai, Chunhua Shen, and Songcen Xu. Indices matter: Learning to index for deep image matting. International Conference on Computer Vision (ICCV), 2019.
-  Sebastian Lutz, Konstantinos Amplianitis, and Aljosa Smolic. Alphagan: Generative adversarial networks for natural image matting. arXiv preprint arXiv:1807.10088, 2018.
-  Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 2794–2802, 2017.
-  Massimo Piccardi. Background subtraction techniques: a review. In 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No. 04CH37583), volume 4, pages 3099–3104. IEEE, 2004.
-  Richard J Qian and M Ibrahim Sezan. Video background replacement without a blue screen. In Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348), volume 4, pages 143–146. IEEE, 1999.
-  Soumyadip Sengupta, Angjoo Kanazawa, Carlos D. Castillo, and David W. Jacobs. Sfsnet: Learning shape, refectance and illuminance of faces in the wild. In Computer Vision and Pattern Regognition (CVPR), 2018.
-  Ehsan Shahrian, Brian Price, Scott Cohen, and Deepu Rajan. Temporally coherent and spatially accurate video matting. In Computer Graphics Forum, volume 33, pages 381–390. Wiley Online Library, 2014.
-  Ehsan Shahrian, Deepu Rajan, Brian Price, and Scott Cohen. Improving image matting using comprehensive sampling sets. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 636–643, 2013.
-  Xiaoyong Shen, Xin Tao, Hongyun Gao, Chao Zhou, and Jiaya Jia. Deep automatic portrait matting. In European Conference on Computer Vision, pages 92–107. Springer, 2016.
-  Jian Sun, Jiaya Jia, Chi-Keung Tang, and Heung-Yeung Shum. Poisson matting. In ACM Transactions on Graphics (ToG), volume 23, pages 315–321. ACM, 2004.
-  Jingwei Tang, Yagiz Aksoy, Cengiz Oztireli, Markus Gross, and Tunc Ozan Aydin. Learning-based sampling for natural image matting. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
-  Jue Wang and Michael F Cohen. An iterative optimization approach for unified image segmentation and matting. In Tenth IEEE International Conference on Computer Vision (ICCV’05) Volume 1, volume 2, pages 936–943. Citeseer, 2005.
-  Jue Wang and Michael F Cohen. Optimized color sampling for robust matting. In 2007 IEEE Conference on Computer Vision and Pattern Recognition, pages 1–8. IEEE, 2007.
-  Jue Wang, Michael F Cohen, et al. Image and video matting: a survey. Foundations and Trends® in Computer Graphics and Vision, 3(2):97–175, 2008.
-  Ning Xu, Brian Price, Scott Cohen, and Thomas Huang. Deep image matting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2970–2979, 2017.
-  Yunke Zhang, Lixue Gong, Lubin Fan, Peiran Ren, Qixing Huang, Hujun Bao, and Weiwei Xu. A late fusion cnn for digital matting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7469–7478, 2019.
-  Bingke Zhu, Yingying Chen, Jinqiao Wang, Si Liu, Bo Zhang, and Ming Tang. Fast deep matting for portrait animation on mobile phone. In Proceedings of the 25th ACM international conference on Multimedia, pages 297–305. ACM, 2017.
-  Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223–2232, 2017.
Appendix A Overview
We provide additional details and results in this Appendix. In Sec. B we describe the details of our network architecture. In Sec. C.1 we clarify our choice of automatic trimap generation, especially to show why background subtraction is not good enough for this problem. In Sec. C.2, we show why algorithms that do not predict foreground introduce additional artifacts in compositing. In Sec. D, we provide additional qualitative examples for ablation studies. Specifically, we show the role of Content Switching Block in Sec. D.1, motion cue in Sec. D.2, self-supervised adversarial training on real data in Sec. D.3 and robustness w.r.t. segmentation and background in Sec. D.4.
Appendix B Network Architectures
Our proposed matting network , shown again for reference in Figure 2 (same as Figure 2 in the main paper) consists of the Context Switching Block (CS Block) followed by residual blocks (ResBLKs) and decoders to predict alpha matte and foreground layer . Below we describe the network architecture in details. Most of our Generator architecture, especially residual blocks and decoders, and Discriminator architecture are based on that of .
‘Image Encoder’ and ‘Prior Encoder’ (CS Block): C64(k7) - C*128(k3) - C*256(k3)
‘CN(kS)’ denotes convolution layers with N
‘Selector’ (CS Block): C64(k1)
‘CN(kS)’ denotes convolution layers with N filters with stride 1, followed by Batch Normalization and ReLU. The ‘Selector’ block takes as input the concatenation of image feature and a prior feature as a blob of spatial resolution . The output of the ‘Selector’ network is a blob of spatial resolution . The goal of the ‘Selector’ block is to generate prior features conditioned on the image. This will help the network to generalize from synthetic-composite dataset (on which it was trained) to real images by not over-relying on one kind of features (e.g. color difference with background).
‘Combinator’ (CS Block): C256(k1)
‘CN(kS)’ denotes convolution layers with N filters with stride 1, followed by Batch Normalization and ReLU. The ‘Combinator’ block takes as input the concatenation of image features of spatial resolution , along with 3 other prior features from the ‘Selector’ network of spatial resolution each. Thus the input to the ‘Combinator’ block is of spatial resolution and the output is of spatial resolution . The ‘Combinator’ block learns to combine the individual priors features with the original image feature for the goal of improving matting.
‘ResBLKs’: K ResBLK
The output of the combinator is first passed through ‘ResBLK’s and then provided as input to 2 separate ‘ResBLK’s of for alpha matte and foreground layer separately. All ‘ResBLK’s operate at a spatial resolution of . Each ‘ResBLK’ consists of Conv256(k3) - BN -ReLU - Conv256(k3) - BN, where ‘BN’ denote Batch Normalization.
‘Decoder’ for alpha matte : CU*128(k3)-CU*64(k3)-Co1(k7)-Tanh
The input to the ‘Decoder’ is of resolution .‘CU*N(kS)’ denotes bilinear up-sampling by factor of 2, followed by convolution layer with filters with stride 1, Batch Normalization and ReLU. The last layer Co3k(7) consists of only convolution layers of 1 filters, followed by Tanh layer. This scales the output alpha between .
‘Decoder’ for foreground : CU*128(k3)-CU*64(k3)-Co1(k7)
The input to the ‘Decoder’ is of resolution .‘CU*N(kS)’ denotes bilinear up-sampling by factor of 2, followed by convolution layer with filters with stride 1, Batch Normalization and ReLU. There is also a skip connection from the image input features of resolution which is combined with the output of CU*128(k3) and passed on to CU*64(k3). The last layer Co3k(7) consists of only convolution layers of 1 filters.
Discriminator : C*64(k4) - C*I128(k4) - C*I256(k4) - C*I512(k4)
We use 70 × 70 PatchGAN . ‘C*N(kS)’ denotes convolution layers with N filters with stride 2, followed by leaky ReLUs of slope 0.2. ‘I’ denotes the presense of Instance Norm before leaky ReLU, in all layers except the first one. After the last layer a convolution filter of kernel is applied to produce a 1 dimensional output. The PatchGAN is applied over the whole composite image by convolving with every patch to determine if it is real or fake.
Appendix C Experimental Evaluation
c.1 Automatic Trimap Generation
We compare our method with algorithms that require user defined trimaps (CAM, IM, BM). It is extremely time consuming to annotate trimaps for every frame of a video, or even for a bunch of keyframes whose trimaps then need to be propagated to the remaining frames and then touched up. As described in the paper, we instead created trimaps automatically by applying segmentation , and labelling each pixel with person-class probability as foreground, as background, and the rest as unknown. We tried, and rejected, alternative methods, including background subtraction and erosion-dilateion of the segmentation mask, which we now describe and illustrate here for completeness.
Background subtraction is popularly used for change detection, but is extremely sensitive to color differences and produces only a binary matte. Thus it is not in itself a suitable candidate for matting. However background subtraction could in principle be used to generate a trimap. In our experiments, we observed that the best thresholds varied from image to image and even then produced mediocre results (see hand-tuned example in Figure 8). We also tried erosion-dilation of the segmentation mask (erode 5 steps, dilate 15 steps) to try to produce a fixed width ‘unknown’ band but we often ended up with mattes that pulled in the background as a part of the foreground. Figure 8 shows examples of trimaps and resulting alpha mattes using erosion-dilation, hand-tuned thresholding of background subtraction, and our probability-thresholded ‘Automatic Trimap’ method.
c.2 Predicting Foreground layer
To produce composites – the primary reason for extracting a matte in the first place – we require both and . Since IM and LFM do not estimate , we are still left with the task of estimating it ourselves. Why is this non-trivial? From the matting equation () after estimating only , we can say that observed pixel color must be of the way along a line segment from to . Clearly, there is an infinite family of ’s and ’s that satisfy this constraint; thus, given just and , we cannot readily infer . Our seemingly naive solution is to set for these methods. We also tried estimating directly from the matting equation given – i.e., when , with clamped so that each color channel is in – but the results were worse than , largely due to discrepancies between and , particularly in the handheld camera case, where small misalignments can arise. We show a comparison of these two options for IM matting, plus a comparison to our result, in Figure 7. The figure shows that matting-equation based estimation of pulls some of the background texture into the matte. The solution is better, but picks up some of the background colors (a bit of green and yellow in this case), since it is just copying foreground-background mixed pixels and blending them over another background. Our method, which estimates directly, shows fewer artifacts. Our matte is a bit softer, but it captures structure like the curls on the top of the head; further, some of the apparent sharpness of the other composites comes from copying over too much of the original image rather (which has detail) rather than fully separating from the background.
c.3 Results on Real Data
After conducting the user study, we realized that we had given a slight advantage to our method by retaining only the largest connected component for our background matting approach but not for the competing approaches. We noted that the CAM and IM methods had small “floaters” after matting, so applied the connected component removal to these videos and re-ran the study. We did not observe that BM had floaters in our examples; any that appeared to remain, were actually connected to the largest component by “bridges” of small . LFM was more problematic. We found that LFM would at times pull in pieces of the background that were larger than the foreground person; the result of retaining the largest connected component would then mean losing the foreground subject altogether, an extremely objectionable artifact. Rather than continuing to refine the post-process for LFM, we simply did not apply a post-process for its results. As seen in the videos, LFM, in any case, had quite a few other artifacts that made it not competitive with the others.
Table 5 and 6 shows the result of the updated user study. We observe that the results are similar to the ones reported in the main paper (i.e., Tables 2 and 3 in the main paper); the excess connected components for CAM and IM did not have a significant impact relative to other errors in matte estimation.
|Ours vs.||much better||better||similar||worse||much worse|
|Ours vs.||much better||better||similar||worse||much worse|
Appendix D Ablation Study
In this section, we provide additional details and more results for the ablation studies already presented in the main paper. Specifically, we analyze (i) Role of Context Switching Block (ii) Role of motion cues and (iii) Compare ‘Ours-Real’ to ‘Ours-Adobe’.
d.1 Role of Context Switching Block
Here we go in more depth on the paper’s CS block ablation study, again showing that the CS Block network is largely effective in utilizing all cues and in generalizing better from the synthetic-composite Adobe dataset  to real world images. To this end we train with CS Block on the Adobe dataset which we term as ‘Ours-Adobe’ (with Context Switching). Additionally, we construct another network (No Context Switching), where we remove the CS block. The input to this network is the concatenation of the input image, background, soft segmentation, and motion cues , which is passes through the ‘Image Encoder’ architecture to produce a dimensional feature. Since there is no CS Block, we directly pass this feature to the ResBLKs and then continue through the same architecture presented in Figure 2. We also train this network on the Adobe dataset, following the exact protocol of training ‘Ours-Adobe’.
We then test both and on our real video dataset. Note that for this experiment we use the motion cue for both and . We captured videos at 60fps and set frames.
In Figure 9 of this document and in Figure 4 of the main paper, we show multiple examples from different videos where a part of the foreground person matches the background color. (‘No Context Switching’) fails in these situations, this is because while training on the synthetic-composite dataset it learns to focus too much on the color differences to perform matting and fails when colors coincide. On the other hand (‘With Context Switching’) handles color coincidences better by utilizing the soft segmentation cues provided in the input. Thus the CS Block learns to properly utilize the cues at hand, when compared to ‘No Context Switching’, which focuses more on the color differences to produce a matte. Note that, the holes produced by ‘No Context Switching’ as shown in Figure 9 appears in multiple frames of that video where the color coincides significantly; we show only 1 sample from all of these frames.
d.2 Role of Motion Cues
Here we provide more detail and examples on the paper’s ablation study for motion cues. When the input is video, we have the option of setting the motion cue to , where T=20 for a 60fps video. We train another network , by removing the motion cue input and its related ‘Prior Encoder’ and ‘Selector’ from the architecture presented in Figure 2. Thus is the same as , but without the motion cue block. We train both and on the synthetic-composite Adobe dataset, following the same training protocol as described in the paper, for both networks. We then test both and on our real video dataset.
In Figure 10 of this document and in Figure 5 of the main paper, we show the role motion cue plays in improving the alpha matte estimation. Specifically, the motion cue helps when foreground color matches the background, and when the foreground is moving significantly. Due to the foreground motion, the network can utilize additional frames (4 in this case) to determine that the regions which move are more likely to be foreground than the background, even though the color matches with the background. Note that, this may not be always true, e.g. a shadow cast on the background also moves with the foreground. Small camera motion with a handheld camera can also effectively cause motion in the background due to misregistration. Additionally, since we consider only a small time window of 1.33 secs for the motion cue, often there is lack of information as the foreground appears to be almost static during that time.
To reiterate: whenever comparing to competing methods, we set the motion cue to and treat all images (including video frames) independently.
d.3 ‘Ours-Real’ vs ‘Ours-Adobe’
Here we show more comparisons between using just the network for matting (‘Ours-Adobe’) and using the full network guided by and discriminator (‘Ours-Real’). In the main paper, we present an ablation study comparing ‘Ours-Real’ with ‘Ours-Adobe’ as both a user study in Table 3 and with qualitatively comparisons in Figure 6. Additional visual comparison on our test videos are presented in our project webpage. In Figure 11 of this document, we provide additional qualitative comparison between ‘Ours-Real’ and ‘Ours-Adobe’. We find that ‘Ours-Real’ is generally better though on occasion it is not; (k) and (l) are instances where ‘Ours-Real’ produces an inferior matte compared to ‘Ours-Adobe’.
d.4 Role of background and segmentation
The captured background image without the subject and the estimated soft segmentation map are two key additional inputs that helps in estimating the foreground and alpha matte. We found that omitting the background image from the baseline model degrades results substantially: SAD error of 8.33 without background vs. 1.73 with background on synthetic-composite Adobe dataset. For backgrounds that are relatively distant or roughly planar, homography-based alignment is accurate, and the network learns to handle remaining mis-registrations by training on hand-held videos. Alignment fails when the background, e.g., has two planes (Fig 4f, with two orthogonal walls).
Soft segmentation is obtained by eroding and dilating the segmentation predicted by Deeplabv3+. We observe that eroding and dilating the segmentation by 20 steps only increase the SAD error from 1.73 to 1.76 on synthetic-composite Adobe dataset. Hence our method is quite robust to errors in segmentation, and soft segmentation only indicates which is the foreground subject in the image. In contrast, the captured background plays more crucial role in the performance of the method.