PyMatting: A Python Library for Alpha Matting

03/25/2020 ∙ by Thomas Germer, et al. ∙ 0

An important step of many image editing tasks is to extract specific objects from an image in order to place them in a scene of a movie or compose them onto another background. Alpha matting describes the problem of separating the objects in the foreground from the background of an image given only a rough sketch. We introduce the PyMatting package for Python which implements various approaches to solve the alpha matting problem. Our toolbox is also able to extract the foreground of an image given the alpha matte. The implementation aims to be computationally efficient and easy to use. The source code of PyMatting is available under an open-source license at https://github.com/pymatting/pymatting.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 2

Code Repositories

pymatting

A Python library for alpha matting


view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

For an image with foreground pixels and background pixels , alpha matting asks to determine opacities , such that the equality

(1)

holds for every pixel

. This problem is ill-posed since, for each pixel, we have three equations (one for each color channel) with seven unknown variables. The implemented methods rely on a trimap, which is a rough classification of the input image into foreground, background and unknown pixels, to further constrain the problem. Giving the alpha matte, foreground estimation aims to extract the foreground

from image .

2 Implemented Methods for Alpha Estimation

Closed-form Matting:

Levin et al. (2007) show that assuming local smoothness of pixel colors yields a closed-form solution to the alpha matting problem.

KNN Matting:

Lee and Wu (2011) and Chen et al. (2013) use nearest neighbor information to derive closed-form solutions to the alpha matting problem which they note to perform particularly well on sparse trimaps.

Large Kernel Matting:

He et al. (2010) propose an efficient algorithm based on a large kernel matting Laplacian. They show that the computational complexity of their method is independent of the kernel size.

Random Walk Matting:

Grady et al. (2005)

use random walks on the pixels to estimate alpha. The calculated alpha of a pixel is the probability that a random walk starting from that pixel will reach a foreground pixel before encountering a background pixel.

Learning Based Digital Matting:

Zheng and Kambhamettu (2009)

estimate alpha using local semi-supervised learning. They assume that the alpha value of a pixel can be learned by a linear combination of the neighboring pixels.

3 Implemented Methods for Foreground Estimation

Closed-form Foreground Estimation:

For given , the foreground pixels can be determined by making additional smoothness assumptions on and . Our toolbox implements the foreground estimation by Levin et al. (2007).

Multi-level Foreground Estimation:

Furthermore, the PyMatting toolbox implements a novel multi-level approach for foreground estimation. For this method our toolbox also provides GPU implementations for OpenCL and CUDA.

4 Installation and Code Example

(a) Input

(b) Input trimap

(c) Estimated

(d) Estimated
Figure 1: Example output of the PyMatting library. Image from Rhemann et al. (2009).

The following code snippet demonstrates the usage of the library:

from pymatting import *

image =  load_image( " plant_ image. png" ,  " RGB" )

trimap =  load_image( " plant_ trimap. png" ,  " GRAY" )

alpha =  estimate_alpha_cf( image,  trimap)

foreground =  estimate_foreground_cf( image,  alpha)

cutout =  stack_images( foreground,  alpha)

save_image( " results. png" ,  cutout)

The estimate_alpha_cf method implements closed-form alpha estimation (Levin et al., 2007), whereas the estimate_foreground_cf method implements the closed-form foreground estimation (Levin et al., 2007). The method stack_images can be used to compose the foreground onto a new background.

More code examples at different levels of abstraction can be found in the documentation of the toolbox. PyMatting can be easily installed through pip.

5 Performance Comparison

Figure 2: Mean squared error of the estimated alpha matte to the ground truth alpha matte.

Since all of the considered methods require to solve a large sparse systems of linear equations, an efficient solver is crucial for good performance. Therefore, the PyMatting package implements the conjugate gradients method (Hestenes et al., 1952) together with different preconditioners that improve convergence: Jacobi, V-cycle (Lee and Wu, 2014) and thresholded incomplete Cholesky decomposition (Jones and Plassmann, 1995).

To evaluate the performance of our implementation, we calculate the mean squared error on the unknown pixels of the benchmark images of Rhemann et al. (2009). Figure 2 shows the mean squared error to the ground truth alpha matte. Our results are consistent with the results achieved by the authors implementation (if available).

(a) Peak memory usage for each solver in MB
(b) Mean running time of each solver in seconds
Figure 3: Comparison of computational time and peak memory usage of our implementation of the preconditioned CG method with other solvers for closed-form matting.
Figure 4: Comparison of runtime for different image sizes.

We compare the computational runtime of our solver with other solvers: PyAMG (Olson and Schroder, 2018), UMFPACK (Davis, 2004), AMGCL (Demidov, 2019), MUMPS (Amestoy et al., 2001, 2006), Eigen (Guennebaud et al., 2010) and SuperLU (Li et al., 1999). Figure 4 shows that our implemented conjugate gradients method in combination with the incomplete Cholesky decomposition preconditioner outperforms the other methods by a large margin. For the iterative solver, we use an absolute tolerance of , which we scale with the number of known pixels, i.e. pixels that are either marked as foreground or background in the trimap. The benchmarked linear system arises from the matting Laplacian by Levin et al. (2007). Figure 3 shows that our solver also outperforms the other solvers in terms of memory usage.

6 Compatibility and Extendability

The PyMatting package has been tested on Windows 10, Ubuntu 16.04 and macOS 10.15.2. The package can be easily extended by adding new definitions of the graph Laplacian matrix. We plan on continuously extending our toolbox with new methods.

References

  • P. R. Amestoy, I. S. Duff, J. Koster, and J.-Y. L’Excellent (2001) A fully asynchronous multifrontal solver using distributed dynamic scheduling. SIAM Journal on Matrix Analysis and Applications 23 (1), pp. 15–41. Cited by: §5.
  • P. R. Amestoy, A. Guermouche, J.-Y. L’Excellent, and S. Pralet (2006) Hybrid scheduling for the parallel solution of linear systems. Parallel Computing 32 (2), pp. 136–156. Cited by: §5.
  • Q. Chen, D. Li, and C. Tang (2013) KNN matting. IEEE transactions on pattern analysis and machine intelligence 35 (9), pp. 2175–2188. Cited by: §2.
  • T. A. Davis (2004) Algorithm 832: UMFPACK v4.3—an unsymmetric-pattern multifrontal method. ACM Transactions on Mathematical Software (TOMS) 30 (2), pp. 196–199. Cited by: §5.
  • D. Demidov (2019) AMGCL: an efficient, flexible, and extensible algebraic multigrid implementation. Lobachevskii Journal of Mathematics 40 (5), pp. 535–546. External Links: ISSN 1818-9962, Document, Link Cited by: §5.
  • L. Grady, T. Schiwietz, S. Aharon, and R. Westermann (2005) Random walks for interactive alpha-matting. In Proceedings of VIIP, Vol. 2005, pp. 423–429. Cited by: §2.
  • G. Guennebaud, B. Jacob, et al. (2010) Eigen v3. Note: http://eigen.tuxfamily.org Cited by: §5.
  • K. He, J. Sun, and X. Tang (2010) Fast matting using large kernel matting laplacian matrices. In

    2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition

    ,
    pp. 2165–2172. Cited by: §2.
  • M. R. Hestenes, E. Stiefel, et al. (1952) Methods of conjugate gradients for solving linear systems. Journal of research of the National Bureau of Standards 49 (6), pp. 409–436. Cited by: §5.
  • M. T. Jones and P. E. Plassmann (1995) An improved incomplete cholesky factorization. ACM Transactions on Mathematical Software (TOMS) 21 (1), pp. 5–17. Cited by: §5.
  • P. G. Lee and Y. Wu (2014) Scalable matting: a sub-linear approach. arXiv preprint arXiv:1404.3933. Cited by: §5.
  • P. Lee and Y. Wu (2011) Nonlocal matting. In CVPR 2011, pp. 2193–2200. Cited by: §2.
  • A. Levin, D. Lischinski, and Y. Weiss (2007) A closed-form solution to natural image matting. IEEE transactions on pattern analysis and machine intelligence 30 (2), pp. 228–242. Cited by: §2, §3, §4, §5.
  • X. S. Li, J. W. Demmel, J. R. Gilbert, L. Grigori, M. Shao, and I. Yamazaki (1999) SuperLU users’ guide. Lawrence Berkeley National Laboratory. Cited by: §5.
  • L. N. Olson and J. B. Schroder (2018) PyAMG: algebraic multigrid solvers in Python v4.0. Note: Release 4.0 External Links: Link Cited by: §5.
  • C. Rhemann, C. Rother, J. Wang, M. Gelautz, P. Kohli, and P. Rott (2009) A perceptually motivated online benchmark for image matting. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1826–1833. Cited by: Figure 1, §5.
  • Y. Zheng and C. Kambhamettu (2009) Learning based digital matting. In 2009 IEEE 12th international conference on computer vision, pp. 889–896. Cited by: §2.