Chan-Vese Reformulation for Selective Image Segmentation

Selective segmentation involves incorporating user input to partition an image into foreground and background, by discriminating between objects of a similar type. Typically, such methods involve introducing additional constraints to generic segmentation approaches. However, we show that this is often inconsistent with respect to common assumptions about the image. The proposed method introduces a new fitting term that is more useful in practice than the Chan-Vese framework. In particular, the idea is to define a term that allows for the background to consist of multiple regions of inhomogeneity. We provide comparitive experimental results to alternative approaches to demonstrate the advantages of the proposed method, broadening the possible application of these methods.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 11

page 12

page 13

page 14

page 15

page 16

page 18

page 20

02/05/2015

Fast Constraint Propagation for Image Segmentation

This paper presents a novel selective constraint propagation method for ...
06/03/2019

Computing Valid p-values for Image Segmentation by Selective Inference

Image segmentation is one of the most fundamental tasks of computer visi...
01/15/2015

Screen Content Image Segmentation Using Least Absolute Deviation Fitting

We propose an algorithm for separating the foreground (mainly text and l...
07/30/2018

A Restricted-Domain Dual Formulation for Two-Phase Image Segmentation

In two-phase image segmentation, convex relaxation has allowed global mi...
01/08/2022

A Baseline Statistical Method For Robust User-Assisted Multiple Segmentation

Recently, several image segmentation methods that welcome and leverage d...
03/11/2021

PointFlow: Flowing Semantics Through Points for Aerial Image Segmentation

Aerial Image Segmentation is a particular semantic segmentation problem ...
11/21/2020

Contextual Interference Reduction by Selective Fine-Tuning of Neural Networks

Feature disentanglement of the foreground target objects and the backgro...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Image segmentation is an important application of image processing techniques in which some, or all, objects in an image are isolated from the background. In other words, for an image , we find the partitioning of the image domain into subregions of interest. In the case of two-phase approaches this consists of the foreground domain and background domain , such that . In this work we concentrate on approaching this problem with variational methods, particularly in cases where user input is incorporated. Specifically, we consider the convex relaxation approach of Chan:06 ; Bresson:07 and many others. This consists of a binary labelling problem where the aim is to compute a function indicating regions belonging to and , respectively. This is obtained by imposing a relaxed constraint on the function, , and minimising a functional that fits the solution to the data with certain conditions on the regularity of the boundary of the foreground regions.

(a) (i) Image with ground truth
(b) (ii) Foreground,
(c) (iii) Background,
Figure 1: CT image with ground truth segmentation shown (green) and associated average intensity values ( and ).

We will first introduce the seminal work of Chan and Vese ACWE , a segmentation model that uses the level set framework of Osher and Sethian Osher:88 . This approach assumes that the image is approximately piecewise-constant, but is dependent on the initialisation of the level set function as the minimisation problem is nonconvex. The Chan-Vese model was reformulated to avoid this by Chan et al. Chan:06 , using convex relaxation methods, that has the following data fitting functional

(1)

where and are data fitting terms indicating the foreground and background regions, respectively. In particular, in ACWE and Chan:06 these are given by

(2)

It should be noted that it is common to fix . The data fitting functional is balanced against a regularisation term. Typically, this penalises the length of the contour. This is represented by the total variation (TV) of the function ACWE ; Rudin:92 , and is sometimes weighted by an edge detection function Bresson:07 ; Perona:90 ; Geo ; CDSS . Therefore, the regularisation term is given as

(3)

The convex segmentation problem, assuming fixed constants and , is then defined by

(4)

In the case where the intensity constants are unknown it is also possible to minimise alternately with respect to , and , however, this would make the problem non-convex and hence dependent on the initialisation of . Functionals of this type have been widely studied with respect to two-phase segmentation Bresson:07 ; Chan:06 ; ACWE , which is our main interest. Alternative choices of data fitting terms can be used when different assumptions are made on the image, . Examples include Ali:16 ; Ali:17 ; VMS ; RSF ; SBF ; LCV . We note that multiphase approaches Brox:06 ; VeseChan:02 are also closely related to this formulation although in this paper we focus on the two-phase problem due to associated applications of interest. It is also important to acknowledge analogous methods in the discrete setting such as Bai:07 ; Falcao:02 ; RW ; Grabcut . However, we do not go into detail about such methods here, although we introduce the work of SRW in §3 and compare corresponding results in §7.

In selective segmentation the idea is to apply additional constraints such that user input is incorporated to isolate specific objects of interest. It is common for the user to input marker points to form a set , where and from this we can form a foreground region whose interior points are inside the object to be segmented. In the case that is provided will be a polygon, but any user-defined region in the foreground is consistent with the proposed method. Some examples of selective or interactive methods include Cai:13 ; SRW ; Gout:05 ; Liu:18 ; Nguyen:12 ; Geo ; RW ; LRW ; CDSS ; Zhang:10 . A particular application of this in medical imaging is organ contouring in computed tomography (CT) images. This is often done manually which can be laborious and inefficient and it is often not possible to enhance existing methods with training data. In cases where learning based methods are applicable, the work of Xu et al. Xu:16 and Bernard and Gygli Benard:17 are state of the art approaches. At this stage we define the additional constraints in selective segmentation as follows:

(5)

where is some distance penalty term, such as Rada:13 ; Geo ; CDSS , and is a selection parameter. Essentially, the idea is that the selection term (based on the region formed by the user input marker set) should penalise regions of the background (as defined by the data fitting term ) and also pixels far from . In this paper we choose to be the geodesic distance penalty proposed in Geo . Explicitly, the geodesic distance from the region formed from the marker set is given by:

where is the solution of the following PDE:

(6)

The function is image dependent and controls the rate of increase in the distance. It is defined as a function similar to

(7)

where is a small non-zero parameter and is a non-negative tuning parameter. We set the value of and throughout. Note that if then the distance penalty is simply the normalised Euclidean distance, as used in CDSS .

A general selective segmentation functional, assuming homogeneous target regions, is therefore given by:

(8)

Assuming that the optimal intensity constants and are fixed, the minimisation problem is then:

(9)

Again, it is possible to alternately minimise with respect to the constants and to obtain the average intensity in and , respectively. However, in selective segmentation it is often sufficient to fix these according to the user input. In the framework of (9) the Chan-Vese terms Chan:06 ; ACWE ; MumfordShah have limitations due to the dependence on . In conventional two-phase segmentation problems it makes sense to penalise deviances from outside the contour, however for selective segmentation we need not consider the intensities outside of the object we have segmented. Regardless of whether the intensity of regions outside the object is above or below , it should be penalised positively. The Chan-Vese terms cannot ensure this as they work based on a fixed ”exterior” intensity and can lead to negative penalties on regions which are outside the object of interest. It is our aim in this paper to address this problem.

The motivation for this work comes from observing contradictions in using piecewise-constant intensity fitting terms in selective segmentation. Whilst good results are possible with this approach, the exceptional cases lead to severe limitations in practice. This is quite common in medical imaging as demonstrated in Fig. 1, where the target foreground has a low intensity. Given that the corresponding background includes large regions of low intensity, the optimal average intensities for this segmentation problem are and . For cases where , we see that by (1), almost everywhere in the domain . This means that it is very difficult to achieve an adequate result, without an over-reliance on the user input or parameter selection.

The central premise for applying Chan-Vese type methods is the assumption that the image approximately consists of

(10)

where is noise,

is the characteristic function of the region

, for

respectively. The idea of selective segmentation is to incorporate user input to apply constraints that exclude regions classified as foreground, based on their location in the image. We use a distance constraint which penalises the distance from the user input markers. However, a key problem for selective segmentation is that for cases where the optimal intensity values

and are similar, the intensity fitting term will become obsolete as the contour evolves. This is illustrated in Fig. 3. The purpose of our approach is to construct a model that is based on assumptions that are consistent with the observed image and any homogeneous target region of interest. A common approach in selective segmentation is to discriminate between objects of a similar intensity Rada:13 ; Geo ; CDSS . However, the fitting terms in previous formulations Klodt:13 ; Rada:13 ; Geo ; CDSS aren’t applicable in many cases as there are contradictions in the formulation in this context. We will address this in detail in the following section.

In this paper our main contribution is to highlight a crucial flaw in the assumptions behind many current selective segmentation approaches and propose a new fitting term in relation to such methods. We demonstrate how our reformulation is capable of achieving superior results and is more robust to parameter choices than existing approaches, allowing for more consistency in practice. In §2 we give a brief review of alternative intensity fitting terms proposed in the literature, and detail them in relation to selective segmentation. We then briefly detail alternative selective segmentation approaches to compare our method against in §3. In §4 we introduce the proposed model, focussing on a fitting term that allows for significant intensity variation in the background domain. In §5 we discuss the implementation of each approach in a convex relaxation framework, provide the algorithm in §6, and detail some experimental results in §7. Finally, in §8 we give some concluding remarks.

2 Related Approaches

Here, we introduce and discuss work that has introduced alternative data fitting terms closely related to Chan-Vese ACWE . In order to make direct comparisons, we convert each approach to the unified framework of convex relaxation Chan:06 . It is worth noting that this alternative implementation is equivalent in some respects, but that the results might differ slightly if using the original methods. We are considering these models in the terms of selective segmentation, so all formulations have the following structure:

(11)

We are interested in the effectiveness of in this context, which we will focus on next. In particular, we detail various choices of from the literature that are generalisations of the Chan-Vese approach. In the following we refer to minimisers of convex formulations, such as (11), by . Here, the minimiser of is thresholded for in a conventional way Chan:06 .

2.1 Region-Scalable Fitting (RSF) Rsf

The data fitting term from the work of Li et al. RSF , known as Region-Scalable Fitting (RSF), consistent with the convex relaxation technique of Chan:06 is given by

(12)

where

(13)

and is chosen as a Gaussian kernel with scale parameter . The RSF selective formulation is then given as follows:

(14)

The functions and , which are generalisations of and from Chan-Vese, are updated iteratively by

(15)

Using the RSF fitting term, any deviations of from and are smoothed by the convolution operator, . This allows for intensity inhomogeneity in the foreground and background of target objects.

2.2 Local Chan-Vese (LCV) Fitting Lcv

Wang et al. LCV proposed the Local Chan-Vese (LCV) model. In terms of the equivalent convex formulation, the data fitting term is given by

(16)

where

(17)

and . Here, is an averaging convolution with window. The LCV selective formulation is then given as

(18)

The values which minimise this functional for are given by

(19)

The formulation is minimised iteratively. The LCV fitting term that and includes an additional term weighted by the parameters and . The principle for the LCV model is that the difference image is a higher contrast image than and a two-phase segmentation on this image can be computed.

2.3 Hybrid (HYB) Fitting Ali:16

Based on extending the LCV model, Ali et al. Ali:16 proposed the following data fitting term,

(20)

where

(21)

Here, , , and , with the averaging convolution as used in the LCV model. The values are updated in a similar way to LCV , with further details found in Ali:16 . The authors refer to this approach as the Hybrid (HYB) Model. The HYB selective formulation is then given as

(22)

The key aim of the HYB model is to account for intensity inhomogeneity in the foreground and background of the image through the product image . In LCV, the presence of the blurred image in the data fitting term deals with intensity inhomogeneity, whilst including helps identify contrast between regions. The authors found that the product image can improve the data fitting in both respects. Therefore they construct a LCV-type function with rather than the original . Their results suggest that this approach is more robust.

2.4 Generalised Averages (GAV) Fitting Ali:17

Recently, Ali et al. Ali:17 proposed using the data fitting terms of Chan-Vese in a signed pressure force function framework Zhang:10 . They refer to this approach as Generalised Averages (GAV) as they update the intensity constants in an alternative way, detailed below. In the convex framework, we consider the selective GAV functional:

(23)

where . This is identical to the CV selective formulation (8). However, the authors propose an alternative update for the fitting constants and , given as follows:

(24)

with . If , the approach is identical to CV. In Ali:17 the authors assert that the proposed adjustments have the following properties. As , and approach the maximum and minimum intensity in the foreground and background of the image, respectively. Also, as , and approach the minimum intensity in the foreground and background of the image, respectively. For example, if a high value of is set, will take a larger value than in CV which can be useful for selective segmentation. For example, if we consider the image in Fig. 1 we can achieve a larger value by setting and a smaller value by setting . Therefore, there is more flexibility when using this data fitting term in selective formulations. However, it should be noted that it involves the selection of the parameter , which can be difficult to optimise.

3 Alternative Selective Segmentation Models

We now introduce two recent methods that incorporate user input to perform selective segmentation. Each involves input in the form of foreground/background regions to indicate relevant structures of interest. An example of this can be seen in Fig. 18, where red regions indicate foreground and blue regions indicate background. We compare against the work of Nguyen et al. Nguyen:12 , which uses a similar convex relaxation framework to the proposed approach, and Dong et al. SRW , which uses a variation of the random walk approach. We summarise the essential aspects of each approach in the following.

3.1 Constrained Active Contours (CAC) Nguyen:12

The authors use a probability map,

, from Bai and Sapiro Bai:07 where the geodesic distances to the foreground/background regions are denoted by and , respectively. An approximation of the probability that a point belongs to the foreground is then given by

(25)

Foreground/background Gaussian mixture models (GMM) are estimated from the user input. The terms

and denote the probability that a point, , belongs to the the foreground and background, respectively. The normalised log likelihood for each is then given by

(26)

GMMs are widely used in selective segmentation Falcao:02 ; Grabcut ; Bai:07 ; RW ; SRW and the authors in Nguyen:12 incorporate this idea into the framework we consider with the following data fitting term:

(27)

for a weighting parameter . It is proposed that is selected automatically as follows:

(28)

where is the total number of pixels in the image. Defining as the function applied to the image and applied to the GMM probability map , an enhanced edge function is defined as

(29)

for a weighting parameter , which can be set automatically in a similar way to (28). Thus, Nguyen et al. Nguyen:12 define the Constrained Active Contours (CAC) Model as

(30)

They obtain a solution using the split Bregman method of Goldstein et al. Goldstein:10 , although other methods are applicable and will yield similar results. However, that is not the focus of this paper so we omit the details here. In the results section, §7, we will compare our method against CAC to see our data fitting term compares against a GMM-based approach.

3.2 Submarkov Random Walks (SRW) Srw

We now introduce a recent selective segmentation method by Dong et al. SRW known as Submarkov Random Walks (SRW). Rather than using the continuous framework of Chan:06 , this approach is based in the discrete setting where each pixel in the image is treated as a node in a weighted graph. Random walks (RW) have been widely used for segmentation since the work of Grady RW . SRW is capable of achieving impressive results with user-defined foreground and background regions. The selective segmentation result can be obtained by assigning a label to each pixel based on the computed probabilities of the random walk approach. For brevity, we do not provide the full details of the method here, however, further details can be found in SRW . We compare SRW to our proposed approach on a CT data set in §7.4.

We now introduce essential notation to understand the approach of SRW . In RW an image is formulated as a weighted undirected graph with nodes and edges . Each node represents an image pixel . An edge connects two nodes and and a weight of edge measures the likelihood that a random walker will cross this edge:

(31)

where and are pixel intensities, with . In SRW a user indicates foreground/background regions in a similar way to CAC, as shown in Fig. 18, and can be viewed as a traditional random walker with added auxiliary nodes. In SRW , these are defined as a set of labelled nodes . A set of labels is defined, , with the number of labels , and the number of seeds labelled . The prior is then constructed from the seeded nodes (defined by the user). Assuming a label has an intensity distribution (based on GMM learning), a set of auxiliary nodes is added into an expanded graph to define a graph with prior . Each prior node is connected with all nodes in and the weight, , of an edge between a prior node and a node is proportional to , the probability density belonging to at .

The authors define the probabilities of each node belonging to label as the average reaching probability, denoted . This term incorporates the auxillary nodes introduced above and is dependent on multiple variables and parameters, including (31). Further details can be found in SRW . The segmentation result is then found by solving the following discrete optimisation problem:

(32)

where represents the final label for each node. In other words, for a two-phase segmentation problem, is analogous to the discretised solution of a convex relaxation problem in the continuous setting. Comparisons in terms of accuracy can therefore be made directly, which we elaborate on further in §7. The authors also detail the optimisation procedure and aspects of dealing with noise reduction.

4 Proposed Model

(a) (i)
(b) (ii)
(c) (iii)
Figure 2: Three 1D plots of whilst varying and (with ).

In this section we introduce the proposed data fitting term for selective segmentation. We consider objects that are approximately homogeneous in the target region. Intrinsically, it is then assumed that the region , provided by the user, is likely to provide a reasonable approximation of the optimal value and therefore an appropriate foreground fitting function, , is given by CV (2). For this reason, it makes sense to retain this term in the proposed approach. The contradiction is in how the background fitting function is defined. Considering piecewise-constant assumptions of the image, and many of the related approaches, the background is expected to be defined by a single constant value, . If then everywhere, and therefore the fitting term can’t accurately separate background regions from the foreground. It is not practical to rely on to overcome this difficulty as it will produce an over-dependence on the choice of and . This is prohibitive in practice. An alternative function must therefore be defined which is compatible with and . Here, we define a new data fitting term that penalises background objects in such a way that avoids these problems by allowing intensity variation above and below the value . In order to design a new functional, we first look at the original CV background fitting function

It is clear that in an approximately piecewise-constant image this function will be small outside the target region (i.e. where the image takes values near ) and positive inside the target region. Our aim in a new fitting term is to mimic this in such a way that is consistent with selective segmentation, where regions with a ‘foreground intensity’ are forced to be in the background. It is beneficial to introduce two parameters, and , to enforce the penalty on regions of intensity in the range , i.e. enforce the penalty asymmetrically around . We propose the following function to achieve this:

(33)

This function takes its maximum value where and is for and . In Fig. 2 we provide a 1D representation of for various choices of and , with and . Here, it can be seen how the proposed data fitting term acts as a penalty in relation to a fixed constant . It is analogous to CV, whilst accounting for the idea of selective segmentation with a data fitting term. The main advantage of this term is that it replaces the dependence on in the formulation, which has no meaningful relation to the solution of a selective segmentation problem. Even when the foreground is relatively homogeneous, the background may have intensities of a similar value to which will cause difficulties in obtaining an accurate solution. We detail the proposed fitting term in the following section.

4.1 New Fitting Term

(a) (i) Image and
(b) (ii) CV fitting
(c) (iii) New fitting
Figure 3: An image with user input shown in red (). Here, we show the difference between the CV fitting function and the proposed approach. The target region is clearly defined by negative values in (iii).

We define the proposed data fitting functional as follows:

(34)

for and as defined in (33). This is consistent with respect to the intensities of the observed object and the concept of selective segmentation. In Fig. 3 we see the difference between CV and the proposed fitting terms for given user input on a CT image. For the CT image, the CV fitting terms are near 0 within the target region. This is despite there being a distinct homogeneous area with good contrast on the boundary. This illustrates the problem we are aiming to overcome. With the proposed fitting term this phenomenon should be avoided in cases like this. By defining as in (33) there is no contradiction if the foreground and background intensities of the target region are similar.

For images where we assume that the target foreground is approximately homogeneous, we have generally found that fixing according to the user input is preferable. We compute as the average intensity inside the region formed from the user input marker point set. We therefore propose to minimise the following functional with respect to , given a fixed :

(35)

where is the geodesic distance computed as described earlier using (6). The minimisation problem is given as

(36)

The model consists of weighted TV regularisation with a geodesic distance constraint as in Geo . However, alternative constraints are possible, such as Euclidean CDSS

, or moments

Klodt:13 . It is important to note that we have defined the model in a similar framework to the related approaches discussed previously. The main idea is to establish how the proposed fitting term, , performs compared to alternative methods. Next we describe how we determine the values of and in the function automatically. This is important in practice as it avoids any additional user input or parameter dependence to achieve an accurate result. In subsequent sections we provide details of how we obtain a solution for the proposed model.

4.2 Parameter Selection

(a) (i) Test Image 1
(b) (ii) Test Image 7
(c) (iii) Test Image 9
Figure 4: The histograms of intensities for some example images. The red lines are the automatic thresholds obtained by Otsu’s thresholding with .

For a particular problem it is quite straightforward to optimise the choice of and experimentally, but we would like a method which is not sensitive to the choice of and and would also prefer that the user need not choose these values manually. Therefore, in this section we explain how to choose these values automatically based on justifiable assumptions about general selective segmentation problems. To select the parameters and we use Otsu’s method Otsu:79 to divide the histogram of image intensities into

partitions. Otsu’s thresholding is an automatic clustering method which chooses optimal threshold values to minimise the intra-class variance. This has been implemented very efficiently in MATLAB in the function

multithresh for dividing a histogram such that there are thresholds .

We use the thresholds from Otsu’s method to find and as follows. There are three cases to consider, based on the value of computed from the user input: i) for some , ii) , iii) . For each case we set the parameters as follows:

Choosing too large could mean and are too small as the histogram would be partitioned too precisely. Generally we only ever need to consider a maximum of 3 phases for selective segmentation. If there is a large number of pixels in the image with intensity above or below the image can be considered two-phase in practice. Conversely, if a large number of pixels in the image have intensity above and below the image can essentially be considered three-phase in the context of selective segmentation. This is due to the way has been defined. Therefore, we set for all tests. In Fig. 4 we can see the Otsu thresholds chosen for various images given in this paper. They divide the peaks in the histogram well and once we know the value of (the approximation of the intensity of the object we would like to segment) we can automatically choose and according to this criteria.

5 Numerical Implementation

We now introduce the framework in which we compute a solution to the minimisation of the proposed model, as well the related models introduced in Sections 1 and 2. All consist of the minimisation problem

(37)

for respectively. Minimisation problems of this type (37) have been widely studied in terms of continuous optimisation in imaging, including two-phase segmentation. A summary of such methods in recent years is given by Chambolle and Pock CPintro . Our numerical scheme follows the original approach in Chan:06 : enforcing the constraint in (37) with a penalty function, and deriving the Euler-Lagrange of the regularised functional. We then solve the corresponding PDE by following a splitting scheme first applied to this kind of problem by Spencer and Chen CDSS . Whilst the numerical details are not the focus of the work, it is important to note widely used alternative methods. It has proved very effective to exploit the duality in the functional and avoid smoothing the TV term. A prominent example is the split Bregman approach for segmentation by Goldstein et al. Goldstein:10 . This is closely related to augmented lagrangian methods, a matter further discussed by Boyd et al. Boyd:11 . Analogous approaches also consist of the first-order primal dual algorithm of Chambolle and Pock ChambollePock and the max-flow/min-cut framework detailed by Yuan et al. Yuan:13 . There are practical advantages in implementing such a numerical scheme for our problem, primarily in terms of computational speed. However, in the numerical tests we include we’re mainly interested in accuracy comparisons. For this purpose the convex splitting algorithm of CDSS is sufficient, and the extension of splitting schemes for convex segmentation problems may be of interest. Further details can be found in CDSS and Geo . In the following, we first discuss the minimisation of (37) in a general sense and then mention some important aspects in relation to the alternative fitting terms discussed in §2.

5.1 Finding the Global Minimiser

To solve this constrained convex minimisation problem (38) we use the Additive Operator Splitting (AOS) scheme from Gordeziani et al. Gordeziani:74 , Lu et al. Tai:91 and Weickert et al. Weickert:98 . This is used extensively for image segmentation models Rada:13 ; Geo ; CDSS . It allows the 2D problem to be split into two 1D problems, each solved separately, with the results combined in an efficient manner. We address some aspects of AOS in §6, with further details provided in Geo ; CDSS .

A challenge with the functional (35), particularly with respect to AOS, is that this is a constrained minimisation problem. Consequently, it is reformulated by introducing an exact penalty function, , given in Chan:06 . To simplify the formulation we define

is the function associated with . We introduce a new parameter, , which allows us to balance the data fitting terms to the regularisation term more reliably. To be clear, we still only have two main tuning parameters ( and ) as we fix any variable parameters in according to the choices in the corresponding papers. The unconstrained minimisation problem is then given as:

(38)

We rescale the data term with . In effect this change is simply a rescaling of the parameters. This allows for the parameter choices between different models to be more consistent, as the fitting terms are similar in value. The problem (38) has the corresponding Euler-Lagrange equation (for fixed ):

(39)

in and where is the outward unit normal. The constraint is enforced for by Chan:06 . Two parameters, and , are introduced here. The former is to avoid singularities in the TV term and the latter is associated with the regularised penalty function from CDSS :

(40)

with and regularised Heaviside function

(41)

The viscosity solution of the parabolic formulation of (39), obtained by multiplying the PDE by , exists and is unique. The general proof for a class of PDEs to which (39) belongs, is included in Geo and we refer the reader there for the details. Once the solution to (39) is found, denoted , we define the computed foreground region as follows:

(42)

We select (although other values would yield a similar result according to Chan et al. Chan:06 ). In the following we use the binary form of the solution, , denoted . This partitions the domain into and according to the labelling function .

5.2 Implementation for Related Models

The discussion in this section so far has used the function associated with the data fitting functional . This corresponding equations for the RSF, LCV, HYB and GAV models are detailed in §2, CV is discussed in §1, and our approach is given by eqn. (34). We use this implementation to obtain selective segmentation versions of each of those models, given by (37). When these terms contain parameter choices we follow the advice in the corresponding papers as far as possible, unless we have found that alternatives will improve results. In the next section we will give the results of these models and compare them to our proposed approach.

Note. We now discuss details behind tuning parameters for the GAV model. It is noted in §2 that the GAV model requires a parameter to adapt the and calculation. We find that it is actually better to consider and separately to achieve improved results, as sometimes we wish to tune the values to have a higher and lower (or vice-versa) simultaneously. Therefore we introduce parameters and to tune and as follows:

(43)

In all experiments, we tested the following combinations of : , , , , , , and . For each choice, we optimised the values of and according to the procedure described in §7.1. This allowed us to select the optimal combination of for each image.

6 Algorithm

Here, we will discuss the algorithm that we use to minimise the selective segmentation model (37). We utilise additive operator splitting techniques to solve the minimisation problem efficiently.

6.1 An Additive Operator Splitting (AOS) Scheme

Additive Operator Splitting (AOS) Gordeziani:74 ; Tai:91 ; Weickert:98 is a widely used method for solving PDEs with linear and non-linear diffusion terms Rada:13 ; Geo ; CDSS such as

(44)

AOS allows us to split the two-dimensional problem into two one-dimensional problems, which we solve separately and then combine. Each one-dimensional problem gives rise to a tridiagonal system of equations which can be solved efficiently by Thomas’ algorithm, hence AOS is a very efficient method for solving PDEs of this type. AOS is a semi-implicit method and permits far larger time-steps than the corresponding explicit schemes would. Hence AOS is more stable than an explicit method Weickert:98 . Note here that

(45)

and . The standard AOS scheme assumes does not depend on , however in this instance that is not the case. This requires a modification to be used for convex segmentation problems, first introduced by CDSS . This non-standard formulation incorporates the regularised penalty term, , into the AOS scheme which we briefly detail next.

The authors consider the Taylor expansions of around and . They find that the coefficient of the linear term in is the same for both expansions. Therefore, for a change in of around and the change in can be approximated by . To address this, the relevant interval is defined as

and a corresponding update function is given as

The solution for (44) is then obtained by discretising the equation as follows:

where and are discrete forms of and
, respectively (given in CDSS ; Geo ). The modified AOS update is then given by

(46)

where and . This scheme allows for more control on the changes in between iterations due to the function and parameter , and therefore leads to a more stable convergence. We refer the reader to CDSS for full details of the numerical method.

6.2 The Proposed Algorithm

In Algorithm 1 we provide details of how we find the minimiser of the various selective segmentation models detailed above, defined by (37). The algorithm is in a general form to be applied to any of the approaches discussed so far. It is important to reiterate that alternative solvers to AOS are available, such as the dual formulation Aujol:06 ; Bresson:07 ; Chambolle:04 , split-Bregman Goldstein:10 , augmented Lagrange Bertsekas:14 , primal dual ChambollePock , and max-flow/min-cut Yuan:13 . In all experiments we use the tolerance of for the stopping criteria and set , and .

  Provide user input region and compute , according to (6).
  Define appropriately for the model (CV, RSF, LCV, HYB, GAV, or the proposed approach).
  Compute and .
  Initialise (arbitrary for ).
  while  do
     .
     Update according to the AOS iteration (46).
     .
  end while
  return   and binary labelling function, .
Algorithm 1 Selective Segmentation Algorithm

7 Results

(a) Test Image 1
(b) Test Image 2
(c) Test Image 3
(a)
(b)
(c)
Figure 5: Test Images 1–3; the ground truth contours are defined in the first row and the corresponding user input marker set is shown in the second row. These are synthetic images with homogeneous foregrounds selected to highlight the benefits of the proposed model.
Figure 5: Test Images 1–3; the ground truth contours are defined in the first row and the corresponding user input marker set is shown in the second row. These are synthetic images with homogeneous foregrounds selected to highlight the benefits of the proposed model.
(d) Test Image 4
(e) Test Image 5
(f) Test Image 6
(a)
(b)
(c)
Figure 6: Test Images 4–6; the ground truth contours are defined in the first row and the corresponding user input marker set is shown in the second row. These are real images with some degree of intensity inhomogeneity in the foreground, with potential medical applications in mind.
Figure 6: Test Images 4–6; the ground truth contours are defined in the first row and the corresponding user input marker set is shown in the second row. These are real images with some degree of intensity inhomogeneity in the foreground, with potential medical applications in mind.
(d) Test Image 7
(e) Test Image 8
(f) Test Image 9
(a)
(b)
(c)
Figure 7: Test Images 7–9; the ground truth contours are defined in the first row and the corresponding user input marker set is shown in the second row. These are real images with approximately homogeneous foregrounds. The challenge is that the background contains substantial regions of a similar intensity.
Figure 7: Test Images 7–9; the ground truth contours are defined in the first row and the corresponding user input marker set is shown in the second row. These are real images with approximately homogeneous foregrounds. The challenge is that the background contains substantial regions of a similar intensity.
Figure 8: Colour scaling corresponding to TC values, representing the accuracy of the result. This scale is used in subsequent figures.

In this section we will present results obtained using the proposed model and compare them to using fitting terms from similar models (CV ACWE , RSF RSF , LCV LCV , HYB Ali:16 , GAV Ali:17 ), detailed in §2. We intend to provide an overview of how effective each model is in a number of key respects and analyse their potential for practical use in a reliable and consistent manner. Our focus is on how each model can be extended to a consistent selective segmentation framework. The key questions we consider are:

  1. How sensitive are the results to variations of the parameters and ?

  2. Is the model capable of achieving accurate results?

  3. To what extent is the proposed model dependent on the user input?

Test Images. We will perform all tests on the images shown in Figs. 57. We have provided the ground truth and initialisation used for each image. Test Images 1–3 are synthetic, Test Image 4 is an MRI scan of a knee, Test Images 5–6 are abdominal CT scans, and Test Images 7–9 are lung CT scans. They have been selected to present challenges relevant to the discussion in §2. We focus on medical images as this is the application of most interest to our work. In the following we will discuss the results in terms of synthetic images (1–3) and real images (4–9).

Measuring Segmentation Accuracy. In our tests we use the Jaccard Coefficient Jaccard:12 , often referred to as the Tanimoto Coefficient (TC), to measure the quality of the segmentation. We define accuracy with respect to a ground truth, , given by a manual segmentation:

The Tanimoto Coefficient is then calculated as

where refers to the number of points in the enclosed region. This takes values in the range , with higher TC values indicating a more accurate segmentation. In the following we will represent accuracy visually from red () to green (), with the intermediate scaling of colours used shown in Fig. 8. This will be particularly relevant in §7.2.

Note. In §2.4 we mentioned the tuning of parameters in the GAV model. To be explicit the optimal pairs used in the following tests were (4,-2) for Test Images 1 and 2, (1.5,0.5) for Test Images 3,4, and 6, (2,0) for Test Image 5, and (-2,4) for Test Images 7,8, and 9. Results vary significantly as are varied, but we found these to be the best choices for each image.

The discussion of results is split into four sections, addressing the questions introduced above. First, in Section 7.1, we will examine the robustness to the parameters and for each model. Then, in Section 7.2, we will compare the optimal accuracy achieved by each method to determine what each data fitting term is capable of in the context of selective segmentation for these examples. In Section 7.3, we will test the proposed model with respect to the user input. By randomising the input we will determine to what extent the proposed model is suitable for use in practice. Finally, we will compare the proposed approach to the methods introduced in §3 on an additional CT data set. This will help establish how the algorithm performs against competitive approaches in the literature.

7.1 Parameter Robustness

Figure 9: Example heatmap of TC values to display segmentation accuracy for parameters .
(a) (i) = 1, TC = 0.00
(b) (ii) = 2, TC = 0.79
(c) (iii) = 3, TC = 0.91
(d) (iv) = 4, TC = 0.95
(e) (v) = 5, TC = 0.95
(f) (vi) = 6, TC = 0.95
(g) (vii) = 7, TC = 0.94
(h) (viii) = 8, TC = 0.94
(i) (ix) = 9, TC = 0.93
(j) (x) = 10, TC = 0.93
(k) (xi) = 15, TC = 0.93
(l) (xii) = 20, TC = 0.85
Figure 10: Segmentation results and TC values for the proposed model whilst varying (with ). The colours correspond to the TC value (green is TC = 1, red is TC = 0), consistent with the scale in Fig. 8. This is for Test Image 5, with the corresponding heatmap provided in Fig. 12.
CV ACWE
RSF RSF
LCV LCV
HYB Ali:16
GAV Ali:17
Proposed
(a) Test Image 1
(b) Test Image 2
(c) Test Image 3
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
(k)
(l)
(m)
(n)
(o)
(p)
(q)
(r)
Figure 11: Heatmaps of TC values for permutations of and . Each row and column is labelled according to the model used and the image tested. The colour is consistent with the scale in Fig. 8. Here, we present Test Images 1 – 3.
CV ACWE
RSF RSF
LCV LCV
HYB Ali:16
GAV Ali:17
Proposed
Figure 11: Heatmaps of TC values for permutations of and . Each row and column is labelled according to the model used and the image tested. The colour is consistent with the scale in Fig. 8. Here, we present Test Images 1 – 3.
CV ACWE
RSF RSF
LCV LCV
HYB Ali:16
GAV Ali:17
Proposed
(s) Test Image 4
(t) Test Image 5
(u) Test Image 6
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
(k)
(l)
(m)
(n)
(o)
(p)
(q)
(r)
Figure 12: Heatmaps of TC values for permutations of and . Each row and column is labelled according to the model used and the image tested. The colour is consistent with the scale in Fig. 8. Here, we present Test Images 4 – 6.
CV ACWE
RSF RSF
LCV LCV
HYB Ali:16
GAV Ali:17
Proposed
Figure 12: Heatmaps of TC values for permutations of and . Each row and column is labelled according to the model used and the image tested. The colour is consistent with the scale in Fig. 8. Here, we present Test Images 4 – 6.
CV ACWE
RSF RSF
LCV LCV
HYB Ali:16
GAV Ali:17
Proposed
(s) Test Image 7
(t) Test Image 8
(u) Test Image 9
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
(k)
(l)
(m)
(n)
(o)
(p)
(q)
(r)
Figure 13: Heatmaps of TC values for permutations of and . Each row and column is labelled according to the model used and the image tested. The colour is consistent with the scale in Fig. 8. Here, we present Test Images 7 – 9.
CV ACWE
RSF RSF
LCV LCV
HYB Ali:16
GAV Ali:17
Proposed
Figure 13: Heatmaps of TC values for permutations of and . Each row and column is labelled according to the model used and the image tested. The colour is consistent with the scale in Fig. 8. Here, we present Test Images 7 – 9.

In these tests we aim to demonstrate how sensitive to parameter choices each choice of fitting term is. To accomplish this we perform the segmentations for each of the models discussed (CV, RSF, LCV, HYB, GAV) and the proposed model for a wide range of parameters and compute the TC value. The parameter range used is . Due to computational constraints, we run for each integer between 1 and 10, and every fifth from 15 to 50. This aspect of a model’s performance is vital when used in practice. The less sensitive to parameter choices a model is the more relevant it is in relation to potential applications.

[capbesideposition=left,top,capbesidewidth=1.5in]table[]

   Model Test Image 1 2 3 4 5 6 7