The Trace Criterion for Kernel Bandwidth Selection for Support Vector Data Description

I Introduction

Support vector data description (SVDD) is a machine learning technique that is used for single-class classification and anomaly detection. First introduced by Tax and Duin

[11]

, SVDD’s mathematical formulation is almost identical to the one-class variant of support vector machines: one-class support sector machines (OCSVM), which is attributed to Schölkopf et al.

[9]. The use of SVDD is popular in domains in which the majority of data belongs to a single class and no distributional assumptions can be made. For example, SVDD is useful for analyzing sensor readings from reliable equipment for which almost all the readings describe the equipment’s normal state of operation.

Like other one-class classifiers, SVDD provides a geometric description of the observed data. The SVDD classifier assigns a distance to each point in the domain space; the distance measures the separation of that point from the training data. During scoring, any observation found to be at a large distance from the training data might be an anomaly, and the user might choose to generate an alert.

Several researchers have proposed using SVDD for multivariate process control [10, 2]. Other applications of SVDD involve monitoring condition of machines [12, 14] and image classification [8].

I-a Mathematical Formulation

In this section, we describe the mathematical formulation of SVDD; the description is based on [11].
Normal Data Description:
The SVDD model for normal data description builds a hypersphere that contains most of the data within a small radius. Given observations $x_{1}, \dots, x_{n}$ , we need to solve the following optimization problem to obtain the SVDD data description.
Primal Form:
Objective:

m i n R^{2} + C n \sum i = 1 ξ_{i},

(1)

subject to:

	$∥ x_{i} - a ∥^{2} \leq R^{2} + ξ_{i}, \forall i = 1, \dots, n,$		(2)
	$ξ_{i} \geq 0, \forall i = 1, \dots, n,$		(3)

where:
$x_{i} \in R^{m}, i = 1, \dots, n$ represent the training data,
$R$ is the radius and represents the decision variable,
$ξ_{i}$ is the slack for each variable,
$a$ is the center (a decision variable),
$C = \frac{1}{n f}$ is the penalty constant that controls the trade-off between the volume and the errors, and
$f$ is the expected outlier fraction.

Dual Form:
The dual formulation is obtained using Lagrange multipliers.
Objective:

max n \sum i = 1 α_{i} ⟨ x_{i}, x_{i} ⟩ - \sum i, j α_{i} α_{j} ⟨ x_{i}, x_{j} ⟩,

(4)

subject to:

	$n \sum i = 1 α_{i} = 1,$		(5)
	$0 \leq α_{i} \leq C, \forall i = 1, \dots, n$		(6)

where $α_{i} \in R$ are the Lagrange constants, and $C = \frac{1}{n f}$ is the penalty constant.

Duality Information:
The position of observation $x_{i}$ is connected to the optimal $α_{i}$ , $R$ the radius of the sphere, and $a$ the center of the sphere in the following manner:

Center position:

n \sum i = 1 α_{i} x_{i} = a

(7)

Inside position:

∥ x_{i} - a ∥ < R ⟺ α_{i} = 0

(8)

Boundary position:

∥ x_{i} - a ∥ = R ⟺ 0 < α_{i} < C

(9)

Outside position:

∥ x_{i} - a ∥ > R ⟺ α_{i} = C

(10)

Any $x_{i}$ for which the corresponding $α_{i} > 0$ is known as a support vector.

Let $S V_{< C}$ denote the set ${x_{j} : 0 < α_{j} < C}$ . Then the radius of the hypersphere is calculated as follows for any $x_{k} \in S V_{< C}$ :

R^{2} = ⟨ x_{k}, x_{k} ⟩ - 2 \sum i α_{i} ⟨ x_{i}, x_{k} ⟩ + \sum i, j α_{i} α_{j} ⟨ x_{i}, x_{j} ⟩ .

(11)

The value of $R^{2}$ does not depend on the choice of $x_{k} \in S V_{< C}$ .
Scoring:

For any point $z$ , the distance ${dist}^{2} (z)$ is calculated as follows:

{dist}^{2} (z) = ⟨ z, z ⟩ - 2 \sum i α_{i} ⟨ x_{i}, z ⟩ + \sum i, j α_{i} α_{j} ⟨ x_{i}, x_{j} ⟩

(12)

Points whose ${dist}^{2} (z) > R^{2}$ are designated as outliers.

The spherical data boundary can include a significant amount of space that has a sparse distribution of training observations. Using this model to score can lead to a lot of false positives. Hence, instead of a spherical shape, a compact bounded outline around the data is often desired. Such an outline should approximate the shape of the single-class training data. This is possible by using kernel functions.

Flexible Data Description:

The support vector data description is made flexible by replacing the inner product $⟨ x_{i}, x_{j} ⟩$ with a suitable kernel function $K (x_{i}, x_{j})$ . The Gaussian kernel function used in this paper is defined as

K_{s} (x, y) = e x p \frac{- ∥ x - y ∥^{2}}{2 s^{2}};

(13)

where $s$ is the Gaussian bandwidth parameter.

The modified mathematical formulation of SVDD with a kernel function is as:

Objective:

m a x n \sum i = 1 α_{i} K_{s} (x_{i}, x_{i}) - \sum i, j α_{i} α_{j} K_{s} (x_{i}, x_{j}),

(14)

subject to:

	$n \sum i = 1 α_{i}$	$= 1,$		(15)
	$0 \leq α_{i} \leq C, \forall i = 1, \dots, n$			(16)

In perfect analogy with the previous section, any $x_{i}$ for which $α_{i} = 0$ is an inside point and any $x_{i}$ for which $α_{i} > 0$ is called a support vector.

$S V_{< C}$ is similarly defined as ${x_{j} : 0 < α_{j} < C}$ and the threshold $R^{2}$ is calculated, as follows for any $x_{k} \in S V_{< C}$ :

R^{2} = K (x_{k}, x_{k}) - 2 \sum i α_{i} K (x_{i}, x_{k}) + \sum i, j α_{i} α_{j} K_{s} (x_{i}, x_{k})

(17)

The value of $R^{2}$ does not depend on which $x_{k} \in S V_{< C}$ is used.
Scoring: For any observation $z$ , the distance ${dist}^{2} (z)$ is calculated as follows:

{dist}^{2} (z) = K (z, z) - 2 \sum i α_{i} K (x_{i}, z) + \sum i, j α_{i} α_{j} K (x_{i}, x_{j})

(18)

Any point $z$ for which ${dist}^{2} (z) > R^{2}$ is designated as an outlier.

I-B Importance of the Kernel Bandwidth Value

In practice, SVDD is almost always computed by using the Gaussian kernel function, and it is important to set the value of bandwidth parameter correctly. A small bandwidth leads to overfitting, and the resulting SVDD classifier overestimates the number of anomalies. A large bandwidth leads to underfitting, and many anomalies cannot be detected by the classifier.

Because SVDD is an unsupervised learning technique, it is desirable to have an automatic, unsupervised bandwidth selection technique that does not depend on labeled data that separate the

inliers from the outliers. In [4], Kakde et al. present the peak criterion, which is an unsupervised bandwidth selection technique, and show that it performs better than alternative unsupervised methods. However, determining the bandwidth that is suggested by the peak criterion requires that the SVDD solution be computed multiple times for the training data for a list of bandwidth values that lie on a grid. Even though using sampling techniques can speed up the computation (see [7]), this method is still expensive. Moreover, it is also necessary to initiate the grid search at a good starting value in order to avoid unnecessary computation, and it is not immediately obvious what a good starting value is.

In [5], Liao et al. present the modified mean criterion for bandwidth selection. The suggested bandwidth has a closed-form expression in terms of the training data, and it can be computed very quickly. SVDD that is trained by using the modified mean criterion bandwidth is reasonably accurate for many data sets; however, it is less accurate than the peak criterion in general.

In this paper, we introduce the trace criterion for Gaussian bandwidth selection. The computation of the trace criterion consists of finding the inflection point of a smooth function of the bandwidth parameter. The computation terminates quickly when standard nonlinear optimization methods such as Newton-Raphson are used. This method is efficient for moderately large data sets.

Our results show that the trace criterion is competitive with the peak criterion for many data sets. Simulation studies suggest that this method is more accurate than the mean criterion for a certain class of high-dimensional data.

These properties make the trace criterion method a good bandwidth selection technique. However, unsupervised bandwidth tuning is an extremely difficult problem, so it is quite possible that there is a class of data sets for which the trace criterion does not give good results.

The rest of the paper is organized as follows. Section II defines the trace criterion for bandwidth tuning, and the remaining sections compare the mean, peak; and trace criterion with each other.

Ii The Trace Criterion for Bandwidth Selection

Ii-a Parameter tuning using inflection points

Parameter tuning for unsupervised learning methods such as clustering and one-class classification can be difficult when external validation data are not available, as is quite frequently the case. A popular method for parameter tuning in such cases is to look at the values of a “validation measure” as a function of the hyperparameter of interest, and choose the value of the hyperparameter where the function has an inflection point

[1]. For a specific example, consider the problem of determining the number of clusters for

K

-means.

S S Q

: the sum of the squared distance of each point in the training data from the cluster center closest to can be taken as a validation measure. If the number of clusters is held fixed, then a lower value of

S S Q

indicates better clustering. Let

S S Q (K)

denote the value of

S S Q

for the clustering that is suggested by

K

-means with

K

clusters. As

K

increases,

S S Q (K)

decreases, so you cannot choose the number of clusters as

K = {argmin}_{K} S S Q (K)

because it would suggest as many clusters as the number of points in the training data. However, it is observed that the value of

K

at which the function

K \to S S Q (K)

has an inflection point is quite frequently a good choice for the number of clusters. The inflection point of other validation measures such as the silhouette coefficient is also used to determine the number of clusters [1].

We will now propose a validation measure for SVDD whose inflection point provides our suggested bandwidth.

Ii-B Validation measure for trace criterion

Assume we have a training data set $x_{1}, \dots, x_{N}$ that consists of $N$ distinct points in $R^{p}$ and we want to determine a good kernel bandwidth value for training this data set. In kernel methods, including SVDD, the objective function to be optimized depends on how the data are transformed through the kernel matrix $K = (K (x_{i}, x_{j}))_{i, j = 1}^{N} .$ Since this matrix has $N^{2}$ elements, where $N$ is the number of observations, it is impossible to work with the entire kernel matrix even for moderate values of $N$ . The Gaussian kernel matrix is always positive semidefinite, and it typically has a rapidly decaying spectrum. The rapidly decaying spectrum can be exploited to create a low-rank positive semidefinite approximation $~ K$ of $K$

by replacing all the eigenvalues in the spectral decomposition of

K

below a certain threshold by

0

[13].

~ K

has a square root of low rank, that is,

~ K = G G^{T}

, where

G

is an

n \times r

matrix with

r ≪ N

. Since

K \approx ~ K

G

can be considered an approximate low- rank square root of

K

. In many cases, computation of expressions that involve

K

can be made tractable by replacing

K

with

G G^{T}

. However, it is not feasible to compute

G

such that

K \approx G G^{T}

by using the eigendecomposition of

K

when

N

is large. The Nyström methods form a class of popular methods to compute a low-rank representation of

K

even when

N

is large. We use a variant of Nyström method as described in [15] to construct a validation measure whose inflection point will be suggested as the bandwidth.

Given data $x_{1}, \dots, x_{N}$ in $R^{p}$ and an integer $0 < r ≪ N$ , let $z_{2}, z_{2}, \dots, z_{r}$ be distinct landmark points in $R^{p}$ . The landmark points are chosen so that they are evenly distributed within the training data. Following [15], we choose $z_{1}, \dots, z_{r}$ as the cluster centers that are obtained from a $K -$ means clustering with $r$ clusters of the training data.

From the theory of reproducing kernel Hilbert spaces (RKHS), we know that for any $s > 0$ there is a Hilbert space $H_{s}$ and a mapping $Φ : R^{p} \to H_{s}$ such that for any $x, y \in R^{p}$ we have $⟨ Φ (x), Φ (y) ⟩_{s} = exp - \frac{∥ x - y ∥^{2}}{2 s^{2}} : = K_{s} (x, y)$ , where $⟨, ⟩_{s}$ denotes the inner product on $H_{s}$ and $∥ ∥_{s}$ denotes the norm on $H_{s}$ .

Let $V_{0} \subseteq H_{s}$ denote the subspace spanned by $Φ (z_{1}), \dots, Φ (z_{r})$ . Note that $V_{0}$ is a finite-dimensional and hence closed subspace of $H_{s}$ of dimension exactly $r$ . Given $x \in R^{p}$ , let $ˆ Φ (x)$ denote the projection of $Φ (x)$ on $V_{0} .$

In the Nyström methods, the kernel matrix ${(K_{s} (x_{i}, x_{j}))}_{s}^{N} = {(⟨ Φ (x_{i}), Φ (x_{j}) ⟩)}_{i}^{N}$ is approximated by $(⟨ ˆ Φ (x_{i}), ˆ Φ (x_{j}) ⟩)_{i, j = 1}^{N}$ , a rank $r$ matrix that has a low-dimensional square root that can be easily computed.

We use the projected values $ˆ Φ (x)$ for a different purpose. We use the discrepancy between $ˆ Φ (x)$ and $Φ (x)$ for $x$ in the training data to come up with a validation measure.

From usual least square arguments, we know that for any $x \in R^{p}$ , we have $ˆ Φ (x) = \sum_{i = 1}^{r} γ_{i} Φ (z_{i})$ , where $γ_{1}, \dots, γ_{r}$ are such that

⟨ Φ (x) - r \sum i = 1 γ_{i} Φ (z_{i}), Φ (z_{k}) ⟩ = 0 % f o r k = 1, \dots, r .

This implies the normal equations,

K_{s} (x, z_{k}) = r \sum i = 1 γ_{i} K_{s} (z_{i}, z_{k}), 1 \leq k \leq r,

and implies that $γ_{i}$ can be explicitly computed as

$⎛ ⎜ ⎜ ⎜ ⎝ \begin{matrix} γ_{1} γ_{2} \dots γ_{r} \end{matrix} ⎞ ⎟ ⎟ ⎟ ⎠ = {⎛ ⎜ ⎜ ⎜ ⎜ ⎝ \begin{matrix} 1 & K_{s} (z_{1}, z_{2}) & \dots & K_{s} (z_{1}, z_{r}) K_{s} (z_{1}, z_{2}) & 1 & \dots & K_{s} (z_{2}, z_{r}) \dots & \dots & \dots & \dots K_{s} (z_{r}, z_{1}) & K_{s} (z_{r}, z_{2}) & \dots & 1 \end{matrix} ⎞ ⎟ ⎟ ⎟ ⎟ ⎠}^{- 1} ⎛ ⎜ ⎜ ⎜ ⎜ ⎝ \begin{matrix} K_{s} (x, z_{1}) K_{s} (x, z_{2}) \dots K_{s} (x, z_{r}) \end{matrix} ⎞ ⎟ ⎟ ⎟ ⎟ ⎠ .$

Since $ˆ Φ (x)$ and $Φ (x) - ˆ Φ (x)$ are orthogonal, the squared norm of the residual is given by

	$ρ (x, s)$	$: = ∥ ˆ Φ (x) - Φ (x) ∥_{s}^{2}$
		$= ∥ Φ (x) ∥_{s}^{2} - ∥ ˆ Φ (x) ∥_{s}^{2}$
		$= 1 - ∥ r \sum i = 1 γ_{i} Φ (z_{i}) ∥_{s}^{2} .$

This shows that $0 \leq ρ (x, s) \leq 1$ . Let $ψ (x, s) : = 1 - ρ (x, s)$ ; the closer $ψ (x, s)$ is to $1$ , the lower the residual error is. From the preceding expression for $γ_{i}$ you can see that
$ψ (x, s) = {⎛ ⎜ ⎜ ⎜ ⎜ ⎝ \begin{matrix} K_{s} (x, z_{1}) K_{s} (x, z_{2}) \dots K_{s} (x, z_{r}) \end{matrix} ⎞ ⎟ ⎟ ⎟ ⎟ ⎠}^{T} {⎛ ⎜ ⎜ ⎜ ⎜ ⎝ \begin{matrix} 1 & K_{s} (z_{1}, z_{2}) & \dots & K_{s} (z_{1}, z_{r}) K_{s} (z_{1}, z_{2}) & 1 & \dots & K_{s} (z_{2}, z_{r}) \dots & \dots & \dots & \dots K_{s} (z_{r}, z_{1}) & K_{s} (z_{r}, z_{2}) & \dots & 1 \end{matrix} ⎞ ⎟ ⎟ ⎟ ⎟ ⎠}^{- 1} ⎛ ⎜ ⎜ ⎜ ⎜ ⎝ \begin{matrix} K_{s} (x, z_{1}) K_{s} (x, z_{2}) \dots K_{s} (x, z_{r}) \end{matrix} ⎞ ⎟ ⎟ ⎟ ⎟ ⎠$

So, given training data $x_{1}, \dots, x_{N}$ , if we define

g (s) : = \frac{1}{N} N \sum i = 1 ψ (x_{i}, s),

then $g (s)$ is a measure of the loss of accuracy due to projection into $V_{0}$ , where $g (s)$ lies between $0$ and $1$ for each $s > 0.$ Higher values of $g (s)$ indicate lower loss in precision due to projection into $V_{0}$ . In particular, if $g (s) = 1$ , there is absolutely no loss of precision. That is,

ˆ Φ (x_{i}) = Φ (x_{i}) for 1 \leq i \leq N .

It is empirically observed that typically $g (s)$ increases, as $s$ increases (that is, $g^{'} (s) > 0$ ). Moreover $h (s) = g^{'} (s)$ is unimodal: it initially increases until it reaches its maximum value, and then decreases. So $g (s)$ typically has a well-defined inflection point where $h (s)$ takes its maximum value. See Figure 1 for the graph of a typical plot of $g ()$ and $h ()$ .

The trace criterion suggests using $s^{*} = {argmax}_{s > 0} h (s)$ as the bandwidth that empirical observations suggest coincides with the inflection point of $g$ in most cases.

Even though $h (s)$ is a scalar function, it is defined in terms of matrix operations and we can come up with an explicit closed form for $h (s)$ by using usual rules of matrix differential calculus [6].

Let $U (s)$ denote the $r \times r$ kernel matrix of representative points $(K_{s} (z_{i}, z_{j}))_{i, j = 1}^{r}$ . Its element-by-element derivative is given by the $r \times r$ matrix,

For $1 \leq i \leq N$ , let

W_{i} (s) = (K_{s} (x_{i}, z_{k}))_{k = 1}^{r}

denote a $r \times 1$ column vector. Then its element-by-element derivative is given by the $r \times 1$ column vector,

W_{i}^{'} (s) = \frac{1}{s^{3}} {(∥ x_{i} - z_{k} ∥^{2} K_{s} (x_{i}, z_{k}))}_{i}^{2}

Then we have

g (s) = \frac{\sum_{i = 1}^{N} W_{i}^{T} (s) U^{- 1} (s) W_{i} (s)}{N}

, and hence

h (s) = g^{'} (s) = \frac{2 \sum_{i = 1}^{N} W_{i}^{T} (s) U^{- 1} (s) W_{i}^{'} (s)}{N} - \frac{\sum_{i = 1}^{N} W_{i}^{T} (s) U^{- 1} (s) U^{'} (s) U^{- 1} (s) W_{i} (s)}{N} .

Let $B_{i} (s) : = U^{- 1} (s) W_{i} (s)$ . Then the preceding equation simplifies to

h (s) = \frac{2 \sum_{i = 1}^{N} B_{i}^{T} (s) W_{i} (s)}{N} - \frac{\sum_{i = 1}^{N} B_{i}^{T} (s) U^{^{'}} (s) B_{i} (s)}{N} .

(19)

From (19), it is apparent that $h (s)$ is differentiable on $0 < s < \infty$ , such as we can use standard nonlinear optimization methods such as Newton-Raphson to maximize it.

The cost of evaluating $h (s)$ is $O (N r^{2})$ , which is tractable because $r ≪ N$ .

We have to choose the number of representative points, $r$ , for applying the trace criterion. We have observed that $r = 5$ gives a good result for most data sets, and we have used the trace criterion with $r = 5$ throughout this paper unless explicitly stated otherwise.

Fig. 1: Plot of a validation function and its derivative

Iii Evaluating the Trace Criterion

In this section, we will use different data sets evaluate the trace criterion and compare its performance with the modified mean [5] and the peak criteria [4]. The evaluation is based on a series of data sets. Starting from simple two-dimensional connected data, we gradually moved toward more complex high-dimensional disconnected data. For our evaluation, we used both real-life and simulated data.
We observed that for two-dimensional connected data and for real-life data sets that are used in this evaluation, the performance of the modified mean and trace criteria is comparable. When data were two-dimensional and disconnected (meaning that data contained multiple disjoint clusters), the bandwidth value that was computed using the trace criterion provided much better data description than the modified mean criterion (see results in Table 4 and 5).
We also evaluated the trace criterion by using high-dimensional hyperspheres and hypercubes. The training data set consisted of 1 to 12 disjoint hyperspheres or hypercubes, and the dimension of the data varied between 5 to 40 in increments of 5. The scoring data used for evaluation consisted of observations from inside one or more hypercubes or one or more hyperspheres, and observations that are just outside the training data, in close proximity of the training data as seen in Figure 8, 10, 12 and 13. For this set of evaluations, we observed that the bandwidth value provided by the trace criterion produced a magnitude better performance than the modified mean criterion.

Iii-a Choice of Data Sets

We evaluate the trace criterion by using following four types of data sets:

Connected two-dimensional data: These are data that have two variables and no clusters.
Disconnected two-dimensional data: These are data that have two variables and two or more clusters.
Higher-dimensional data: These refer to real-life data that have more than two variables.
High-dimensional simulated data: These data consist of one or more hyperspheres or hypercubes.

Iv Comparison Using Two-Dimensional Connected Data

Iv-a Data Description

This section compares the trace criterion with the peak and modified mean criteria using two-dimensional connected data. Such data have no clusters. We use the star-shaped data and banana-shaped data, and we train an SVDD model by using the bandwidth value obtained by the peak, modified mean and trace criteria. To evaluate the results, we scored the bounding rectangle of the data by dividing it into a 200 $\times$ 200 grid. With a good bandwidth value, the inlier region that is obtained from scoring should match the geometry of the training data. Figures 2 and 3 display the results, along with the bandwidth value.

Iv-B Scoring Results

The scoring results indicate that the bandwidth value obtained using the modified mean and trace criteria provide good-quality data description. The value of bandwidth $s$ that is obtained using the modified mean criterion is comparable to the one obtained using the peak criterion.

Fig. 2: Results for two-dimensional connected data (Banana-shaped data)

Fig. 3: Results for Two-Dimensional Connected Data (Star-shaped Data)

V Comparison Using Two-Dimensional Disconnected Data

V-a Data Description

This section compares the trace criterion with the peak and modified mean criteria using two-dimensional data that lie across multiple clusters. We have observed that computing a good bandwidth value for such data is more difficult than for connected data. Since the data are two-dimensional, we can visually judge the quality of results. To evaluate the results, we scored the bounding rectangle of the data by dividing it into a 200 $\times$ 200 grid. With a good bandwidth value, the inlier region obtained from scoring should match the geometry of the training data. We use following two data sets for evaluations:

The refrigerant data which consist of four clusters [3]
A simulated “two-donut and a circle” data set, which consists of two donut-shaped clusters and one circular cluster

Figures 4 and 5 display the results, along with the bandwidth value.

V-B Scoring Results

The scoring results indicate that the bandwidth value obtained using the peak and the trace criteria provides a data description of reasonably good quality, for both data sets. For refrigerant data, the bandwidth values that are computed by peak and trace criteria are close. The scoring results indicate that the data description that is obtained using these values is able to separate out all four clusters, whereas the description that is obtained using the modified mean criterion bandwidth value merges the two clusters that lie close to each other. As indicated in Figure 4, the bandwidth value that is obtained using the modified mean criterion is significantly larger than the one obtained using the trace and the peak criteria.

Fig. 4: Results for two-dimensional disconnected data (Refrigerant Data)

Fig. 5: Results for two-dimensional disconnected data (Two Donuts and a circle data)

Vi Comparison Using High-Dimensional Data

Comparing the different criteria for high-dimensional data is much more difficult than comparing them for two-dimensional data. In two-dimensional data, the quality of the result can be easily judged by looking at the plot of the scoring results. But this is not possible for high-dimensional data. For the purpose of evaluation, we selected labeled high-dimensional data that have a dominant class. We used SVDD on a subset of the dominant class to obtain a description of the dominant class, and then we scored the rest of the data to evaluate the criteria. We expected the points in the scoring data set that correspond to the dominant class to be classified as inliers and all other points to be classified as outliers. Because the data are labeled, we could also use cross validation to determine the bandwidth that best describes the dominant class in the sense of maximizing a measure of fit, such as the $F_{1}$ score. So in this section we compare the bandwidths that are suggested by the different unsupervised criteria with the bandwidth that is obtained through cross validation for various benchmark data sets. The results are summarized in Table I . The benchmark data sets used for the analysis are described in sections VI-A1 through VI-A2.

	Shuttle	Tennessee Eastman
# Variables	9	41
# Observations	2000	2000
Max $(F_{1}) (s)$	0.96 (17)	0.19 (17)
Peak $(F_{1}) (s)$	0.96 (14)	0.16 (8)
Modified Mean $(F_{1}) (s)$	0.96 (17.2)	0.181 (11.37)
Trace $(F_{1}) (s)$	0.958 (13.1)	0.181 (11.22)

TABLE I: Results for high-dimensional Data

Vi-a Data Description

Vi-A1 Shuttle Data

This data set consists of measurements made on a shuttle. The data set contains nine numeric attributes and one classification attribute. Of 58,000 total observations, 80% belong to class 1. A random sample of 2,000 observations belonging to class 1 was selected for training, and the remaining 56,000 observations were used for scoring. This data set is from the UC Irvine Machine Learning Repository.

Vi-A2 Tennessee Eastman Data

The data set was generated using the MATLAB simulation code, which provides a model of an industrial chemical process. The data were generated for normal operations of the process and 20 faulty processes. Each observation consists of 41 variables, out of which 22 were measured continuously every 6 seconds on average and the remaining 19 were sampled at a specified interval of every 0.1 or 0.25 hours. From the simulated data, we created an analysis data set that uses the normal operations data of the first 90 minutes and data that correspond to faults 1 through 20. A data set that contains observations of normal operations was used for training. Scoring was performed to determine whether the model could accurately classify an observation as belonging to normal operation of the process. The MATLAB simulation code is available at [7].

Vi-B Scoring Results

The results outlined in table I indicate that the $F_{1}$ measure values that were obtained using all three criteria are equivalent and the values are in the neighborhood of the best $F_{1}$ value that can be obtained by cross validation.

Vii Comparison Using Simulated Data

In this section, we present results of a simulation study that we conducted to compare performance of the trace criterion with the peak and modified mean criteria. The simulations were performed to generate training data with a known geometry. The data dimensions were varied between 2 and 10. We conducted three simulation studies. Table II provides details of these studies.

Simulation	Data
Dimension	# of
data sets	Details
Two-dimensional polygons	2	600	Polygons with varying number of vertices and length of sides
Hypercubes	5 to 40 in increments of 5	400	Single hypercube used for training
Hyperspheres	5 to 40 in increments of 5	400	Single hypersphere used for training
High-dimensional disconnected data with multiple hyperspheres	2 to 10	880	Each training data set contains multiple disjoint hyperspheres, ranging from 2 to 12.
High-dimensional disconnected data with multiple hypercubes	2 to 10	880	Each training data set contains multiple disjoint hypercubes, ranging from 2 to 12.

TABLE II: Simulation study

Vii-a Evaluation Using Two-Dimensional Polygons

In this section, we measure the performance of the trace criterion when it is applied to randomly generated polygons. Given the number of vertices, $k$ , we generate the vertices of a randomly generated polygon in the anticlockwise sense as $r_{1} exp i θ_{(1)}, \dots, r_{k} exp i θ_{(k)} .$ Here $θ_{(1)} = 0$ and $θ_{(i)}$ for $i = 2, \dots, n$ are the order statistics of an i.i.d sample that is uniformly drawn from $(0, 2 π) .$ The $r_{i}$ are uniformly drawn from an interval $[r_{min}, r_{max}] .$

For this simulation, we chose $r_{min} = 3$ and $r_{max} = 5$ and varied the number of vertices from $5$ to $30$ . We generated $20$ random polygons for each vertex size. Having determined a polygon, we randomly sampled $600$ points uniformly from the interior of the polygon and used trace criterion and this sample to determine a bandwidth value. Figure 6 shows two random polygons.

However, since we can easily determine whether a point lies in the interior of a polygon, we can also use cross validation to determine a good bandwidth value. To do so, we found the bounding rectangle of each of the polygons and divided it into a $200 \times 200$ grid. We then labeled each point on this grid as an “inside” or an “outside” point. We then fit SVDD on the sampled data and scored the points on this grid for different values of $s$ and chose the value of $s$ that maximized the $F_{1}$ -measure.

The performance of a bandwidth selection criterion can be measured by the $F_{1}$ -measure ratio, which is defined as $F_{Trace} / F_{best}$ , where $F_{Trace}$ is the $F_{1}$ -measure that is obtained when the value suggested by the trace criterion is used and $F_{best}$ is the best possible value of $F_{1}$ -measure over all values of $s$ . A value close to 1 indicates that a bandwidth selection criterion is competitive with cross validation. We have $20$ values of this ratio for each vertex size.
The box and whiskers plot in Figure 7 summarizes the simulation study results for the modified mean and trace criteria. The X-axis shows the number of vertices of the ploygon and Y-axis shows the $F_{1}$

-measure ratio. The bottom and the top of the box shows the first and the third quartile values. The ends of the whiskers represent the minimum and the maximum values of the

F_{1}

-measure ratio. The plot shows that the

F_{1}

-measure ratio is greater than 0.9 across all values of number of vertices. Because the complexity of the ploygon increases as the number of vertices increases, we observed that the spread of the

F_{1}

-measure ratio increased slightly. The fact that the

F_{1}

-measure ratio is always greater than 0.9 provides necessary evidence that both the trace criterion and the modified mean criterion generalize across different training data sets.

Fig. 7: Evaluation using random polygons

Vii-B Evaluation Using Hyperspheres

In this section, we evaluate the trace criterion by using spherical data of varying dimensions. The observations in such spherical data (or hypersphere) are uniformly distributed. We use scoring to evaluate the quality of the data description that is obtained using the trace criterion bandwidth value. The scoring data set consists of 50%

inlier observations, which are uniformly distributed inside the training sphere and 50% of outlier observations, which are uniformly distributed outside the sphere. The points outside the sphere lie in a narrow annular ring, just outside the sphere. Figure 8 illustrates two variables in the training and scoring data. The rationale behind creating such scoring data set is that if the bandwidth value is good, then the data description that is obtained using such a value should be able to discriminate between observations that are inside and observations that are just outside the sphere. We varied the hypershpere dimension from 5 to 40 in increments of 5. For each dimension, 50 sets of training and scoring data sets were simulated. We computed the

F_{1}

measure for each data set to determine the quality of data description. Figure 9 shows a box-and-whiskers plot of the

F_{1}

measure for various values of data dimension. The

F_{1}

measure decreases as the number of variables (the data dimension) increases from 5 to 40. The

F_{1}

measure is consistently above 0.9 for all simulated data sets across different dimensions, the

F_{1}

measure value dropped rapidly with the increase in the hypersphere dimension for the modified mean criterion. This observation confirms that a bandwidth value that is obtained using the trace criterion provides a much better-quality data description than the modified mean criterion provides.

Vii-C Evaluation Using Hypercubes

In this section, we evaluate the trace criterion using cube-shaped data of varying dimensions. The observations in such cubic data (or hypercube) are uniformly distributed. We used scoring to evaluate the quality of data description that was obtained using the trace criterion bandwidth value. The scoring data set consists of 50% inlier observations, which are uniformly distributed inside the training cube and 50% outlier observations which are uniformly distributed outside the cube. The points outside the cube lie in a narrow frame, just outside the cube. Figure 10 illustrates the two variables in the training and scoring data. The rationale behind creating such a scoring data set is that if the bandwidth value is good, then the data description that is obtained using such a value should be able to discriminate between observations that are inside and observations that are just outside the cube. We varied the hypercube dimension from 5 to 40 in increments of 5. For each dimension, 50 sets of training and scoring data sets were simulated. We computed the $F_{1}$ measure for each data set to determine the quality of data description. Figure 11 shows a box-and-whiskers plot of the $F_{1}$ measure for various values of data dimension. The $F_{1}$ measure decreases as the number of variables (the data dimension) increases from 5 to 40. Although, the $F_{1}$ measure is consistently above 0.7 for all simulated data sets across different dimensions, the $F_{1}$ measure value dropped rapidly with the increase in the the hypercube dimension for the modified mean criterion. This observation confirms that bandwidth value that is obtained using the trace criterion provides a much better-quality data description than the modified mean criterion provides.

Vii-D Evaluation Using High-Dimensional Disconnected Data

In this section, we use high-dimensional disconnected data to evaluate the performance of the trace criteria. The training data consist of two or more disjoint hyperspheres or the hypercubes used earlier in sections VII-B and VII-C. The details of the simulation study are presented below.

Vii-D1 Evaluation using multiple hyperspheres

We evaluated the trace and modified mean criteria by using data sets that contain multiple numbers of spheres. The number of spheres in a data set was varied between 2 and 12, and the data dimension was varied between 5 and 40 in increments of 5. For each combination of data dimension and number of spheres, we used different seed values to generate 10 different sets of training and scoring data sets. Figure 12 illustrates sample training and scoring data sets that have five spheres and use two dimensional data.

Fig. 12: Evaluation using multiple hyperspheres

For each simulation run, we computed the $F_{1}$ measure. Table LABEL:fig:sphereboxWhisker (a) to Table LABEL:fig:sphereboxWhisker (k) provide the box-and-whiskers plot of the $F_{1}$ measure for simulations results that were obtained using different number of hyperspheres. Each section of Table LABEL:fig:sphereboxWhisker provides $F_{1}$ measure for different values of data dimension. As seen in Table LABEL:fig:sphereboxWhisker (a) to Table LABEL:fig:sphereboxWhisker (d), show that when the number of hyperspheres is between 2 and 4, both the modified mean and then the trace criteria provide $F 1$ values above 0.9, with the modified mean criterion, performing slightly better than the trace criterion. But as the number of hyperspheres increases from 5 to 12, Table LABEL:fig:sphereboxWhisker (e) to Table LABEL:fig:sphereboxWhisker(k) indicate that the performance of the modified mean criterion drops compared to the trace criterion. The trace criteria consistently provides a $F_{1}$ measure higher than 0.9, whereas the $F_{1}$ measure obtained using the modified mean criterion is in the range of 0.7 to 0.8, depending on the number of hyperspheres.

TABLE III: Evaluation using multiple hyperspheres

Vii-D2 Evaluation Using Multiple Hypercubes

We evaluated the trace and modified mean criteria by using data sets that contain multiple numbers of cubes. The number of cubes in a data set was varied between 2 and 12, and the data dimension was varied between 5 and 40 in increments of 5. For each combination of data dimension and number of cubes, we generated 10 different sets of training and scoring data sets by different seed value. Figure 13 illustrates sample training and scoring data set that have five cubes and use two-dimensional data.

Fig. 13: Evaluation using multiple hypercubes

For each simulation run, we computed the $F_{1}$ measure. Table LABEL:fig:cubeboxWhisker(a) to Table LABEL:fig:cubeboxWhisker(k) provide the box-and-whiskers plot of the $F_{1}$ measure for simulation results that were obtained using different number of hypercubes. Each section of Table LABEL:fig:cubeboxWhisker provides $F_{1}$ measure for different values of data dimension. Table LABEL:fig:cubeboxWhisker(a) to Table LABEL:fig:cubeboxWhisker(c), show that when the number of hypercubes is between 2 and 4, both the modified mean and the trace criteria provide $F 1$ values above 0.9, with the modified mean criterion, performing slightly better than the trace criterion. But as the number of hypercubes increases from 5 to 12, Table LABEL:fig:cubeboxWhisker(d) to Table LABEL:fig:cubeboxWhisker(k) indicate that the performance of the modified mean criterion drops compared to the trace criterion. The trace criterion consistently provides an $F_{1}$ measure higher than 0.9, whereas the $F_{1}$ measure that is obtained using the modified mean criterion is in the range of 0.7 to 0.8, depending on the number of hypercubes.

TABLE IV: Evaluation using multiple hypercubes

Viii Conclusion

The trace criterion for computing the bandwidth value of a Gaussian kernel for SVDD, as proposed in this paper, exploits the low-rank representation of the kernel matrix to suggest a bandwidth value. Several evaluations that use synthetic and real-life data sets indicate that the bandwidth value that is obtained using the trace criterion provides similar or better results compared to existing methods. The trace criterion method provides good values when data are high-dimensional and disjoint.

Acknowledgement

Authors would like to thank Anne Baxter, Principal Technical Editor at SAS, for her assistance in creating this manuscript.

References

[1] Aggarwal, C. C. Data mining: the textbook. Springer, 2015.
[2] Camci, F., and Chinnam, R. B. General support vector representation machine for one-class classification of non-stationary classes. Pattern Recognition 41, 10 (2008), 3021–3034.
[3] Heckert, N. A., and Filliben, J. J. NIST handbook 148: DATAPLOT reference manual, Volume I: Commands. http://www.itl.nist.gov/div898/software/dataplot/, 2000. [Online; accessed 4-August-2017].
[4] Kakde, D., Chaudhuri, A., Kong, S., Jahja, M., Jiang, H., and Silva, J. Peak Criterion for Choosing Gaussian Kernel Bandwidth in Support Vector Data Description. In 2017 IEEE International Conference on Prognostics and Health Management (ICPHM) (PHM2017) (2017). [Online preprint.] Available: https://arxiv.org/abs/1602.05257.
[5] Liao, Y., Kakde, D., Chaudhuri, A., Jiang, H., Sadek, C., and Kong, S. A new bandwidth selection criterion for using svdd to analyze hyperspectral data. In Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XXIV (2018), vol. 10644, International Society for Optics and Photonics, p. 106441M. [Online preprint.] Available: https://arxiv.org/abs/1803.03328.
[6] Magnus, J. R., and Neudecker, H. Matrix differential calculus with applications in statistics and econometrics.
Wiley series in probability and mathematical statistics
(1988).
[7] Peredriy, S., Kakde, D., and Chaudhuri, A. Kernel bandwidth selection for SVDD: The sampling peak criterion method for large data. In 2017 IEEE International Conference on Big Data (Big Data) (Dec 2017), pp. 3540–3549.
[8] Sanchez-Hernandez, C., Boyd, D. S., and Foody, G. M. One-class classification for mapping a specific land-cover class: SVDD classification of fenland. Geoscience and Remote Sensing, IEEE Transactions on 45, 4 (2007), 1061–1073.
[9] Schölkopf, B., Williamson, R. C., Smola, A. J., Shawe-Taylor, J., and Platt, J. C.
Support vector method for novelty detection.
In Advances in neural information processing systems (2000), pp. 582–588.
[10] Sukchotrat, T., Kim, S. B., and Tsung, F. One-class classification-based control charts for multivariate process monitoring. IIE transactions 42, 2 (2009), 107–120.
[11] Tax, D. M., and Duin, R. P. Support vector data description. Machine learning 54, 1 (2004), 45–66.
[12] Widodo, A., and Yang, B.-S. Support vector machine in machine condition monitoring and fault diagnosis. Mechanical Systems and Signal Processing 21, 6 (2007), 2560–2574.
[13] Williams, C. K. I., and Seeger, M. Using the nyström method to speed up kernel machines. In Advances in Neural Information Processing Systems 13, T. K. Leen, T. G. Dietterich, and V. Tresp, Eds. MIT Press, 2001, pp. 682–688.
[14] Ypma, A., Tax, D. M., and Duin, R. P.
Robust machine fault detection with independent component analysis and support vector data description.
In Neural Networks for Signal Processing IX, 1999. Proceedings of the 1999 IEEE Signal Processing Society Workshop. (1999), IEEE, pp. 67–76.
[15] Zhang, K., Tsang, I. W., and Kwok, J. T. Improved nyström low-rank approximation and error analysis. In Proceedings of the 25th international conference on Machine learning (2008), ACM, pp. 1232–1239.


(a) Training Data	(b) Peak $s = 0.775$

(c) Modified Mean $s = 1.144$	(d) Trace $s = 0.657$


(a) Training Data	(b) Peak $s = 0.9$

(c) Modified Mean $s = 0.936$	(d) Trace $s = 0.846$


(a) Training Data	(b) Peak $s = 10.0145$

(c) Modified Mean $s = 0.197$	(d) Trace $s = 0.010$


(a) Training Data	(b) Peak $s = 1.05$

(c) Modified Mean $s = 1.28$	(d) Trace $s = 0.79$


(a) Number of hyperspheres = 2	(b) Number of hyperspheres = 3

(c) Number of hyperspheres = 4	(d) Number of hyperspheres = 5

(e) Number of hyperspheres = 6	(f) Number of hyperspheres = 7

(g) Number of hyperspheres = 8	(h) Number of hyperspheres = 9

(i) Number of hyperspheres = 10	(j) Number of hyperspheres = 11

(k) Number of hyperspheres = 12

The Trace Criterion for Kernel Bandwidth Selection for Support Vector Data Description

Authors

The Mean and Median Criterion for Automatic Kernel Bandwidth Selection for Support Vector Data Description

Sampling Method for Fast Training of Support Vector Data Description

Kernel Bandwidth Selection for SVDD: Peak Criterion Approach for Large Data

Gaussian RBF Centered Kernel Alignment (CKA) in the Large Bandwidth Limit

Simple and Effective Prevention of Mode Collapse in Deep One-Class Classification

Fast Incremental SVDD Learning Algorithm with the Gaussian Kernel

Peak Criterion for Choosing Gaussian Kernel Bandwidth in Support Vector Data Description

I Introduction

I-a Mathematical Formulation

I-B Importance of the Kernel Bandwidth Value

Ii The Trace Criterion for Bandwidth Selection

Ii-a Parameter tuning using inflection points

Ii-B Validation measure for trace criterion

Iii Evaluating the Trace Criterion

Iii-a Choice of Data Sets

Iv Comparison Using Two-Dimensional Connected Data

Iv-a Data Description

Iv-B Scoring Results

V Comparison Using Two-Dimensional Disconnected Data

V-a Data Description

V-B Scoring Results

Vi Comparison Using High-Dimensional Data

Vi-a Data Description

Vi-A1 Shuttle Data

Vi-A2 Tennessee Eastman Data

Vi-B Scoring Results

Vii Comparison Using Simulated Data

Vii-a Evaluation Using Two-Dimensional Polygons

Vii-B Evaluation Using Hyperspheres

Vii-C Evaluation Using Hypercubes

Vii-D Evaluation Using High-Dimensional Disconnected Data

Vii-D1 Evaluation using multiple hyperspheres

Vii-D2 Evaluation Using Multiple Hypercubes

Viii Conclusion

Acknowledgement

References


(a) Number of hypercubes = 2	(b) Number of hypercubes = 3

(c) Number of hypercubes = 4	(d) Number of hypercubes = 5

(e) Number of hypercubes = 6	(f) Number of hypercubes = 7

(g) Number of hypercubes = 8	(h) Number of hypercubes = 9

(i) Number of hypercubes = 10	(j) Number of hypercubes = 11

(k) Number of hypercubes = 12

The Trace Criterion for Kernel Bandwidth Selection for Support Vector Data Description

Authors

Related Research

The Mean and Median Criterion for Automatic Kernel Bandwidth Selection for Support Vector Data Description

Sampling Method for Fast Training of Support Vector Data Description

Kernel Bandwidth Selection for SVDD: Peak Criterion Approach for Large Data

Gaussian RBF Centered Kernel Alignment (CKA) in the Large Bandwidth Limit

Simple and Effective Prevention of Mode Collapse in Deep One-Class Classification

Fast Incremental SVDD Learning Algorithm with the Gaussian Kernel

Peak Criterion for Choosing Gaussian Kernel Bandwidth in Support Vector Data Description

This week in AI

I Introduction

I-a Mathematical Formulation

I-B Importance of the Kernel Bandwidth Value

Ii The Trace Criterion for Bandwidth Selection

Ii-a Parameter tuning using inflection points

Ii-B Validation measure for trace criterion

Iii Evaluating the Trace Criterion

Iii-a Choice of Data Sets

Iv Comparison Using Two-Dimensional Connected Data

Iv-a Data Description

Iv-B Scoring Results

V Comparison Using Two-Dimensional Disconnected Data

V-a Data Description

V-B Scoring Results

Vi Comparison Using High-Dimensional Data

Vi-a Data Description

Vi-A1 Shuttle Data

Vi-A2 Tennessee Eastman Data

Vi-B Scoring Results

Vii Comparison Using Simulated Data

Vii-a Evaluation Using Two-Dimensional Polygons

Vii-B Evaluation Using Hyperspheres

Vii-C Evaluation Using Hypercubes

Vii-D Evaluation Using High-Dimensional Disconnected Data

Vii-D1 Evaluation using multiple hyperspheres

Vii-D2 Evaluation Using Multiple Hypercubes

Viii Conclusion

Acknowledgement

References