A Framework for Dynamic Image Sampling Based on Supervised Learning (SLADS)

03/14/2017 ∙ by G. M. Dilshan P. Godaliyadda, et al. ∙ 0

Sparse sampling schemes have the potential to dramatically reduce image acquisition time while simultaneously reducing radiation damage to samples. However, for a sparse sampling scheme to be useful it is important that we are able to reconstruct the underlying object with sufficient clarity using the sparse measurements. In dynamic sampling, each new measurement location is selected based on information obtained from previous measurements. Therefore, dynamic sampling schemes have the potential to dramatically reduce the number of measurements needed for high fidelity reconstructions. However, most existing dynamic sampling methods for point-wise measurement acquisition tend to be computationally expensive and are therefore too slow for practical applications. In this paper, we present a framework for dynamic sampling based on machine learning techniques, which we call a supervised learning approach for dynamic sampling (SLADS). In each step of SLADS, the objective is to find the pixel that maximizes the expected reduction in distortion (ERD) given previous measurements. SLADS is fast because we use a simple regression function to compute the ERD, and it is accurate because the regression function is trained using data sets that are representative of the specific application. In addition, we introduce a method to terminate dynamic sampling at a desired level of distortion, and we extended the SLADS methodology to sample groups of pixels at each step. Finally, we present results on computationally-generated synthetic data and experimentally-collected data to demonstrate a dramatic improvement over state-of-the-art static sampling methods.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 5

page 9

page 10

page 11

page 13

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

In conventional point-wise image acquisition, an object is measured by acquiring every measurement in a rectilinear grid. However, in certain imaging techniques one high fidelity pixel measurement can take up to to seconds, which translates to between hours of acquisition time for an image with resolution. Examples of such imaging techniques include X-ray diffraction spectroscopy [1], high resolution electron back scatter diffraction (EBSD) microscopy[2], and Raman spectroscopy, which are of great importance in material science and chemistry

Sparse sampling offers the potential to dramatically reduce the time required to acquire an image. In sparse sampling a subset of all available measurements are acquired, and the full resolution image is reconstructed from this set of sparse measurements. In addition to reducing image acquisition time, sparse sampling also has the potential to reduce the exposure of the object/person being imaged to potentially harmful radiation. This is critically important when imaging biological samples using X-rays, electrons, or even optical photons [3, 4]. Another advantage of sparse sampling is that it reduces the amount of measurement data that must be stored.

However, for a sparse sampling method to be useful, it is critical that the reconstruction made from the sparse set of samples allows for accurate reconstruction of the underlying object. Therefore, the selection of sampling locations becomes critically important. All methods that researchers have proposed for sparse sampling can broadly be sorted into two primary categories: static and dynamic.

Static sampling refers to any method that collects measurements in a predetermined order. Random sampling strategies such as in [5], low-discrepancy sampling [6] and uniformly spaced sparse sampling methods [7] are examples of static sparse sampling schemes. Static sampling methods can also be based on a model of the object being sampled such as in [8, 9]. In these methods knowledge of the object geometry and sparsity are used to pre-determine the measurement locations.

Alternatively, dynamic sampling refers to any method that adaptively determines the next measurement location based on information obtained from previous measurements. Dynamic sampling has the potential to produce a high fidelity image with fewer measurements because of the information available from previous measurements. Intuitively, the previous measurements provide a great deal of information about the best location for future measurements.

Over the years, a wide variety of dynamic sampling methods have been proposed for many different applications. We categorize these dynamic sampling methods into three primary categories — dynamic compressive sensing methods where measurements are unconstrained projections, dynamic sampling methods developed for specific applications where measurements are not point-wise measurements, and dynamic sampling methods developed for single pixel measurements.

In dynamic compressive sensing methods the objective at each step is to find the measurement that reduces the entropy the most. In these methods the entropy is computed using the previous measurements and a model for the underlying data. Examples of such methods include [10, 11, 12]. However, in these methods the measurements are unconstrained projections and therefore cannot readily be generalized for point-wise sampling.

The next category of dynamic sampling (DS) methods in the literature are those developed for specific applications where the measurements are not point-wise measurements. One example is [13], where the authors modify the optimal experimental design [14] framework to incorporate dynamic measurement selection in a biochemical network. Another example is presented by Seeger et al. in [15] to select optimal K-space spiral and line measurements for magnetic resonance imaging (MRI). Then Batenburg et al. in [16] presents another example for binary computed tomography, where in each step the measurement that maximizes the information gain is selected.

There are also a few DS methods developed specifically for point-wise measurements. One example is presented in [17] by Kovačević et al. In this algorithm, an object is initially measured using a sparse rectilinear grid. Then, if the intensity of a pixel is above a certain threshold, the vicinity of that pixel is also measured. However, the threshold here is empirically determined and therefore is not robust for general applications. Another point-wise dynamic sampling method was proposed in [18]

. Here, in each step the pixel which when measured reduces the posterior variance the most is selected for measurement. The posterior variance is computed using samples generated from the posterior distribution using the Metropolis-Hastings algorithm. However, Monte-Carlo methods such as the Metropolis-Hastings algorithm can be very slow for cases where the dimensions of the random vector are large. Another shortcoming of this method is that it does not account for the change of conditional variance in the full image due to a new measurement.

In this paper, we present an algorithm for point-wise dynamic sampling based on supervised learning techniques that we first presented in [19]. We call this algorithm a supervised learning approach for dynamic sampling (SLADS). In each step of the SLADS algorithm we select the pixel that maximizes the expected reduction in distortion (

) given previous measurements. We estimate the

from previous measurements using a simple regression function that we train offline. As a result, we can compute the very fast during dynamic sampling. However, for our algorithm to be accurate we need to train this regression function with reduction in distortion () values resulting from many different measurements. In certain cases, where the image sizes are large, computing sufficiently many entries for the training database can be very time consuming. To solve this problem we introduce an efficient training scheme that allows us to extract many entries for the training database with just one reconstruction. We empirically validate this approximation for small images and then detail a method to find the parameters needed for this approximation when dealing with larger images. Then we introduce a stopping condition for dynamic sampling, which allows us to stop when a desired distortion level is reached, as opposed to stopping after a predetermined number of samples are acquired. Finally we extend our algorithm to incorporate group-wise sampling so that multiple measurements can be selected in each step of the algorithm.

In the results section of this paper, we first empirically validate our approximation to the by performing experiments on sized computationally generated EBSD images. Then we compare SLADS with state-of-the-art static sampling methods by sampling both simulated EBSD and real SEM images. We observe that with SLADS we can compute a new sample location very quickly (in the range of - ), and can achieve the same reconstruction distortion as static sampling methods with dramatically fewer samples. Finally, we evaluate the performance of group-wise SLADS by comparing it to SLADS and to static sampling methods.

Ii Dynamic Sampling Framework

In sparse image sampling we measure only a subset of all available pixels. However, for a sparse sampling technique to be useful, the measured pixels should yield an accurate reconstruction of the underlying object. Dynamic sampling (DS) is a particular form of sparse sampling in which each new measurement is informed by all the previous pixel measurements.

To formulate the dynamic sampling problem, we first denote the unknown image formed by imaging the entire underlying object as . We then denote the value of the pixel at location by , where is the set of all locations in the image.

Now, we assume that pixels have already been measured at a set of locations . We then represent the measurements and the corresponding locations as a matrix

From these measurements, , we can compute an estimate of the unknown image . We denote this best current estimate of the image as .

Now we would like to determine the next pixel location to measure. If we select a new pixel location and measure its value , then we can presumably reconstruct a better estimate of . We denote this improved estimate as .

Of course, our goal is to minimize the distortion between and , which we denote by the following function

(1)

where is some scalar measure of distortion between the two pixels and . Here, the function may depend on the specific application or type of image. For example we can let where .

In fact, minimizing this distortion is equivalent to maximizing the reduction in the distortion that occurs when we make a measurement. To do this, we define as the reduction-in-distortion at pixel location resulting from a measurement at location .

(2)

Notice that typically will be a positive quantity since we expect that the distortion should reduce when we collect additional information with new measurements. While this is typically true, in specific situations can actually be negative since a particular measurement might mislead one into increasing the distortion. Perhaps more importantly, we cannot know the value of because we do not know .

Therefore, our real goal will be to minimize the expected value of given our current measurements. We define the expected reduction-in-distortion () as

(3)

The specific goal of our greedy dynamic sampling algorithm will be to select the pixel according to the following rule.

(4)

Intuitively, equation (4) selects the next pixel to maximize the expected reduction-in-distortion given all the available information .

Once is determined, we then form a new measurement matrix given by

(5)

We repeat this process recursively until the stopping condition discussed in Section IV is achieved.

In summary, the greedy dynamic sampling algorithm is given by the following iteration.

repeat                                                   until Desired fidelity is achieved

In Section IV, we will introduce a stopping condition that can be used to set a specific expected quality level for the reconstruction.

Iii Supervised Learning Approach for Dynamic Sampling (SLADS)

The challenge in implementing this greedy dynamic sampling method is accurately determining the function, . Our approach to solve this problem is to use supervised learning techniques to determine this function from training data.

More specifically, the supervised learning approach for dynamic sampling (SLADS) will use an off-line training approach to learn the relationship between the and the available measurements, , so that we can efficiently predict the . More specifically, we would like to fit a regression function , so that

(6)

Here

denotes a non-linear regression function determined through supervised learning, and

is the parameter vector that must be estimated in the learning process. Notice above that we have dropped the superscript since we would like to estimate a function that will work for any number of previously measured pixels.

Now, to estimate we must construct a training database containing multiple corresponding pairs of . Here, , where is the best estimate of computed using the measurements , and is the best estimate of computed using the measurements along with an additional measurement at location .

Notice that since is the reduction-in-distortion, it requires knowledge of the true image . Since this is an off-line training procedure, is available, and the regression function, , will compute the required conditional expectation . However, in order to compute for a single value of , we must compute two full reconstructions, both and . Since reconstruction can be computationally expensive, this means that creating a large database can be very computationally expensive. We will address this problem and propose a solution to it in Section III-A.

For our implementation of SLADS, the regression function

will be a function of a row vector containing features extracted from

. More specifically, at each location , a -dimensional feature vector will be computed using and used as input to the regression function. The specific choices of features used in are listed in Section III-D; however, other choices are possible.

From this feature vector, we then compute the using a linear predictor with the following form:

(7)

We can estimate the parameter by solving the following least-squares regression

(8)

where R is an -dimensional column vector formed by

(9)

and V is given by

(10)

So together consist of training pairs, , that are extracted from training data during an off-line training procedure. The parameter is then given by

(11)

Once is estimated, we can use it to find the for each unmeasured pixel during dynamic sampling. Hence, we find the location to measure by solving

(12)

where denotes the feature vector extracted from the measurements at location . It is important to note that this computation can be done very fast. The pseudo code for SLADS is shown in Figure (1).

function  SLADS()                            while Stopping condition not met do                    for   do                              Extract                                                                    end for                                                                                                             end while                       end function

Fig. 1: SLADS algorithm in pseudo code. The inputs to the function are the initial measurements , the coefficients needed to compute the , found in training, , and , the number of measurements. When the stopping condition is met, the function will output the selected set of measurements .

Iii-a Training for SLADS

In order to train the SLADS algorithm, we must form a large training database containing corresponding pairs of and . To do this, we start by selecting training images denoted by . We also select a set of sampling densities represented by .

For image and each sampling density, , we randomly select of all pixels in the image to represent the simulated measurement locations. Then for each of the remaining unmeasured locations, , in the image , we compute the pairs and save them to the training database. This process is then repeated for all the sampling densities and all the images to form a complete training database.

Figure 2 illustrates this procedure. Note that by selecting a set of increasing sampling densities, the SLADS algorithm can be trained to accurately predict the in conditions where the sampling density is low or high. Intuitively, by sampling a variety of images with a variety of sampling densities, the final training database is constructed to represent the behaviors that will occur as the sampling density increases when SLADS is used.

Fig. 2: Illustration of how training data is extracted from one image in the training database. We first select of the pixels in the image and consider them as measurements . Then for all unmeasured pixel locations () we extract a feature vector and also compute . We the repeat the process for when and of the pixels are considered measurements. Here again is the set of all locations in the training image and is the set of measured locations when of pixels are selected as measurements. All these pairs of are then stored in the training database.

Iii-B Approximating the reduction-in-distortion

While the training procedure described in Section III-A is possible, it is very computationally expensive because of the need to exactly compute the value of . Since this computation is done in the training phase, we rewrite equation (2) without the dependence on to be

(13)

where is known in training, is the reconstruction using the selected sample points, and is the reconstruction using the selected sample points plus the value . Notice that must be recomputed for each new pixel . This requires that a full reconstruction be computed for each entry of the training database. While this maybe possible, it typically represents a very large computational burden.

In order to reduce this computational burden, we introduce in this section a method for approximating the value of so that only a single reconstruction must be performed in order to evaluate for all pixels in an image. This dramatically reduces the computation required to build the training database.

In order to express our approximation, we first rewrite the reduction-in-distortion in the form

where

So here is the reduction-in-distortion at pixel due to making a measurement at pixel . Using this notation, our approximation is given by

(14)

where

(15)

and is the distance between the pixel and the nearest previously measured pixel divided by a user selectable parameter . More formally, is given by

(16)

where is the set of measured locations. So this results in the final form for the approximate reduction-in-distortion given by

(17)

where is a parameter that will be estimated for the specific problem.

In order to understand the key approximation of equation (14), notice that the reduction-in-distortion is proportional to the product of and . Intuitively, represents the weighted distance of from the location of the new measurement, ; and is the initial distortion at before the measurement was made. So for example, when , then and we have that

(18)

In this case, the reduction-in-distortion is exactly the initial distortion since the measurement is assumed to be exact. However, as becomes more distant from the pixel being measured, , the reduction-in-distortion will be attenuated by the weight .

Figures 3(b) and (c) illustrate the shape of for two different cases. In Figures 3(b), the pixel is further from the nearest measured pixel, and in Figures 3(c), the pixel is nearer. Notice that as becomes more distant from the measurement location , the reduction-in-distortion becomes smaller.

(a) Measurements
(b)
(c)
Fig. 3: This figure illustrates the shape of the function as a function of . (a) The red squares represent the two new measurement locations and , and the yellow squares represent the locations of previous measurements. (b) The function resulting from a measurement at location . (c) The function resulting from a measurement at location . Notice that since then the weighting function is wider than the function .

Iii-C Estimating the parameter

In this section, we present a method for estimating the parameter used in equation (17). To do this, we create a training database that contains the approximate reduction-in-distortion for a set of parameter values. More specifically, each entry of the training database has the form

where is a set of possible parameter values, and is the approximate reduction-in-distortion computed using the parameter value .

Using this training database, we compute the associated parameter vectors , and using these parameter vectors, we apply the SLADS algorithm on images and stop each simulation when samples are taken. Then for each of these SLADS simulations, we compute the total distortion as

(19)

where is the actual image, and is the associated image reconstructed using the first samples and parameter value . Next we compute the average total distortion over the training images given by

(20)

From this, we then compute the area under the curve as the overall distortion metric for each given by

(21)

where is the total number of samples taken before stopping. The optimal parameter value, , is then selected to minimize the overall distortion metric given by

(22)

Iii-D Local Descriptors in Feature Vector

In our implementation, the feature vector is formed using terms constructed from the scalar descriptors listed in Table I. More specifically, we take all the unique second-degree combinations formed from these descriptors to form . This gives us a total of elements for the vector .

The first two descriptors in Table I are the gradients of the image computed from the measurement in the horizontal and vertical directions. The second two descriptors are measures of the variance for each unmeasured location. So the first four descriptors quantify the intensity variation at each unmeasured location. Then the last two descriptors quantify how densely (or sparsely) the region surrounding an unmeasured pixel is measured. In particular, the first of these descriptors is the distance from the nearest measurement to an unmeasured pixel and the second is the area fraction that is measured in a circle surrounding an unmeasured pixel.

Measures of gradients
Gradient of the reconstruction in horizontal () direction.
where, and are pixels adjacent to the horizontal direction
Gradient of the reconstruction in vertical () direction.
where, and are pixels adjacent to the horizontal direction

Measures of standard deviation

Here is the set containing the indices of the nearest measurements to .
Here,
and is the euclidean distance between and .
Measures of density of measurements
The distance from to the closest measurement.
Here is the area of a circle the size of the image. is the measured area inside .
TABLE I: List of descriptors used to construct the feature vector . There are three main categories of descriptors: measures of gradients, measures of standard deviation, and measures of density of measurements surrounding the pixel .
Fig. 4: The SLADS algorithm as a flow diagram. The inputs to the function are the initial measurements , the coefficients needed to compute the , found in training, , and the set containing the indices of the measurements. When the stopping condition is met the function will output the selected set of measurements .

Iv Stopping Condition for SLADS

In applications, it is often important to have a stopping criteria for terminating the SLADS sampling process. In order to define such a criteria, we first define the expected total distortion (ETD) at step by

Notice that the expectation is necessary since the true value of is unavailable during the sampling process. Then our goal is to stop sampling when the ETD falls below a predetermined threshold.

In order to make this practical, at each step of the SLADS algorithm we compute

(23)

where , is a user selected parameter that determines the amount of temporal smoothing, is the measured value of the pixel at step , and is the reconstructed value of the same pixel at step .

Intuitively, the value of measures the average level of distortion in the measurements. So a large value of indicates that more samples need to be taken, and a smaller value indicates that the reconstruction is accurate and the sampling process can be terminated. However, in typical situations, it will be the case that

because the SLADS algorithm will tend to select pixels to measure whose values have great uncertainty.

In order to compensate for this effect, we compute a function using a look-up-table (LUT) and stop sampling when

The function is determined using a set of training images, . For each image, we first determine the number of steps, , required to achieve the desired distortion.

(24)

Then we average the value of for each of the images to determine the adjusted threshold:

(25)

Selection of the parameter allows the estimate of to be smoothed as a function of . In practice, we have used the following formula to set :

where is the number of pixels in the image.

Figure 4 shows the SLADS algorithm as a flow diagram after the stopping condition in the previews section is incorporated.

V Group-wise SLADS

function  Group-wise SLADS()                            while Stopping condition not met do                    for   do                              Form pseudo-measurement vector as shown in equation (27)                              Compute pseudo- from using equation (28)                                                                                     end for                                                            end while                       end function

Fig. 5: Our proposed method for Group-wise SLADS. Here instead of just selecting measurement in each step of SLADS, as we did in Figure 1, we now select measurement locations at each step.

In this section, we introduce a group-wise SLADS approach in which measurements are made at each step of the algorithm. Group-wise SLADS may be more appropriate in applications where it is faster to measure a set of predetermined pixels in a single burst. So at the step, our goal will be to select a group of measurement positions, , that will yield the greatest expected reduction-in-distortion.

(26)

where is the expected reduction-in-distortion due to measurements . However, solving this problem requires that we consider different combinations of measurements.

In order to address this problem, we introduce a method in which we choose the measurements sequentially, just as we do in standard SLADS. Since group-wise SLADS requires that we make measurements in groups, we cannot make the associated measurement after each location is selected. Consequently, we cannot recompute the function after each location is selected, and therefore, we cannot select the best position for the next measurement. Our solution to this problem is to estimate the value at each selected location, and then we use the estimated value as if it were the true measured value.

More specifically, we first determine measurement location using equation (4), and then let . Now without measuring the pixel at , we would like to find the location of the next pixel to measure, . However, since has now been chosen, it is important to incorporate this information when choosing the next location . In our implementation, we temporarily assume that the true value of the pixel, , is given by its estimated value, , computed using all the measurements acquired up until the step. We will refer to as a pseudo-measurement since it takes the place of a true measurement of the pixel. Now using this pseudo-measurement along with all previous real measurements, we estimate a pseudo- for all and from that select the next location to measure. We repeat this procedure to find all measurements.

So the procedure to find the measurement is as follows. We first construct a pseudo-measurement vector,

(27)

where . Then using this pseudo-measurement vector we compute the pseudo- for all

(28)

where is the feature vector that corresponds to location . It is important to note that when the pseudo- is the actual computed using the actual measurements only. Now we find the location that maximizes the pseudo- by

(29)

Then finally we update the set of measured locations by

(30)

Figure 5 shows a detailed illustration of the proposed group-wise SLADS method.

Vi Results

In the following sections, we first validate the approximation to the , and then we compare SLADS to alternative sampling approaches based on both real and simulated data. We then evaluate the stopping condition presented in Section IV and finally compare the group-wise SLADS method presented in Section V with SLADS. The distortion metrics and the reconstruction methods we used in these experiments are detailed in Appendices -A and -B. It is also important to note that we start all experiments by first acquiring of the image according to low-discrepancy sampling[6].

(a)
(b)
Fig. 6: Images used to compute regression coefficients and perform SLADS in the experiment detailed in Section VI-A. These images have pixels each, and are synthetic EBSD image generated using the Dream.3D software. The images here are discretely labeled and the different colors correspond to different crystal orientations. In particular, (a) shows the images that were used to find the regression coefficients and (b) shows the images on which SLADS was performed.

Vi-a Validating the Approximation

In this section, we compare the results using the true and approximate described in Section III-B in order to validate the efficacy of the approximation. The SLADS algorithm was trained and then tested using the synthetic EBSD images shown in Figures 6(a) and 6(b). Both sets of images were generated using the Dream.3D software [20].

The training images were constructed to have a small size of so that it would be tractable to compute the true reduction-in-distortion from equation (13) along with the associated true regression parameter vector . This allowed us to compute the true for this relatively small problem.

We selected the optimal parameter value, , using the method described in Section III-C from the possible values . Figure 7 shows a plot of versus . In this case, the optimal parameter value that minimizes the overall distortion metric is given by . However, we also note that the metric is low for a wide range of values.

Figure 8 shows the results of plotting the total distortion, , versus the percentage of samples for both the true regression parameter, , and the approximate regression parameter, . While the two curves are close, the approximate reduction-in-distortion results in a lower curve than the true reduction-in-distortion. This indicates that the approximation is effective, but it is surprising that the approximate parameter works better than the true parameter. We conjecture that this suggests that there approximations in our models that might allow for future improvements.

Fig. 7: Experimental results from the experiment we performed to pick the best value for in equation 16. The figure shows the overall distortion metric of equation 21 computed from the SLADS experiments for the . The distortion metric is smallest when .
Fig. 8: Experimental results from the experiments we performed to empirically validate the approximation to the . The curves show the TD averaged over different experiments versus the percentage of samples when the true value of was used to estimate and when the approximate value of was used to estimate . In this experiments we selected to compute the approximate RD.
Fig. 9: Images used for training and testing in the experiment detailed in Section VI-B in which we compared SLADS with LS and RS. These images have pixels each, and are synthetic EBSD image generated using the Dream.3D software. The images here are discretely labeled and the different colors correspond to different crystal orientations. In particular, (a): shows the images that were used for training and (b): shows the images that were used for testing.
Fig. 10: In this figure we plot the average , , (averaged over sampling experiments) versus the percentage of samples for the experiment detailed in Section VI-B. The plot shows the curves that correspond to SLADS, RS and LS. The images that were sampled in this experiment are shown in Figure 9(b).
(a) Original Image
(b) RS: Sample locations ()
(c) LS: Sample locations ()
(d) SLADS: Sample locations ()
(e) RS: Reconstructed Image
(f) LS: Reconstructed Image
(g) SLADS: Reconstructed Image
(h) RS: Distortion Image         (TD )
(i) LS: Distortion Image         (TD )
(j) SLADS: Distortion Image         (TD )
Fig. 11: This figure shows sample locations and reconstructions after of the image in (a) was sampled using SLADS, RS and LS. Here (a) is the image being sampled. (b), (c) and (d) show the images reconstructed from samples collected using RS, LS and SLADS respectively. (e), (f) and (g) are the distortion images that correspond to (b), (c) and (d) respectively. A distortion image is defined as where is the ground truth and the is the reconstructed image. Note that the distortion image only has values and since these are discretely labeled images (see Appendix -A). (h), (i) and (j) are the measurement masks that correspond to (b), (c) and (d) respectively.
Fig. 12: In this figure we compare SLADS with Group-wise SLADS. In particular, we performed SLADS and Group-wise SLADS with on simulated EBSD images shown in Figure 9(b) and then plotted versus the percentage of samples. We see that Group-wise SLADS is comparable to SLADS and that as gets larger the distortion for the same number of samples gets larger. Note that SLADS is group-wise SLADS with .

Vi-B Results using Simulated EBSD Images

In this section we first compare SLADS with two static sampling methods – Random Sampling (RS) [5] and Low-discrepancy Sampling (LS) [6]. Then we evaluate the group-wise SLADS method introduced in Section V and finally we evaluate the stopping method introduced in Section IV. Figures 9(a) and 9(b) show the simulated EBSD images we used for training and testing, respectively, for all the experiments in this section. All results used the parameter value that was estimated using the method described in Section III-C. The average total distortion, , for the experiments was computed over the full test set of images.

Figure 10 shows a plot of the average total distortion, , for each of the three algorithms that were compared, LS, RS and SLADS. Notice that SLADS dramatically reduces error relative to LS or RS at the same percentage of samples, and that it achieves nearly perfect reconstruction after approximately of the samples are measured.

Figure 11 gives some insight into the methods by showing the sampled pixel locations after of samples have been taken for each of the three methods. Notice that SLADS primarily samples in locations along edges, but also selects some points in uniform regions. This both localizes the edges more precisely while also reducing the possibility of missing a small region or “island” in the center of a uniform region. Alternatively, the LS and RS algorithms select sample locations independent of the measurements; so samples are used less efficiently, and the resulting reconstructions have substantially more errors along boundaries.

To evaluate the group-wise SLADS method we compare it with SLADS and LS. Figure 12 shows a plot of the average total distortion, , for SLADS, LS, group-wise SLADS with the group sampling rates of and performed on the images in Figure 9(b). We see that group-wise SLADS has somewhat higher distortion for the same number of samples as SLADS and that the distortion increases with increasing values of . This is reasonable since SLADS without group sampling has the advantage of having the most information available when choosing each new sample. However, even when collecting samples in a group, the increase in distortion is still dramatically reduced relative to LS sampling.

We then evaluate the stopping method by attempting to stop SLADS at different distortion levels. In particular, we will attempt to stop SLADS when for . For each value we found the threshold to place on the stopping function, in equation 23, by using the method described in Section IV on a subset of the images in Figure 9(a). Again we used the images shown in Figures 9(a) and 9(b) for training and testing, respectively. After each SLADS experiment stopped we computed the true value, , and then computed the average true value for a given , , by averaging the values over the testing images.

Figure 13 shows a plot of and . From this plot we can see that the experiments that in general . This is the desirable result since we intended to stop when . However, from the standard deviation bars we see that in certain experiments the deviation from is somewhat high and therefore note the need for improvement through future research.

Fig. 13: This plot corresponds to the experiment performed to evaluate the stopping method we presented in Section IV. Here we plot and where . was computed by averaging the values over experiments. Note that we also show the standard deviation bars for . From this plot we see that the stopping method on average is accurate.

It is also important to mention that the SLADS algorithm (for discrete images) was implemented for protein crystal positioning by Simpson et al. in the synchrotron facility at the Argonne national laboratory [1].

Vi-C Results using Scanning Electron Microscope Images

(a) Original Image
(b) RS: Sample locations         ()
(c) LS: Sample locations         ()
(d) SLADS: Sample locations ()
(e) RS: Reconstructed Image
(f) LS: Reconstructed Image
(g) SLADS: Reconstructed Image
(h) RS: Distortion Image         (TD )
(i) LS: Distortion Image         (TD )
(j) SLADS: Distortion Image         (TD )
Fig. 14: This figure shows sample locations and reconstructions after of the image in (a) was sampled using SLADS, RS and LS. Here (a) is the image being sampled. (b), (c) and (d) show the images reconstructed from samples collected using RS, LS and SLADS respectively. (e), (f) and (g) are the distortion images that correspond to (b), (c) and (d) respectively. A distortion image is defined as where is the ground truth and the is the reconstructed image. Note that the distortion image has values ranging from to since these are continuous bit images (see Appendix -A). (h), (i) and (j) are the measurement masks that correspond to (b), (c) and (d) respectively.
Fig. 15: Images used for training and testing in the experiment detailed in Section VI-C in which we compared SLADS with LS and RS. These images have pixels each, and experimental collected SEM images. In particular, (a) shows the images that were used for training and (b) shows the images that were used for testing. These images were collected by Ali Khosravani & Prof. Surya Kalidindi from Georgia Institute of Technology.
Fig. 16: In this figure we plot the average ,, (averaged over sampling experiments) versus the percentage of samples for the experiment detailed in Section VI-C. The plot shows the curves that correspond to SLADS, RS and LS. The images that were sampled in this experiment are shown in Figure 15(b).

In this section we again compare SLADS with LS and RS but now on continuously valued scanning electron microscope (SEM) images. Figures 15(a) and 15(b) show the SEM images used for training and testing, respectively. Using the methods described in Section III-C, the parameter value was estimated and again the average total distortion, was computed over the full test set of images.

Figure 16 shows a plot of for each of the three tested algorithms, SLADS, RS, and LS. We again see that SLADS outperforms the static sampling methods, but not as dramatically as in the discrete case.

Figure 14 shows the results of the three sampling methods after of the samples have been taken along with the resulting sampling locations that were measured. Once more we notice that SLADS primarily samples along edges, and therefore we get better edge resolution. We also notice that some of the smaller dark regions (“islands”) are missed by LS and RS while SLADS is able to resolve almost all of them.

Vii Conclusions

In this paper, we presented a framework for dynamic image sampling which we call a supervised learning approach to dynamic sampling (SLADS). The method works by selecting the next measurement location in a manner that minimizes the expected reduction in distortion (ERD) for each new measurement. The SLADS algorithm dramatically reduces the computation required for dynamic sampling by using a supervised learning approach in which a regression algorithm is used to efficiently estimate the ERD for each new measurement. This makes the SLADS algorithm practical for real-time implementation.

Our experiments show that SLADS can dramatically outperform static sampling methods for the measurement of discrete data. For example, SEM analytical methods such as EBSD[2], or synchrotron crystal imaging [1] are just two cases in which sampling of discrete images is important. We also introduced a group-wise SLADS method which allows for sampling of multiple pixels in a group, with only limited loss in performance. Finally, we concluded with simulations on sampling from continuous SEM images in which we demonstrated that SLADS provides modest improvements compared to static sampling.

-a Distortion Metrics for Experiments

Applications such as EBSD generate images formed by discrete classes. For these images, we use a distortion metric defined between two vectors and as

(31)

where is an indicator function defined as

(32)

where is the element of the vector .

However, for the experiments in Section VI-C we used continuously valued images. Therefore, we defined the distortion between two vectors and as

(33)

-B Reconstruction Methods for Experiments

In the experiments with discrete images experiments all the reconstructions were performed using the weighted mode interpolation method. The weighted mode interpolation of a pixel

is for

(34)

where

(35)

and .

In the training phase of the experiments on continuously valued data, we performed reconstructions using the Plug & Play algorithm [21] to compute the reduction-in-distortion. However, to compute the reconstructions for descriptors (both in testing and training) we used weighted mean interpolation instead of Plug & Play to minimize the run-time speed of SLADS. We define the weighted mean for a location by

(36)

Acknowledgments

The authors acknowledge support from the Air Force Office of Scientific Research (MURI - Managing the Mosaic of Microstructure, grant # FA9550-12-1-0458) and from the Air Force Research Laboratory Materials and Manufacturing directorate (Contract # FA8650-10-D-5201-0038). The authors also thank Ali Khosravani & Prof. Surya Kalidindi, Georgia Institute of Technology for providing the images used for dynamic sampling simulation on an experimentally-collected image.

References

  • [1] N. M. Scarborough, G. M. D. P. Godaliyadda, D. H. Ye, D. J. Kissick, S. Zhang, J. A. Newman, M. J. Sheedlo, A. U. Chowdhury, R. F. Fischetti, C. Das, G. T. Buzzard, C. A. Bouman, and G. J. Simpson, “Dynamic x-ray diffraction sampling for protein crystal positioning,” Journal of Synchrotron Radiation, vol. 24, no. 1, pp. 188–195, January 2017.
  • [2] A. J. Schwartz, M. Kumar, B. L., and D. P. A. Field, Electron Backscatter Diffraction in Materials Science.    Springer US, 2009.
  • [3] R. Smith-Bindman, J. Lipson, and R. Marcus, “Radiation dose associated with common computed tomography examinations and the associated lifetime attributable risk of cancer,” Archives of Internal Medicine, vol. 169, no. 22, pp. 2078–2086, 2009.
  • [4] R. F. Egerton, P. Li, and M. Malac, “Radiation damage in the tem and sem,” Micron, vol. 35, no. 6, pp. 399 – 409, 2004, international Wuhan Symposium on Advanced Electron Microscopy. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0968432804000381
  • [5] H. S. Anderson, J. Ilic-Helms, B. Rohrer, J. Wheeler, and K. Larson, “Sparse imaging for fast electron microscopy,” IS&T/SPIE Electronic Imaging, pp. 86 570C–86 570C, 2013.
  • [6] R. Ohbuchi and M. Aono, “Quasi-monte carlo rendering with adaptive sampling,” 1996.
  • [7] K. A. Mohan, S. V. Venkatakrishnan, J. W. Gibbs, E. B. Gulsoy, X. Xiao, M. D. Graef, P. W. Voorhees, and C. A. Bouman, “Timbir: A method for time-space reconstruction from interlaced views,” IEEE Transactions on Computational Imaging, pp. 96–111, June 2015.
  • [8] K. Mueller, “Selection of optimal views for computed tomography reconstruction,” Patent WO 2,011,011,684, Jan. 28, 2011.
  • [9] Z. Wang and G. R. Arce, “Variable density compressed image sampling,” Image Processing, IEEE Transactions on, vol. 19, no. 1, pp. 264–270, 2010.
  • [10] M. W. Seeger and H. Nickisch, “Compressed sensing and bayesian experimental design,” in Proceedings of the 25th international conference on Machine learning.    ACM, 2008, pp. 912–919.
  • [11] W. R. Carson, M. Chen, M. R. D. Rodrigues, R. Calderbank, and L. Carin, “Communications-inspired projection design with application to compressive sensing,” SIAM Journal on Imaging Sciences, vol. 5, no. 4, pp. 1185–1212, 2012.
  • [12] S. Ji, Y. Xue, and L. Carin, “Bayesian compressive sensing,” IEEE Trans. on Signal Processing, vol. 56, no. 6, pp. 2346–2356, 2008.
  • [13] J. Vanlier, C. A. Tiemann, P. A. J. Hilbers, and N. A. W. van Riel, “A bayesian approach to targeted experiment design,” Bioinformatics, vol. 28, no. 8, pp. 1136–1142, 2012.
  • [14] A. C. Atkinson, A. N. Donev, and R. D. Tobias, Optimum experimental designs, with SAS.    Oxford University Press, Oxford, 2007, vol. 34.
  • [15] M. Seeger, H. Nickisch, R. Pohmann, and B. Schölkopf, “Optimization of k-space trajectories for compressed sensing by bayesian experimental design,” Magnetic Resonance in Medicine, vol. 63, no. 1, pp. 116–126, 2010.
  • [16] K. Joost Batenburg, W. J. Palenstijn, P. Balázs, and J. Sijbers, “Dynamic angle selection in binary tomography,” Computer Vision and Image Understanding, 2012.
  • [17] T. E. Merryman and J. Kovacevic, “An adaptive multirate algorithm for acquisition of fluorescence microscopy data sets,” IEEE Trans. on Image Processing, vol. 14, no. 9, pp. 1246–1253, 2005.
  • [18] G. M. D. Godaliyadda, G. T. Buzzard, and C. A. Bouman, “A model-based framework for fast dynamic image sampling,” in proceedings of IEEE International Conference on Acoustics Speech and Signal Processing, 2014, pp. 1822–6.
  • [19] G. D. Godaliyadda, D. Hye Ye, M. A. Uchic, M. A. Groeber, G. T. Buzzard, and C. A. Bouman, “A supervised learning approach for dynamic sampling,” in IS&T Imaging.    International Society for Optics and Photonics, 2016.
  • [20] M. A. Groeber and M. A. Jackson, “Dream. 3d: a digital representation environment for the analysis of microstructure in 3d,” Integrating Materials and Manufacturing Innovation 3.1, vol. 19, pp. 1–17, 2014.
  • [21] S. Sreehari, S. V. Venkatakrishnan, L. F. Drummy, J. P. Simmons, and C. A. Bouman, “Advanced prior modeling for 3d bright field electron tomography,” in IS&T/SPIE Electronic Imaging.    International Society for Optics and Photonics, 2015, pp. 940 108–940 108.
  • [22] J. Haupt, R. Baraniuk, R. Castro, and R. Nowak, “Sequentially designed compressed sensing,” in Statistical Signal Processing Workshop (SSP).    IEEE, 2012, pp. 401–404.
  • [23] C. Jackson, R. F. Murphy, and J. Kovacevic, “Intelligent acquisition and learning of fluorescence microscope data models,” IEEE Trans. on Image Processing, vol. 18, no. 9, pp. 2071–2084, 2009.
  • [24] X. Huan and Y. M. Marzouk, “Simulation-based optimal bayesian experimental design for nonlinear systems,” Journal of Computational Physic 232.1, pp. 288–317, 2013.