Switchable Deep Beamformer

08/31/2020 ∙ by Shujaat Khan, et al. ∙ KAIST 수리과학과 0

Recent proposals of deep beamformers using deep neural networks have attracted significant attention as computational efficient alternatives to adaptive and compressive beamformers. Moreover, deep beamformers are versatile in that image post-processing algorithms can be combined with the beamforming. Unfortunately, in the current technology, a separate beamformer should be trained and stored for each application, demanding significant scanner resources. To address this problem, here we propose a switchable deep beamformer that can produce various types of output such as DAS, speckle removal, deconvolution, etc., using a single network with a simple switch. In particular, the switch is implemented through Adaptive Instance Normalization (AdaIN) layers, so that various output can be generated by merely changing the AdaIN code. Experimental results using B-mode focused ultrasound confirm the flexibility and efficacy of the proposed methods for various applications.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 3

page 4

page 7

page 8

page 9

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Ultrasound (US) is one of the most versatile medical imaging modalities. Thanks to its ultra-fast frame rate and radiation free nature, it is the first choice for applications such as fetal imaging, cardiac imaging, etc.

The image generation process in US involves scanning and reconstruction steps. In focused mode, the region of interest (ROI) is scanned in a line-by-line fashion, while in planewave (PW) mode, the full ROI is scanned at once. For the reconstruction, out-of-phased RF signals are accumulated after back-propagation. For example, the basic steps of conventional delay-and-sum (DAS) beamforming pipeline are: (1) time-of-flight correction in which RF measurements are shifted in time, and (2) the summation of the time-corrected signal.

While the DAS beamformer is widely used due to its simplicity and robust reconstruction performance, it often suffers from poor resolution due to signal side-lobes. Moreover, DAS images are usually associated with image artifacts such as speckles, blurring, etc., whose removal have been one of the main research topics in US research. However, existing approaches using adaptive beamforming techniques [1, 2, 3, 4, 5, 6, 7, 8, 9] and compressed sensing [10, 11, 12, 13]

are computationally expensive and sensitive to hyperparameters, so a recent trend is using deep learning approaches in the design of beamformers and image processing algorithms

[14, 15, 16, 17, 18].

Fig. 1: Data processing pipeline for the proposed switchable deep beamformer. (a) Neural network input preparation step, (b) switchable deep beamformer, and (c) thresholding and display step for visualization.

For example, Luchies and Byram [14]

proposed a deep learning based frequency domain beamforming method for off-axis scattering suppression. In

[16], a beamformer is designed to produce speckle-free images from channel data. A major limitation of these methods [14, 16] is that the training data are usually obtained from simulation phantom and therefore recovery of accurate images is challenging especially for in-vivo data. On the other hand, recent deep beamformers [19, 20] utilizes in vivo data as target data for supervised training. For example, in the universal deep beamformer [19]

, neural network is trained to generate high resolution DAS, minimum variance beamformer (MVBF), and deconvolution ultrasound outputs from channel data. In another work

[17], a neural network was trained to mimic the output of MV beamformer for fast beamforming.

Although deep beamfomers provide impressive performance and ultra-fast reconstruction, one of the downsides of the deep beamformer is that we need a distinct model for each type of desired output. For instance, to obtained DAS output, a model is needed which mimics DAS; similarly, for MVBF a separate model is needed. Although the architecture of the model will be the same, separate weight is needed to be stored for each output type. Additionally, to enable a deep beamformer to generate output that matches to several post processing results, such as deblurring or speckle reduction, etc., it further increases the number of models to be stored.

To address this issue, here we propose a novel switchable

deep beamformer architecture using adaptive instance normalization (AdaIN) layers. AdaIN was originally proposed as an image style transfer method, in which the mean and variance of the feature vectors are replaced by those of the style reference image

[21]. Recently, AdaIN has attracted significant attention as a powerful component to control generative models. For example, in the StyleGAN [22], the network generated various realistic faces with the same inputs by simply adjusting the mean and variance of the feature maps using AdaIN. In StarGANv2 [23], AdaIN plays the key role in converting styles from input images. Furthermore, a recent study [24] has shown that the change of style by AdaIN is not a cosmetic change but a real one thanks to the important link to the optimal transport [25, 26] between the feature vector spaces.

Inspired by the success of AdaIN and its theoretical understanding, one of the most important contributions of this work is to demonstrate that a single deep beamformer with AdaIN layers can learn target images from various styles. Here, a “style” refers to a specific output processing, such as DAS, MVBF, deconvolution image, despeckled images, etc. Once the network is trained, the deep beamformer can then generate various style output by simply changing the AdaIN code. Furthermore, the AdaIN code generation can be easily done with a very light AdaIN code generator, so the additional memory overhead at the training step is minimal. Once the neural network is trained, we only need the AdaIN code without the generator, which makes the system very simple. As such, our switchable deep beamformer can be considered as the first, fully adaptive multi-purpose deep beamformer for the reconstruction of high-quality medical ultrasound images.

This paper is organized as follows. The deep beamformer and AdaIN are first reviewed in Section II, after which we explain how these techniques can be synergistically combined to produce a switchable deep beamformer in Section III. Section IV describes the data set and experimental setup. Experimental results are provided in Section V, which is followed by Conclusions in Section VI.

Fig. 2: Architecture of our switchable deep beamformer. Adaptive instance normalization (AdaIN) layer is added at the bottleneck layer, for which AdaIN code is generated by a separate AdaIN code generator network.

Ii Related works

Ii-a Deep Beamformer

Among the various forms of deep beamformers, here we review the universal deep beamformer (DeepBF) [18] as a representative form of deep beamformer, since our switchable deep beamformer is built upon this architecture.

In US, measured RF data is usually given as a three-dimensional cube as shown in Fig. 1(a), where and denote the number of scan lines (or transmit events (TE)), depth planes, and the number of probe elements, respectively. The time-delay corrected data cube is similarly denoted by , where

(1)

Then, the received channel data can be explicitly modeled as , where

(2)

and

(3)

where denotes the specific detector offset to indicate the active channel elements, which is determined for each scan line index , and is the aperture size. This implies that the dark triangular regions in as shown Fig. 1(a), which correspond to inactive receiver elements, are removed in constructing .

The standard delay and sum (DAS) beamformer for the -th scanline at the depth sample can be expressed as

(4)

where denotes a -dimensional column-vector of ones. On the other hand, DeepBF [18] generates multiple scan-lines in a depth simultaneously using multiple-depth channel data:

(5)

where denotes the output at the depth :

(6)

with referring to the target image at the -th scan line and the depth , and is a neural network parameterized by , is a channel data collected from, for example, seven depth planes around the depth :

(7)

where is defined as:

(8)

Then, the resulting neural network training is given by

(9)

where denotes the input channel data and the target image data at the depth index , and the superscript index is the training data set index. In fact, the training data can be obtained not only from different experiment data but also from different depth in the same frame [18].

One of the important advantages of the deep beamformer is that depending on the target data , the neural network parameter is trained accordingly to mimic target data. In [18], the authors used the DAS, minimum variance beamformer (MVBF), and deconvolution images as targets, so that deep beamformer can directly produce target domain images. However, as discussed before, separate network parameter should be stored for each target domain, demanding significant amount of scanner resources.

Ii-B Adaptive Instance Normalization (AdaIN)

AdaIN [21] is an extension of instance normalization [27], which goes beyond the classical role of the normalization methods for image style transfer. The key idea of AdaIN is that latent space exploration is possible by adjusting the mean and variance of the feature map. Specifically, style transfer is performed by matching the mean and variance of the feature map of the input image to those of the style image [21].

More specifically, let a multi-channel feature map be represented by

(10)

where refers to the -th column vector of , which represents the vectorized feature map of size of at the -th channel. Suppose, furthermore, the corresponding feature map for the style reference image is given by

(11)

Then, AdaIN changes the feature data for each channel using the following transform:

(12)

where

(13)

where is the -dimensional vector composed of 1, and and

are channel-wise mean and standard deviation (std) for

:

(14)

The AdaIN transform in (13) has an important link to optimal transport [25, 26]

. Specifically, consider the optimal transport scheme between two probability spaces

and , equipped with the Gaussian probability measure and , respectively, where and denote the mean and the covariance, respectively. Then, a recent study [24] showed that the optimal transport map from the measure to the measure with respect to Wasserstein-2 distance is given by

(15)

Therefore, if we assume the i.i.d. distribution, i.e.

and (16)

where

is the identity matrix, then the optimal transport map in (

15) can be simplified to

(17)

which is equivalent to the AdaIN transform in (13). Therefore, the AdaIN transform can be considered the minimal cost transport path when converting one feature map to another under i.i.d. Gaussian feature map assumption.

Iii Theory

Iii-a Switchable Deep Beamformer using AdaIN Layers

Inspired by the observation that AdaIN is an optimal transport plan between the feature layers, our goal is to implement multi-purpose deep beamformer using a single baseline network followed by optimal transport layers to specific target distributions. Specifically, we use DeepBF [18] as the baseline network and then use AdaIN transform to transport the features to different domain features.

More specifically, our switchable deep beamformer is formally defined as

(18)

where denotes the input RF channel data space composed of elements in (7), is the output space for the beamformed data in (5), and is the AdaIN code vector space, whose element is generated by the AdaIN code generator as

(19)

where is the target style code. AdaIN code generator takes the style code representing a target domain, and generate style mean and variance vectors required for that domain.

The architecture of the proposed switchable network is shown in Fig. 2

. In the baseline network, there are nine convolution blocks shown in blue and yellow colors. The blue blocks are composed of two convolution layer with ReLU activation function, while yellow block is composed of convolution, and Leaky ReLU blocks. All convolution layers uses stride value of

, with a 2-dimensional filter that has a dimension of , expect the last layer shown in green color which uses , with a 2-dimensional filter that has a dimension of . As shown in Fig. 2, there exists an AdaIN layer at the bottleneck layer. The AdaIN layers take a mean vector and a variance vector as input. During the training phase, the AdaIN layers take the output vectors from the AdaIN code generator in Fig. 2, which consists of fully connected layers and is trained together with the baseline neural network. For mean vector generation, a linear activation was used, and to avoid negative values of variance ReLU activation function was used for the variance generation.

At the inference phase, we only use the baseline network with the trained AdaIN codes , as the AdaIN code generator is no more necessary. Then, depending on the specific AdaIN code , our switchable deep beamformer output

becomes an image with target domain processing. Accordingly, the total memory requirement can be significantly reduced, and multiple processing can be realized by simply changing the code .

Iii-B Target Image Generations

As for multiple domain target reference data, in addition to DAS, we consider images from three post processing methods: (1) deconvolution (deblurring), (2) despeckle (denoising), and (3) the combination of the two. More details are as follows.

Iii-B1 Das

In contrast to [18] where IQ signal was used as target, for proposed switchable deep beamformer, we use the log-compressed absolute value of Hilbert transformed RF-sum as DAS target. Thanks to such changes, all the beamforming process is now learned by the neural network, and only thresholding for specific dynamic range can be performed for visualization as shown in Fig. 1(c). Similar approaches are taken for other processing as follows.

Iii-B2 Deblurring

Deconvolution method can help reduce the blur to improve spatial resolution of an ultrasonic imaging system. As shown in Figure 3, in US, a received signal is modeled as a convolution product of tissue reflectivity function (TRF) and point spread function (PSF), where TRF represents scatter’s acoustic properties, while the impulse response of the imaging system is modeled by PSF. The PSF is determined by the detector configurations, focused beam width, etc.

Fig. 3: Block diagram of deconvolution ultrasound imaging.

Then, the unknown TRF

is estimated by solving the following

minimization problem:

(20)

where is the Hilbert transformed DAS RF-sum, is the PSF, is a regularization parameter empirically chosen to control the tradeoff between sparsity and fitting fidelity.

In most practical cases, the complete knowledge of is not available, and therefore both unknown TRF and the PSF have to be estimated together, which is called the blind deconvolution problem. In this paper, we first estimate from , and then is estimated based on [28]. To train the proposed switchable beamformer, a log-compressed absolute value, , is used as a training reference.

Iii-B3 Denoising

Speckle noises are the granular patterns appears in US images, which is one of the a major reasons of quality degradation and removal of it can subsequently enhance the structural details in US images [29]. To design a speckle denoising beamformer, the log-absolute value of Hilbert transformed RF-sum was filtered using non-local low-rank (NLLR) method [29] to generate the reference data for supervised training. In NLLR as shown in Figure 4, the image is pre-processed to generate a guidance map and later non-local filtering operations are performed on the candidate patches that are selected using that guidance map. For further refinement of filtered patches, a truncated nuclear norm (TWNN) and structured sparsity criterion are used [30, 31].

Fig. 4: Block diagram of non-local low-rank (NLLR) filtering-based speckle removal method.

Iii-B4 Deconvolution + Denoising

To generate the deconvolution and despeckle reference images, we applied the NLLR filter on deconvoluted log-absolute RF-sum to generate deconvolution+despeckle output.

Iii-C Switchable Deep Beamformer Training

Let the data set denotes the training data set collected across all depth planes and style , where is defined in (7) and is the target image for a given style at the depth . Then, the resulting neural network training for the proposed switchable deep beamformer is given by

(21)

For each target style , the AdaIN code generator network expands the style input to produce desired size AdaIN code with the help of multiple fully-connected layers. In particular, the style code is given by to generate four different vectors representing DAS, Despeckle, Deconvolution, and Deconvolution+Despeckle targets, respectively. Once the neural network is trained and AdaIN codes are learned, our switchable deep beamformer can generate multiple domain outputs by simply changing the AdaIN code.

Iv Method

Iv-a Dataset

Three dataset were collected using E-CUBE 12R US system with L3-12H linear array probe. The first dataset was acquired with an operating frequency of 8.5MHz from the carotid/thyroid regions of volunteers, in which frames were scanned from four different regions (carotid artery, trachea, thyroid right and left lobes) of each subject providing a set of images. The second dataset was acquired with an operating of 10MHz from the calf and forearm region of a volunteers, where frames were acquired from each region resulting in a set of frames. The last dataset was generated using ATF-539 phantom with an operating frequency of 8.5MHz, 10MHz and 11.5MHz, different regions of phantom were scanned and in total frames were recorded. The length of the lateral axis for all samples was , the axial depth was varied from to and the focal depth was also adjusted accordingly. For all samples active channels were used to acquire scan-lines using standard single-line-acquisition (SLA) method.

Iv-B Network training

The input and output data configurations are shown in Fig. 1(a) and (b), respectively. Similar to the DeepBF[18], herein we use a multi-line, multi-depth approach to form a neural network input. First, a raw RF-signal cube was acquired which is later time-delayed to correct the time-of-flight in out-of-phase signals. The RF data cube depends on the axial depth of the image, for this study, we used variable depths ranges from to axial depth planes representing axial samples. From a time corrected cube a subset of size is selected to produce output.

To train our model we used input output pairs of channels data cube and targeted depth plane as described in (21). The model was trained using a set of cubes, which are randomly selected from frames of a subject, which are divided into samples for training and samples for validation. The remaining dataset of carotid/thyroid, phantom were used as a test data. In addition, to see the generalization capability of the algorithm, frame data from totally different anatomical regions (forearm/calf muscles) were used as an independent dataset.

The network was implemented with Python3, using Keras and TensorFlow platform. Specifically, for network training, the parameters were estimated by minimizing the

norm loss function using a Adam stochastic gradient descent. The learning rate started from

and exponentially decreased with the decay steps at a decay rate of for epochs. The training was stopped after epochs using early-stopping method with patience steps of .

Iv-C Comparison methods

To compare our proposed switchable deep beamformer method we choose standard DAS, and DeepBF method [18]. The DAS is a simple sum of channel data as described in (4). DeepBF [18] was trained to produce deconvolution (Deconv.BF).

For the DeepBF training, the target data was generated as defined in Section III-B. For comparison, default configuration of the DeepBF in [18] was used, where convolution blocks each consisting of convolution layers with ReLU activation function and filters of kernel-size was used. The input cube was

and target output was

.

Iv-D Performance metrics

To evaluate the performance of our proposed beamforming method, we used three measures of contrast namely contrast-recovery (CR), contrast-to-noise ratio (CNR) and generalized contrast-to-noise ratio (GCNR).

The contrast in ultrasound images can be calculate with the help of two regions of interest defined as target () and background (). Typically binary masks are used to select two regions in an image, and in this study we prepare two separate masks for each image, and the selected regions are highlighted in the B-mode results.

The CR measure is the most common measure of contrast; however, it only considers the mean intensity difference between two regions. Since ultrasound images are naturally prone to noise (especially the speckle noise) which can degrade the visual quality, so a better measure of contrast is CNR. Mathematically the CR and CNR are defined as follows

(22)
(23)

where and are the local means, and the standard deviations of the target () and background () [32], respectively.

The CNR works well for most of the cases; however, it is scale variant and sensitive to the dynamic range, which make it difficult to interpret. Recently a better generalization of contrast statistics was proposed called generalized CNR (GCNR) [33]. The GCNR measure is based on the overlap between two distributions and mathematically it is defined as

(24)

where , and

are the probability distributions of target and background pixel value

respectively. In ideal case, target and background must be statistically independent and there should not be any overlap, resulting in perfect score , while in the worst case when they completely overlap, the GCNR will be .

V Experimental Results

In this study we verified our switchable deep beamformer using three different dataset mentioned in Section IV and compared the results with DAS, and DeepBF with various targets [18]. All the figures are scaled in the dynamic range of dB.

V-a Qualitative Comparison

V-A1 Tissue mimicking phantom

To validate the reconstruction performance of the proposed switchable deep beamformer method, in Fig. 5, we showed the results of two phantom images generated from the anechoic and hyper-echoic regions. From the results it can be seen that the proposed model can generate four different types of output depending on the style code. Interestingly, in all cases the visual quality is either comparable or even better than the comparative algorithms. For the given image, two regions of interests shown in red and green colors were selected to calculate the CR, CNR and GCNR statistics. With a careful observation, we can see that there is speckle noise in DAS, and switchable deep beamformer DAS output, which is further enhanced by deblurring filtration. The deblurring filtration improves the sharpness of the image; however, it also increases the noise, which results in reduced CNR and GNCR. Although the contrast statistics of deconvolution beamformers is not good but the image looks much sharpers. Especially in hyperechoic case where standard methods shows blurry images with washout artifacts.

The speckle noise can be reduced using switchable deep beamformer (Despeckle): as it can be seen in contrast statistics and example image in Fig. 5, the speckle patterns in the background and target are reduced. To obtain noise-free sharp image the switchable deep beamformer can be switched to (Deconv + Despeckle), which makes the output image notably sharper with reduced speckle noise.

Fig. 5: Reconstruction results of standard DAS-BF, Deconv.BF, and the proposed switchable deep beamformer for tissue mimicking phatom data from anechoic and hyperechoic regions.

V-A2 In-vivo test data

In Fig. 6, we compared two in-vivo examples scanned from the carotid and artery regions using MHz center frequency. Here again the proposed switchable deep beamformer achieves better performance compared to DAS and Deconv.BF methods especially in (Deconvolution/DeSpeckle) case which looks much sharper and have relatively less noise.

Fig. 6: Reconstruction results of standard DAS-BF, Deconv.BF, and the proposed switchable deep beamformer for in-vivo scans from carotid and artery regions.

To further test the reconstruction quality of switchable deep beamformer in Fig. 7, we showed four additional results. From the figures it can be easily seen that the switchable deep beamformer can perform different tasks such as simple image formation, deconvolution, speckle suppression, and noise-suppressed deconvolution effectively.

Fig. 7: Reconstruction results of standard DAS-BF, Deconv.BF, and the proposed switchable deep beamformer for in-vivo scans from trachea, carotid, thyroid (central lobe), and artery region

V-A3 Results using in-vivo independent data

In this experiment we used a dataset acquired from two completely different body parts scanned using MHz operating frequency. In Fig. 8, we showed the results of two in vivo examples from calf and forearm regions. The images are generated using standard DAS, Deconv.BF, and the proposed switchable deep beamformer methods. In Fig. 8, it can be easily seen that our method perform all four tasks effectively using a single model. In despeckle beamformer case, the switchable deep beamformer preserves the structural details while notably reduce the noise level. Interestingly, with combined decovolution and despeckle, we can achieve two tasks (deblurring and denoising) tasks simultaneously.

Fig. 8: Reconstruction results of standard DAS-BF, Deconv.BF, and the proposed switchable deep beamformer for in-vivo scans from calf and forearm regions.

It is also noteworthy to point-out that the switchable deep beamformer was only trained using only a subset of RF-data acquired from frames of a carotid/thyroid regions of a subject scanned with MHz operating frequency. However, the performance in very diverse test scenarios is still remarkable, which clearly shows the generalization power of the proposed method.

V-B Quantitative comparison

To quantify the performance gain we compared the CR, CNR, and GCNR statistics of reconstructed B-mode images for all three dataset. Quantitatively CR, CNR, and GCNR values of switchable deep beamformer were slightly improved compared to the existing methods. Table I shows the comparison of DAS, Deconv.BF and proposed switchable deep beamformer method on all three dataset. In terms of CR, CNR and GCNR, the overall performance of switchable deep beamformer is remarkable. In carotid, thyroid, artery & trachea dataset, for deconvolution switchable deep beamformer produce highest CR among all while maintaining the noise level lower than the Deconv.BF. A similar trend can be seen in other dataset.

As expected the CNR and GCNR statistics of Despeckle beamformer are highest as these measures are based on the sum of noise and background noise variance. Since Despeckle beamformer effectively reduce the noise, it produces best statistics for noise based contrast measures. One interesting feature of the proposed method is its simultaneous deblurring and denoising capability. In Deconvolution+Despeckle mode, proposed method performs two competing tasks of deblurring and denoising, and surprisingly it enhances the CR statistics while maintaining the noise levels.

Interestingly, the proposed switchable beamformer also enhances the quality of B-mode images in DAS mode. In particular, for dataset , , and the GCNR values of proposed method in DAS mode are , , and units higher than the conventional DAS, respectively. Similarly, the deconvolution mode of switchable beamformer also outperform the Deconv.BF method. In particular, for dataset and it produced , and units higher GCNR. While for dataset the results are quite comparable.

Since, the encoder of the proposed switching beamformer was trained to produce a common representation for all types of target output, while the switching function was performed on decoder side, one possible reason for the enhanced performance would be that the network can learn better representation of the channel data through separate paths of forward and skip connections for contrast and speckle patterns, respectively. This will help the decoder to filter particular representation combination depending on the AdaIN encoded output.

Beamforming CR (dB) CNR GCNR
Method a b c a b c a b c
DAS 18.53 16.97 24.92 2.29 2.17 2.63 0.824 0.785 0.923
Switchable DeepBF (DAS) 18.81 15.77 24.71 2.45 2.30 2.63 0.844 0.796 0.928
(Despeckle) 18.13 15.49 23.95 2.92 2.59 3.02 0.888 0.821 0.958
(Deconvolution) 20.13 13.70 26.79 2.00 1.82 2.20 0.773 0.790 0.858
(Deconvolution + Despeckle) 20.02 13.63 26.31 2.29 2.04 2.43 0.822 0.811 0.899
Deconv.BF 19.78 17.27 27.55 1.93 1.92 2.37 0.757 0.777 0.884

Test: carotid, thyroid, artery & trachea, Test: tissue mimicking phantom, Independent: calf & forearm

TABLE I: Comparison of performance statistics with different beamformers

V-C Computational time

One big advantage of proposed switchable deep beamformer method is its reduced complexity, which allows for easily adaptation as only one model is used for various reconstruction tasks. This also allows fast image reconstruction times of ultrasound images with desired post processing filtration effects. The average reconstruction time for each depth planes is around (milliseconds), which is several times lower than the conventional beamformer+filtration (deblurring/despeckle) methods. Interestingly, with combined learning of multiple tasks through AdaIN, the proposed model can achieve DAS and Deconv.BF like results with relatively shallower model compared to DeepBF which took around (milliseconds). The implementation method used in current study can be optimized by replacing the fully-connected neural network with fixed style vector values and optimized implementation for parallel reconstruction of multiple depth planes.

Vi Conclusion

In this paper, we presented a novel switchable deep beamformer to generate desired quality B-mode ultrasound image directly from channel data with different post filtration effects by simply changing the AdaIN code. The proposed method is purely a data-driven method which harnesses the information in the spatio-temporal dimensions along the scan-line, axial depth and active channels measurement, which help in generating improved quality B-mode images compared to standard beamforming methods. In the proposed method, a single network was trained to produce four types of output including DAS, deconvolution, despeckle and deconvolution + despeckle directly from the channel data. The proposed method improved the contrast of B-modes images by preserving the dynamic range and structural details of the RF signal in both the phantom and in vivo scans, and can effectively perform computationally expensive post filtration task during the beamforming process. This post training adaptivity through AdaIN codes allow easily implementation; and therefore, this method can be an important platform for data-driven ultrasound beamformer.

References

  • [1] F. Viola and W. F. Walker, “Adaptive signal processing in medical ultrasound beamforming,” in IEEE Ultrasonics Symposium, 2005., vol. 4, Sep. 2005, pp. 1980–1983.
  • [2] J. Capon, “High-resolution frequency-wavenumber spectrum analysis,” Proceedings of the IEEE, vol. 57, no. 8, pp. 1408–1418, Aug 1969.
  • [3] F. Vignon and M. R. Burcher, “Capon beamforming in medical ultrasound imaging with focused beams,” IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, vol. 55, no. 3, pp. 619–628, March 2008.
  • [4]

    K. Kim, S. Park, J. Kim, S. Park, and M. Bae, “A fast minimum variance beamforming method using principal component analysis,”

    IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, vol. 61, no. 6, pp. 930–945, June 2014.
  • [5] W. Chen, Y. Zhao, and J. Gao, “Improved capon beamforming algorithm by using inverse covariance matrix calculation,” in IET International Radar Conference 2013, April 2013, pp. 1–6.
  • [6] C. . C. Nilsen and I. Hafizovic, “Beamspace adaptive beamforming for ultrasound imaging,” IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, vol. 56, no. 10, pp. 2187–2197, October 2009.
  • [7] A. M. Deylami and B. M. Asl, “A fast and robust beamspace adaptive beamformer for medical ultrasound imaging,” IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, vol. 64, no. 6, pp. 947–958, June 2017.
  • [8] A. C. Jensen and A. Austeng, “An approach to multibeam covariance matrices for adaptive beamforming in ultrasonography,” IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, vol. 59, no. 6, pp. 1139–1148, June 2012.
  • [9] ——, “The iterative adaptive approach in medical ultrasound imaging,” IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, vol. 61, no. 10, pp. 1688–1697, Oct 2014.
  • [10]

    K. H. Jin, Y. S. Han, and J. C. Ye, “Compressive dynamic aperture b-mode ultrasound imaging using annihilating filter-based low-rank interpolation,” in

    2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI).   IEEE, 2016, pp. 1009–1012.
  • [11] Y. H. Yoon, S. Khan, J. Huh, J. C. Ye et al., “Efficient b-mode ultrasound image reconstruction from sub-sampled rf data using deep learning,” IEEE transactions on medical imaging, 2018.
  • [12] A. Burshtein, M. Birk, T. Chernyakova, A. Eilam, A. Kempinski, and Y. C. Eldar, “Sub-nyquist sampling and fourier domain beamforming in volumetric ultrasound imaging,” IEEE Trans. Ultrason., Ferroelectr., Freq. Control, vol. 63, no. 5, pp. 703–716, 2016.
  • [13] M. F. Schiffner and G. Schmitz, “Fast pulse-echo ultrasound imaging employing compressive sensing,” in Proc. IEEE Int. Ultrasonics Symposium (IUS), 2011, pp. 688–691.
  • [14] A. C. Luchies and B. C. Byram, “Deep neural networks for ultrasound beamforming,” IEEE transactions on medical imaging, vol. 37, no. 9, pp. 2010–2021, 2018.
  • [15] A. A. Nair, T. D. Tran, A. Reiter, and M. A. L. Bell, “A deep learning based alternative to beamforming ultrasound images,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).   IEEE, 2018, pp. 3359–3363.
  • [16] D. Hyun, L. L. Brickson, K. T. Looby, and J. J. Dahl, “Beamforming and speckle reduction using neural networks,” IEEE transactions on ultrasonics, ferroelectrics, and frequency control, vol. 66, no. 5, pp. 898–910, 2019.
  • [17] B. Luijten, R. Cohen, F. J. De Bruijn, H. A. W. Schmeitz, M. Mischi, Y. C. Eldar, and R. J. G. Van Sloun, “Adaptive ultrasound beamforming using deep learning,” IEEE Transactions on Medical Imaging, pp. 1–1, 2020.
  • [18] S. Khan, J. Huh, and J. C. Ye, “Adaptive and compressive beamforming using deep learning for medical ultrasound,” IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, 2020.
  • [19] ——, “Deep learning-based universal beamformer for ultrasound imaging,” in International Conference on Medical Image Computing and Computer-Assisted Intervention.   Springer, Cham, 2019, pp. 619–627.
  • [20] R. J. van Sloun, R. Cohen, and Y. C. Eldar, “Deep learning in ultrasound imaging,” Proceedings of the IEEE, vol. 108, no. 1, pp. 11–29, 2019.
  • [21] X. Huang and S. Belongie, “Arbitrary style transfer in real-time with adaptive instance normalization,” in

    Proceedings of the IEEE International Conference on Computer Vision

    , 2017, pp. 1501–1510.
  • [22]

    T. Karras, S. Laine, and T. Aila, “A style-based generator architecture for generative adversarial networks,” in

    Proceedings of the IEEE conference on computer vision and pattern recognition

    , 2019, pp. 4401–4410.
  • [23] Y. Choi, Y. Uh, J. Yoo, and J.-W. Ha, “Stargan v2: Diverse image synthesis for multiple domains,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 8188–8197.
  • [24] Y. Mroueh, “Wasserstein style transfer,” arXiv preprint arXiv:1905.12828, 2019.
  • [25] G. Peyré, M. Cuturi et al.

    , “Computational optimal transport: With applications to data science,”

    Foundations and Trends® in Machine Learning

    , vol. 11, no. 5-6, pp. 355–607, 2019.
  • [26] C. Villani, Optimal transport: old and new.   Springer Science & Business Media, 2008, vol. 338.
  • [27] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Instance normalization: The missing ingredient for fast stylization,” arXiv preprint arXiv:1607.08022, 2016.
  • [28] J. Duan, H. Zhong, B. Jing, S. Zhang, and M. Wan, “Increasing axial resolution of ultrasonic imaging with a joint sparse representation model,” IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, vol. 63, no. 12, pp. 2045–2056, Dec 2016.
  • [29] L. Zhu, C. Fu, M. S. Brown, and P. Heng, “A non-local low-rank framework for ultrasound speckle reduction,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017, pp. 493–501.
  • [30] D. Zhang, Y. Hu, J. Ye, X. Li, and X. He, “Matrix completion by truncated nuclear norm regularization,” in 2012 IEEE Conference on Computer Vision and Pattern Recognition.   IEEE, 2012, pp. 2192–2199.
  • [31] S. Gu, L. Zhang, W. Zuo, and X. Feng, “Weighted nuclear norm minimization with application to image denoising,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2014, pp. 2862–2869.
  • [32] R. M. Rangayyan, Biomedical Image Analysis, ser. Biomedical Engineering, M. R. Neuman, Ed.   Boca Raton, Florida: CRC Press, 2005.
  • [33] A. Rodriguez-Molares, O. M. H. Rindal, J. D’hooge, S.-E. Måsøy, A. Austeng, M. A. L. Bell, and H. Torp, “The generalized contrast-to-noise ratio: a formal definition for lesion detectability,” IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, vol. 67, no. 4, pp. 745–759, 2019.