DeepAI
Log In Sign Up

Learning Sub-Sampling and Signal Recovery with Applications in Ultrasound Imaging

Limitations on bandwidth and power consumption impose strict bounds on data rates of diagnostic imaging systems. Consequently, the design of suitable (i.e. task- and data-aware) compression and reconstruction techniques has attracted considerable attention in recent years. Compressed sensing emerged as a popular framework for sparse signal reconstruction from a small set of compressed measurements. However, typical compressed sensing designs measure a (non)linearly weighted combination of all input signal elements, which poses practical challenges. These designs are also not necessarily task-optimal. In addition, real-time recovery is hampered by the iterative and time-consuming nature of sparse recovery algorithms. Recently, deep learning methods have shown promise for fast recovery from compressed measurements, but the design of adequate and practical sensing strategies remains a challenge. Here, we propose a deep learning solution, termed LASSY (LeArning Sub-Sampling and recoverY), that jointly learns a task-driven sub-sampling pattern and subsequent reconstruction model. The learned sub-sampling patterns are straightforwardly implementable, and based on the task at hand. LASSY's effectiveness is demonstrated in-silico for sparse signal recovery from partial Fourier measurements, and in-vivo for both anatomical-image and motion (Doppler) reconstruction from sub-sampled medical ultrasound imaging data.

READ FULL TEXT VIEW PDF

page 1

page 7

page 9

05/24/2019

Do log factors matter? On optimal wavelet approximation and the foundations of compressed sensing

A signature result in compressed sensing is that Gaussian random samplin...
10/31/2013

Robust Compressed Sensing and Sparse Coding with the Difference Map

In compressed sensing, we wish to reconstruct a sparse signal x from obs...
02/09/2012

Compressed Beamforming in Ultrasound Imaging

Emerging sonography techniques often require increasing the number of tr...
03/01/2021

Deep Unfolded Recovery of Sub-Nyquist Sampled Ultrasound Image

The most common technique for generating B-mode ultrasound (US) images i...
09/28/2022

Learning Filter-Based Compressed Blind-Deconvolution

The problem of sparse multichannel blind deconvolution (S-MBD) arises fr...
08/02/2021

Bespoke Fractal Sampling Patterns for Discrete Fourier Space via the Kaleidoscope Transform

Sampling strategies are important for sparse imaging methodologies, espe...
12/02/2018

Link Delay Estimation Using Sparse Recovery for Dynamic Network Tomography

When the scale of communication networks has been growing rapidly in the...

I Introduction

Advanced medical imaging techniques require transfer and storage of large amounts of data. Due to limited bandwidth and storage capacity, the raw sensor data must be compressed prior to its transfer to the backend system. Data compression, undersampling, and subsequent reconstruction techniques have been an active area of research for medical imaging modalities such as computed tomography (CT) imaging [42, 12, 14, 45], ultrasound CT imaging [49], ultrasound imaging [31, 50], and magnetic resonance imaging (MRI)[35, 34]. In this paper, we propose a framework for learning a task-driven sub-sampling and reconstruction method that permits reduction of sensor data rates, while retaining the information required to perform a given (imaging) task.

Among diagnostic imaging options, ultrasound imaging is an increasingly used modality, owing to its portability, cost-effectiveness, excellent temporal resolution, minimal invasiveness, and radiation-free nature. Compact, portable, and wireless ultrasound probes are emerging[6], enabling ‘pocket-sized’ devices. Also transducers are becoming miniaturized, which facilitates e.g. in-body imaging for interventional applications. As a consequence, available bandwidth is limited due to either wireless data transfer or data being transferred over a thin catheter in case of in-body applications. At the same time, emerging ultrafast 3D ultrasound imaging techniques[41, 44] cause data rates to drastically grow, which in turn poses even higher demands on the probe-to-system communication. Given these challenges, ultrasound imaging serves as an excellent candidate for evaluating the effectiveness of the framework that we will introduce.

Commonly used techniques to reduce data rates in 2D and 3D echography applications are micro-beamforming[29, 51] and slow-time111In ultrasound imaging a distinction is made between slow-time and fast-time: slow-time refers to a sequence of snapshots (i.e., across multiple transmit/receive events), at the pulse repetition rate, whereas fast-time refers to samples along depth. multiplexing. The former compresses data from multiple (adjacent) transducer elements (i.e. channels) into a single focused line, thereby virtually reducing the number of receive channels. While effective, this impairs the attainable resolution and image quality. The latter only communicates a subset of the channel signals to the backend of the system for every slow-time transmission. This comes at the cost of reduced frame rates.

Compressed sensing (CS) permits low data rate sensing (below the Nyquist rate) with strong signal recovery guarantees under specific conditions [7, 8, 10, 9, 20]. In CS, a sparse signal is to be recovered from measurements that are taken at a sub-Nyquist rate through a sensing matrix : , with , .

should preserve distance between distant signal vectors, i.e. it should satisfy the restricted isometry property (RIP)

[11, 8].

Proven (RIP-compliant) designs for take randomly-weighted linear combinations of input vector elements [10, 20]. Unfortunately, such designs often impose challenges regarding practical implementability. For example, in ultrasound imaging, sensing weighted combinations of slow-time frames would require an, often unfeasible and undesirably, large temporal signal support (including past and future values), and measuring linear combinations of channel signals imposes strong connectivity challenges. Alternatively, sampling a random subset of Fourier coefficients was also shown to be RIP-compliant [10, 20]. Whenever measuring in the Fourier domain is possible (e.g. in MRI), such partial Fourier measurements alleviate the above challenges.

After sensing, signal recovery in CS is typically achieved through proximal gradient schemes, such as the Iterative Shrinkage and Thresholding algorithm (ISTA)[19]. Although proximal gradient schemes are effective tools for solving non-differentiable convex optimization problems, in practice, their performance is greatly dependent on tuning of the thresholding parameter and their time-consuming iterative nature makes them less suitable for real-time applications. Recently, a number of deep learning approaches have been proposed for fast signal or image reconstruction in CS [40, 28]

, showing that deep neural networks can serve as a powerful alternative to conventional recovery techniques.

Inspired by both the challenge of finding adequate context-specific sensing matrices, and the given deep learning approaches for signal recovery, we present a deep learning solution that jointly learns a context- and task-based sub-sampling pattern and a corresponding signal reconstruction method. This approach is referred to as LeArning Sub-Sampling and RecoverY (LASSY). Efficient learning by error backpropagation is enabled through the adoption of the Gumbel-Softmax distribution

[25], that circumvents the inherently non-differentiable nature of sampling. We demonstrate LASSY’s effectiveness for signal recovery from both partial Fourier measurements and sub-sampled in-vivo ultrasound radio-frequency (RF) data.

The remainder of this paper is organized as follows, we start by providing some related work in Sec. II, followed by the general framework of LASSY in Sec. III-A. Sections III-B and III-C respectively elaborate on the sub-sampling strategy and signal recovery method of LASSY. The training strategy is described in Sec. III-D. Section IV-A demonstrates LASSY on a common Fourier domain sub-sampling problem. Its applications in ultrasound imaging are subsequently described in Secs. IV-B and IV-C. Results are given in Sec. V, which are discussed in Sec. VI. Final conclusions are drawn in Sec. VII.

Ii Related work

In this section we briefly list recent applications of conventional CS techniques for medical imaging that sub-sample the data. We then give promising applications of sparse arrays. These examples highlight the potential relevance for learning a task-driven sub-sampling pattern across a number of applications. The recent developments in deep learning for CS, that we discuss lastly, show state-of-the-art methods for learning-based data compression.

Ii-a Compressed sensing in medical imaging

Several CS approaches have been introduced for various medical imaging applications. In MRI, CS is applied by randomly sub-sampling the K-space [34, 35]

, i.e. the 2D spatial Fourier transform of the image. The authors of

[46] extend this to sub-sampling in the K-time space, while preserving qualitative image reconstructions using their k-t BLAST and k-t SENSE algorithms for one coil and multiple coils, respectively. Likewise, CS has spurred low-dose X-ray CT through image reconstruction from sub-sampled projection measurements [14, 45], and the authors of [31] show good reconstruction results after sub-sampling 3D US data over RF lines. In [50, 13], the authors apply CS to ultrasound imaging by passing the RF channel signals through analog sum-of-sinc filters, permitting sampling of a partial set of Fourier coefficients. Related to this, we demonstrate how LASSY permits learning of partial Fourier coefficients in Sec. V-A.

Ii-B Sparse arrays

Significant research efforts have been invested in exploration of adequate sparse array designs [30]. Examples in medical ultrasound imaging are a non-uniform slow-time transmission scheme for spectral Doppler [17] and sparse arrays for reduction of the required number of channels for B-mode imaging222In ultrasound imaging, B-mode refers to “brightness mode”, a 2D intensity image at a single point in time., based on sparse periodic arrays [5] or sum coarrays [16]. In Secs. V-B and V-C, we show how LASSY enables learning of these slow-time and array sampling patterns for ultrasound imaging in a task-based fashion.

Ii-C Deep learning for compressed sensing

Recently, a number of deep learning approaches have been proposed for fast signal or image reconstruction in CS [40, 28], showing that deep neural networks can serve as powerful signal or image recovery methods. The authors of[40, 38, 37, 3, 2, 32] extend learning beyond signal recovery, and simultaneously train signal compression methods. However, they all rely on taking (randomly weighted) (non)linear combinations of elements from the input vector, making them challenging to implement in hardware. Instead, LASSY is based on sub-sampling, which is straightforwardly implementable and applicable across the applications given in Secs. II-A and II-B.

Iii Methods

Iii-a General framework

In LASSY, we consider a signal vector  that we wish to sub-sample through a binary sub-sampling matrix parametrized by , to yield a measurement vector333 and can also be higher dimensional. In that case all given formulas are applied on the dimension in which we want to sub-sample . , with :

(1)

We subsequently aim to decode into , some function of the original signal vector in which we are interested (i.e. the task):

(2)

To this end, we adopt a (potentially nonlinear) differentiable function approximator parametrized by a set of parameters :

(3)

where denotes the recovery of from the sub-sampled measurements . The function may for instance be a neural network. Matrix is constrained to have a row-wise norm equal to 1, i.e. every row contains exactly one non-zero element. As such, selects a subset of (out of ) elements from input vector .

To permit joint learning of an adequate sub-sampling pattern for and recovery of through by backpropagation, we will introduce a probabilistic sampling strategy, on which we elaborate in the next section.

Iii-B Learning sub-sampling

Each row of , with

, is defined as a one-hot encoding

444The one-hot encoding, , of a categorical random variable with classes results in a unit-vector of length . Exactly one element is non-zero and its index corresponds to the class of the drawn sample.

of an independent categorical random variable

(4)

where   is a vector containing

class probabilities. Note that

thus represents the probability of sampling the entry in at the measurement . We reparametrize

using unnormalized log-probabilities (logits)

, such that

(5)

where is the unnormalized logit of .

To enable sampling from the categorical probability distribution, we leverage the Gumbel-max trick

[23], i.e. sampling is reparametrized into a function of the distribution parameters and a Gumbel noise vector , with , i.i.d.. A realization of is then defined as:

(6)

The subscript WR denotes sampling without replacement, which we implement across to , i.e. the same sample is never selected more than once. This is achieved by dynamically excluding the categories that have already been sampled, and renormalizing the logits of the resulting distribution. Each row can now be defined as:

(7)

We define as the row of a trainable matrix that contains the unnormalized logits of all distributions. To permit optimization of by backpropagation, we require to exist . Since

is a non-differentiable operator, we adopt the Straight-Through Gumbel Estimator

[25, 36] as a surrogate for :

(8)

with (row operator) as a continuous differentiable approximation of the one-hot encoded operation. We refer to sampling using the function as soft sampling. Its temperature parameter serves as a gradient distributor over multiple entries (i.e. logits) in .

In the limit of , soft sampling approaches the one-hot encoded operator in (III-B)[25, 36], which results in the final trainable sub-sampling pattern of LASSY:

(9)
(10)

with .

Iii-C Signal recovery by deep learning

LASSY’s signal recovery method is implemented using a neural network , with trainable parameters . Suitable network architectures are application-specific and therefore described per application in Sec. IV. A block diagram of the total LASSY algorithm is given in Fig. 1.

Fig. 1: An overview of LASSY, with gradient backpropagation depicted in red. The symbol indicates a matrix multiplication between and the dimension to be sub-sampled of . is a subset of , which is in turn used to approximate resulting in .

Iii-D Training strategy

We train model parameters and by minimizing the mean squared error (MSE) between the model’s output and the target

, assuming normally distributed prediction errors. To prevent overfitting and exploding gradients, the problem is regularized by adding an

 penalty on . Besides, we promote training towards one-hot distributions by penalizing convergence towards high entropy distributions using:

(11)

with defined as in (5).

The resulting optimization problem can be written as:

(12)

with

(13)

and

(14)

where the input and target vectors, i.e. and respectively, follow data-generating distribution . Penalty multipliers and weigh the importance of the different penalties.

The Adam solver with hyperparameters

, , and [27] is used to stochastically optimize (12). In practice, we found that the appropriate learning rates for and were different. As such, two separate learning rates were used, i.e. and , with . The adopted values are reported in Sec. IV, along with the values for the penalty multipliers and , and the number of used iterations for training. We define one iteration as a trainable parameter update using one mini-batch of data.

The temperature parameter in (10) is initialized at and gradually lowered to during training. The initialization of logits matrix , promotes preservation of the original order of elements in . As such, all elements , with and are initialized according to:

(15)

with constants and , and i.i.d..

The pseudocode of LASSY is shown in Algorithm 1

. LASSY was implemented in Python using Keras

[15]

with a TensorFlow backend

[1]. Training and inference were performed on a Titan XP (NVIDIA, Santa Clara, CA).

0:  Training dataset , Number of iterations , , , Initialized trainable parameters and .
0:  Trained logits matrix and reconstruction network parameters .
   - Compute
  for  to  do
      - Draw mini-batches : a random subset of
      - Compute fully sampled target:
     - Draw Gumbel noise vectors for
      - Compute using:          for
      - Sub-sample the signal:
      - Compute reconstruction:
      - Compute loss using :
      - Set
      - Redefine
     - Use Adam optimizer to update and
  end for
Algorithm 1 LeArning Sub-Sampling and RecoverY (LASSY)

Iv Validation methodology

Iv-a Partial Fourier sampling of sparse signals

Many practical CS applications require signal reconstruction from partial Fourier measurements[39, 35, 34], and we therefore first demonstrate LASSY in such a scenario. To that end, we synthetically generate random K-sparse signal vectors , with , which we subsequently Fourier-transform to yield the signal that we aim to partially sample555For each experiment the length of the signal (in the dimension to be sub-sampled) was set to the closest integer multiple of the sub-sampling factor, e.g. 126 for factor 6.. Here, the measurement , with , is a sub-sampled set (learned by ) of Fourier coefficients in , and the task is to recover the sparse signal, , from measurement .

We compare the reconstruction performance of using this task-based learned sub-sampling pattern with performances of using an untrained fixed uniform and a random sub-sampling strategy. The latter is typically adopted in CS[10, 20].

We adopt a specific recovery network architecture that is inspired by the proximal gradient ISTA scheme[19]; it unrolls the iterative solution of ISTA as a 2-layer feedforward neural network with trainable (thresholding) parameters[22]. To prevent dying gradients during backpropagation, we replace the conventional soft-thresholding operators in this learned ISTA (LISTA) method by a sigmoid-based soft-thresholding operator [4].

We train for 96,000 iterations across mini-batches of 16 randomly generated Fourier-transformed data vectors. The learning rates and are set at and , and the penalty multipliers and at 0.0 and , respectively.

Iv-B Slow-time sub-sampling in ultrasound imaging

Iv-B1 Data acquisition and pre-processing

Sequential (slow-time) ultrasound data were acquired from an in-vivo open-chest intracardiac echography measurement of the right atrium of a porcine model. To that end, a 48-element linear array miniTEE s7-3t transducer with a pitch of 0.151 mm was used in combination with a Verasonics Vantrage system (Kirkland, WA). The center frequency for transmission and reception was 4.8 MHz and a 13-angle diverging wave scheme was used. The sampling rate of the received RF data was 19.2 MHz and coherently compounded beamformed frames (each with 68 scanlines) were collected at a frame rate of 474 Hz. These RF data frames were then demodulated into their in-phase and quadrature (IQ) components, and subsequently normalized between -1 and +1. Two such complete acquisitions were performed, of which one was used for training and one served as a hold-out test set.

Iv-B2 Tasks

Using the data acquired according to the procedure described in the previous section, we employ LASSY to learn a sub-sampling pattern for a sequence of IQ scanlines across slow-time and subsequently learn a specific task. We define two different tasks. First, we aim to recover the envelope of the beamformed RF signal in order to produce a standard gray-level ultrasound image. Here, the target is the magnitude of the (fully sampled) complex IQ data . Second, we explore LASSY for learning-based tissue motion estimation (i.e. Doppler recovery [47]) from the sub-sampled IQ scanlines across slow-time. In this case, the target is computed using the well-known Kasai auto-correlator [26]. We expect the two tasks to yield very distinct sampling patterns; where envelope construction is performed independently per frame, Doppler shifts are obtained by measuring phase shifts across the slow-time sequence.

Iv-B3 Recovery neural network architecture

For recovery of from the sub-sampled IQ scanlines in

, we employ a deep convolutional neural network

[21]

. The first 2 layers are 1D convolutional layers with respectively 256 and 128 features and window length = 5, assuming translational invariance across the fast-time dimension. Across slow-time, neurons are fully connected, since a similar invariance in this dimension may be lost after (possibly irregular) sub-sampling. After 2 such layers, spatial structure across both dimensions is assumed to be retained, and 4 2D convolutional layers with kernel sizes

and respectively 32, 64, 32, and 1 feature(s), are added. We use leaky rectified linear unit (leaky ReLU) activation functions (

) across all convolution layers, except the last, which has no activation function [52].

Fig. 2: Top row: Fixed uniform (a), fixed random (b) and learned (using LASSY) (c) sub-sampling patterns (sub-sampling factor ), with selected samples indicated in red. Bottom row: Signal recovery (blue) and ground truth signal (green).
Fig. 3:

Averaged MSE (with 1 standard deviation (SD) error bars in both directions) for signal reconstruction obtained on a randomly generated test set over the last 3200 iterations of training for the different sub-sampling factors.

Iv-B4 Training

For both tasks, the networks are stochastically optimized using the Adam solver, with settings as described in Sec. III-D, and learning rates and . We train for 320,000 iterations with mini-batches consisting of 16 randomly selected patches. Each patch contains 128 sequential slow-time samples of 256 fast-time IQ samples for a single radial scanline. The logits of the categorical distributions in matrix are initialized according to (15). Penalty multipliers and are set at and , respectively.

Iv-C Channel sub-sampling in ultrasound imaging

Iv-C1 Data acquisition and pre-processing

The same imaging setup as described in Sec. IV-B1 was used to demonstrate LASSY for sub-sampling across the 48-channel array, prior to beamforming. To facilitate the subsequent receive beamforming stage, we first pre-delay the channel signals for 68 different scanlines (with steering angles in ) [24]. Taking into account the transmit delay (i.e. the time-of-flight (TOF) between the virtual point source behind the array and the focus point in our diverging wave transmission scheme), and the receive delay (i.e. the TOF of the back-scattered wave between the focus point and the array element location, indexed by ), the total delay function for the central wave transmit is defined as[24]:

(16)

in which

(17)

and

(18)

Focal depth is denoted by , is the distance between the surface of the transducer array and the virtual point source behind the array, and are respectively the pitch and total number of channels of the array, and denotes the speed of sound in soft tissue. The adopted values for these parameters are:  mm,  mm,  mm, , and  m/s.

After computing 68 delayed signals per channel, we obtain a 4D dataset spanning slow-time frames, fast-time samples, channels, and radial scanlines. Note that pre-computing these delays is only done to accelerate training, and can in practice be performed after array sub-sampling. Finally, the pre-delayed RF channel signals were demodulated into their in-phase and quadrature (IQ) components, and thereafter normalized between -1 and +1.

Iv-C2 Tasks

We again distinguish two tasks, envelope reconstruction and tissue-motion (Doppler) estimation. Both target datasets are generated by first beamforming the fully sampled channel data, and then subsequently processing this as described in Sec. IV-B2.

Iv-C3 Recovery neural network architecture

For recovery of from the sub-sampled channel data in , we leverage a convolutional neural network. The network’s first 4 layers are 2D convolutional layers with kernels and respectively 64, 128, 64, and 48 features. Convolutions take place across the fast- and slow-time dimension, i.e. the channels are fully connected. Each of the convolutional layers is followed by a leaky ReLU activation function ()[52]. The network’s last layer is a fully connected layer across the (sub-sampled) channel dimension, which acts as a weighted summation and therefore shares similarities with the array apodization used in typical DAS beamforming [43].

Iv-C4 Training

For both envelope and Doppler reconstruction, the networks are stochastically optimized using Adam optimizer, with its settings as described in Sec. III-D. Learning rates and are set at and respectively, and we train for 160,000 iterations. Randomly selected mini-batches are used for training, each consisting of 16 patches with spanning 32 slow-time frames, 64 fast-time samples, 48 channels, and one radial scanline. Trainable matrix is initialized according to (15) and the penalty multipliers and are set at and , respectively.

V Results

V-a Partial Fourier sampling of sparse signals

Figure 2 displays sparse signal recovery from partial Fourier measurements for a uniform, random, and learned sub-sampling pattern (sub-sampling factor ) using LASSY. A quantitative evaluation of the recoveries for different sub-sampling factors is given in Fig. 3, showing that in all cases the MSE was lowest when using LASSY’s learned sub-sampling pattern. Uniform sub-sampling performed poorly due to aliasing, resulting in a repeated prediction pattern (see Fig. 2a-bottom). However, a (CS-inspired) random sampling pattern approached the performance of LASSY; interestingly, the learned pattern also exhibits (pseudo-random) irregular sampling (see Fig. 2c-top), and showed to be RIP-compliant.

Fig. 4: Envelope (a-e) and Doppler (f-j) reconstructions after sub-sampling across slow-time with a factor 4. (a,f) Fixed uniform sub-sampling pattern. (b,g) Reconstructed images after uniform sub-sampling. (c,h-top) Trained distributions of LASSY. (c,h-bottom) Draw from the distributions resulting in a learned sub-sampling pattern of LASSY. (d,i) Reconstructed images using LASSY. (e,j) Reference fully sampled B-mode (top) and M-mode (bottom) envelope and Doppler images. Dashed lines in the top and bottom image indicate the selected radial M-mode line and B-mode frame, respectively.

V-B Slow-time sub-sampling in ultrasound imaging

Figure 4 demonstrates envelope (a-e) and Doppler (f-j) reconstruction from uniform and learned slow-time sub-sampling patterns. Interestingly, LASSY’s learned patterns for both tasks are very distinct. For envelope reconstruction, the learned pattern exhibited an almost perfectly uniform sampling pattern. As such, the resulting reconstructions (see Fig. 4b and 4d) were found to be similar. This was consistent across all tested sub-sampling factors, displaying increased blurring of the gray-level images for higher sub-sampling factors in both methods. Their MSEs are compared in Fig. 5a.

Fig. 5: Averaged MSE (with 1 SD error bars in both directions) for envelope (a) and Doppler (b) reconstruction after sub-sampling slow-time frames, obtained on the hold-out test set over the last 3200 iterations of training. The black dashed lines indicate the MSE in case only zero values are predicted.

Unlike envelope reconstruction, Doppler recovery was greatly hampered by uniform sub-sampling for (Fig. 5b). Increasing the sub-sampling factor did not only lead to blurring, but strongly impaired Doppler estimation due to slow-time aliasing. Interestingly, LASSY yields a very distinct sub-sampling pattern (Fig. 4h-bottom), exhibiting an ‘ensemble’-type of sampling for . Similar patterns were clearly visible for the other tested sub-sampling factors as well.

The learned ‘ensemble’-style sub-sampling pattern efficiently captures high frequency slow-time signals due to tissue displacements (Doppler shifts) within ensembles, and relatively low frequency information (changes in Doppler shifts over time) among these ensembles. Consequently, LASSY’s performance degraded less for increasing sub-sampling factors, compared to a uniform sub-sampling strategy.

Using the trained network for inference on the test set (256 slow-time frames, containing 68 scanlines and 2048 fast-time samples) took on average 1.29 s (SD ms). Accordingly, the reconstruction network allows a reconstruction speed of 198 sub-sampled frames per second.

V-C Channel sub-sampling in ultrasound imaging

Figure 6 displays the envelope (a-e) and Doppler (f-j) reconstructions after channel selection and subsequent processing for each of these tasks, respectively. The results of using a learned sub-sampling pattern by LASSY are compared to those obtained by fixed uniform undersampling of the channel array. Using LASSY for learning slow-time sub-sampling patterns yielded near one-hot distributions for each of the measurements, whereas this was not the case for channel sub-sampling. As such, each realization of (see (9)) was slightly different. Figures 6c and 6h show histograms of the selected channels for realizations obtained in a Monte-Carlo fashion. A relative occurrence of 1 indicates that the specific channel was selected for each of the realizations of . The depicted sampling pattern below the histogram is one example of such a realization.

It can be seen that both for envelope and Doppler reconstruction, the center channels were found to carry most information for reconstruction. Interestingly, the relative-occurrence histogram for envelope reconstruction is wider than the one for Doppler reconstruction, indicating the need for a larger aperture of the transducer array in case of envelope reconstruction. Since a larger aperture imposes higher lateral resolution, the wider histogram for envelope reconstruction perfectly relates to the fact that lateral resolution is typically higher for gray-level images than for Doppler images.

Common practice is to design channel arrays in ultrasound probes that have a pitch which is half the signal’s wavelength in order to prevent grating lobes in the filed of view [43]. Increasing the pitch between channels by uniformly sub-sampling the channel array thus caused grating lobes to appear in the gray-level images, indicated by the white dashed lines in Fig. 6b-top. The relative angle of the grating lobes (with respect to the main beam) can be calculated as [43]:

(19)

where (mm) is the (original) pitch of the array, and (mm) is the wavelength of the signal.

Figures 7a and 7b respectively show the MSE values for both envelope and Doppler reconstruction using different sub-sampling factors. In both cases we can see that the MSE gradually increases for higher sub-sampling factors for both uniform sub-sampling and learned sub-sampling using LASSY. However, for all factors LASSY’s reconstruction outperformed reconstruction when using a uniform sub-sampling pattern.

Running inference on patches from the test set revealed an average reconstruction time of 36.7 ms (SD ms) for IQ data from 12 channels, steered towards 68 scanlines with 2048 fast-time samples at one point in (slow-)time, implying a frame reconstruction rate of 27 frames per second.

Fig. 6: Envelope (a-e) and Doppler (f-j) reconstructions after sub-sampling across channels with a factor 4. (a,f) Fixed uniform sub-sampling pattern. (b,g) Reconstructed images after uniform sub-sampling. (b-top) Reconstruction after uniform sub-sampling results in grating lobes in the gray-scale images, visible at an angle with respect to the main beam. (c,h) Histogram of Monte-Carlo sampling (without replacement) of all trained distributions by LASSY, with below the histogram one example of a realization. (d,i) Reconstructed images using LASSY. (e,j-top) Reference fully sampled B-mode (top) and M-mode (bottom) envelope and Doppler images. Dashed lines in the top and bottom image indicate the selected radial M-mode line and B-mode frame, respectively.
Fig. 7: Averaged MSE (with 1 SD error bars in both directions) for envelope (a) and Doppler (b) reconstruction after channel sub-sampling, obtained on the hold-out test set over the last 13,600 iterations of training. The black dashed lines indicate the MSE in case only zero values are predicted.

Vi Discussion

Recent technological trends in medical imaging have spurred the demand for imaging pipelines that rely on less data without compromising image quality, temporal resolution, or more generally, diagnostics. We here consider the notion of task-driven sampling, in which sampling schemes are optimized not to recover the sensor signals themselves, but to fulfill a specific imaging task.

In this paper we proposed LASSY, a framework that permits joint learning of a context- and task-specific sub-sampling pattern and an adequate reconstruction method. We demonstrated that these learned sub-sampling patterns yield improved reconstruction results compared to non-learned patterns, and are indeed specific to the imaging task. As opposed to other recently introduced learned compressed sensing techniques, LASSY learns to sub-sample rather than to take full linear measurements that face practical implementation challenges. Sub-sampling permits straightforward implementation of the learned sampling pattern into sensing applications, with examples being array element selection, slow-time ultrasound pulsing schemes, (non-uniform) analog-to-digital converters (ADC) and partial Fourier measurements.

In ultrasound imaging, we specifically applied LASSY for slow-time pulse scheme design and the array channel selection problem. Besides data reduction, the former reduces the amount of transmit events, which has the additional advantage of drastically reducing power consumption. Reduced power consumption also benefits battery life for wireless applications, and reduces heat generation of ADCs, which is particularly relevant for in-body applications.

The applications, or tasks, that we considered within the ultrasound imaging domain were anatomical (gray-level) imaging and tissue-motion (Doppler) imaging. LASSY yielded distinct sampling patterns for each task, with e.g. tissue-motion estimation spurring a pattern that uses compact groups of slow-time samples with a short inter-pulse time. We expect that other ultrasound imaging applications, such as super-resolution ultrasound localization microscopy (ULM), can benefit similarly from learned and dedicated sampling schemes. In ULM, millions of highly sparse point-scatterers (intravascular microbubbles) are to be detected and localized across thousands of frames at ultrafast imaging rates

[18]. Consequently, data rates are extremely high. Recently, deep neural networks have been proposed for fast ULM recovery [48], and one can envisage the use of LASSY to learn adequate sampling patterns that reduce data rates in this context.

Generally, the learned sub-sampling patterns outperformed uniform sub-sampling schemes. In one particular example, this was not the case, namely when sub-sampling across slow-time by only a factor 2 (see Fig. 5). Interestingly, considering that the (fully sampled) Doppler shifts yielded a maximum relative frequency that was just below 0.5, uniformly undersampling by a factor 2 did not introduce aliasing and still permitted adequate reconstruction. This was however not the case for Doppler prediction using uniform sub-sampling patterns with higher factors; Doppler reconstruction was greatly impaired due to aliasing.

We expect that improvements of LASSY (for all sub-sampling factors) can be realized by better fine-tuning of the training hyperparameters. These include the learning rate and learning rate schedulers, the penalty multipliers, and the initialization of the logits in . In addition, the ratio between the learning rates and was found to have great influence on performance. Extensive fine-tuning of these parameters was out of the scope of this research however.

While the focus of this work was on the development of a framework that permits backpropagation-based learning of (hard) sampling, additional improvements can be expected when further optimizing the recovery neural networks, making them more dedicated to the task. For instance, for image recovery after channel sub-sampling, recent work on adaptive beamforming by deep learning can be considered [33].

Beyond the ultrasound applications considered here, future work may include learning sub-sampling and reconstruction for compressed sensing MRI [35], where measurements are inherently performed by sampling the spatial Fourier domain. MRI thus shares strong similarities with signal reconstruction from partial Fourier measurements (shown in Sec. V-A), making it an excellent candidate for LASSY. Also investigating LASSY’s use for sparse view CT imaging is of interest, potentially permitting reduction of the amount of transmit events, and therewith exposure to harmful radiation.

Vii Conclusions

In this paper we have presented LASSY, a probabilistic framework that permits joint optimization of a task-based sub-sampling scheme and a signal recovery method by deep learning. We have demonstrated its effectiveness for sensing partial Fourier coefficients of sparse signals and a number of ultrasound imaging applications, showing that the proposed method indeed learns sampling schemes that are dedicated to a given task. As such, LASSY opens up a wide range of new opportunities; beyond ultrasound imaging, we foresee its application in other medical imaging domains (e.g. MRI and CT) and, more generally, in compressed sensing problems.

References

  • [1] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al. (2016)

    Tensorflow: a system for large-scale machine learning

    .
    In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), pp. 265–283. Cited by: §III-D.
  • [2] A. Adler, D. Boublil, M. Elad, and M. Zibulevsky (2016) A deep learning approach to block-based compressed sensing of images. arXiv preprint arXiv:1606.01519. Cited by: §II-C.
  • [3] A. Adler, M. Elad, and M. Zibulevsky (2016) Compressed learning: a deep neural network approach. arXiv preprint arXiv:1610.09615. Cited by: §II-C.
  • [4] A. M. Atto, D. Pastor, and G. Mercier (2008) Smooth sigmoid wavelet shrinkage for non-parametric estimation. In 2008 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 3265–3268. Cited by: §IV-A.
  • [5] A. Austeng and S. Holm (2002) Sparse 2-d arrays for 3-d phased array imaging-design methods. IEEE Transactions on Ultrasonics, Ferroelectrics, and frequency Control 49 (8), pp. 1073–1086. Cited by: §II-B.
  • [6] J. M. Baran and J. G. Webster (2009) Design of low-cost portable ultrasound systems. In 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 792–795. Cited by: §I.
  • [7] E. J. Candes, J. K. Romberg, and T. Tao (2006) Stable signal recovery from incomplete and inaccurate measurements. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences 59 (8), pp. 1207–1223. Cited by: §I.
  • [8] E. J. Candes and T. Tao (2005)

    Decoding by linear programming

    .
    IEEE Transactions on Information Theory 51 (12), pp. 4203–4215. Cited by: §I.
  • [9] E. J. Candes and T. Tao (2006) Near-optimal signal recovery from random projections: universal encoding strategies?. IEEE Transactions On Information Theory 52 (12), pp. 5406–5425. Cited by: §I.
  • [10] E. J. Candès (2006) Compressive sampling. In Proceedings of the International Congress of Mathematicians: Madrid, August 22-30, 2006: invited lectures, pp. 1433–1452. Cited by: §I, §I, §IV-A.
  • [11] E. J. Candes (2008) The restricted isometry property and its implications for compressed sensing. Comptes rendus mathematique 346 (9-10), pp. 589–592. Cited by: §I.
  • [12] G. Chen, J. Tang, and S. Leng (2008) Prior image constrained compressed sensing (piccs): a method to accurately reconstruct dynamic ct images from highly undersampled projection data sets. Medical physics 35 (2), pp. 660–663. Cited by: §I.
  • [13] T. Chernyakova and Y. C. Eldar (2014) Fourier-domain beamforming: the path to compressed ultrasound imaging. IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control 61 (8), pp. 1252–1267. Cited by: §II-A.
  • [14] K. Choi, J. Wang, L. Zhu, T. Suh, S. Boyd, and L. Xing (2010) Compressed sensing based cone-beam computed tomography reconstruction with a first-order method a. Medical physics 37 (9), pp. 5113–5125. Cited by: §I, §II-A.
  • [15] F. Chollet (2015)

    Keras: deep learning library for theano and tensorflow

    .
    URL: https://keras. io/k 7 (8), pp. T1. Cited by: §III-D.
  • [16] R. Cohen and Y. C. Eldar (2018) Sparse convolutional beamforming for ultrasound imaging. IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control 65 (12), pp. 2390–2406. Cited by: §II-B.
  • [17] R. Cohen and Y. C. Eldar (2018) Sparse doppler sensing based on nested arrays. IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency control 65 (12), pp. 2349–2364. Cited by: §II-B.
  • [18] O. Couture, V. Hingot, B. Heiles, P. Muleki-Seya, and M. Tanter (2018) Ultrasound localization microscopy and super-resolution: a state of the art. IEEE Transactions on Ultrasonics, Ferroelectrics, and frequency Control 65 (8), pp. 1304–1320. Cited by: §VI.
  • [19] I. Daubechies, M. Defrise, and C. De Mol (2004) An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences 57 (11), pp. 1413–1457. Cited by: §I, §IV-A.
  • [20] Y. C. Eldar and G. Kutyniok (2012) Compressed sensing: theory and applications. Cambridge University Press. Cited by: §I, §I, §IV-A.
  • [21] I. Goodfellow, Y. Bengio, and A. Courville (2016) Deep learning. MIT Press. Note: http://www.deeplearningbook.org Cited by: §IV-B3.
  • [22] K. Gregor and Y. LeCun (2010) Learning fast approximations of sparse coding. In Proceedings of the 27th International Conference on International Conference on Machine Learning, pp. 399–406. Cited by: §IV-A.
  • [23] E. J. Gumbel (1954) Statistical theory of extreme values and some practical applications. NBS Applied Mathematics Series 33. Cited by: §III-B.
  • [24] H. Hasegawa and H. Kanai (2011) High-frame-rate echocardiography using diverging transmit beams and parallel receive beamforming. Journal of medical ultrasonics 38 (3), pp. 129–140. Cited by: §IV-C1.
  • [25] E. Jang, S. Gu, and B. Poole (2017) Categorical reparametrization with gumbel-softmax. stat 1050, pp. 17. Cited by: §I, §III-B, §III-B.
  • [26] C. Kasai, K. Namekawa, A. Koyano, and R. Omoto (1985) Real-time two-dimensional blood flow imaging using an autocorrelation technique. IEEE Transactions on sonics and ultrasonics 32 (3), pp. 458–464. Cited by: §IV-B2.
  • [27] D. P. Kingma and J. Ba (2014) Adam: a method for stochastic optimization. ICLR. Cited by: §III-D.
  • [28] K. Kulkarni, S. Lohit, P. Turaga, R. Kerviche, and A. Ashok (2016) Reconnet: non-iterative reconstruction of images from compressively sensed measurements. In

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    ,
    pp. 449–458. Cited by: §I, §II-C.
  • [29] J. D. Larson III (1993-July 20) 2-d phased array ultrasound imaging system with distributed phasing. Google Patents. Note: US Patent 5,229,933 Cited by: §I.
  • [30] C. Liu and P. Vaidyanathan (2017) Maximally economic sparse arrays and cantor arrays. In 2017 IEEE 7th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP), pp. 1–5. Cited by: §II-B.
  • [31] O. Lorintiu, H. Liebgott, M. Alessandrini, O. Bernard, and D. Friboulet (2015) Compressed sensing reconstruction of 3d ultrasound data using dictionary learning and line-wise subsampling. IEEE Transactions on Medical Imaging 34 (12), pp. 2467–2477. Cited by: §I, §II-A.
  • [32] X. Lu, W. Dong, P. Wang, G. Shi, and X. Xie (2018) Convcsnet: a convolutional compressive sensing framework based on deep learning. arXiv preprint arXiv:1801.10342. Cited by: §II-C.
  • [33] B. Luijten, R. Cohen, F. J. de Bruijn, H. A. Schmeitz, M. Mischi, Y. C. Eldar, and R. J. van Sloun (2019) Deep learning for fast adaptive beamforming. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1333–1337. Cited by: §VI.
  • [34] M. Lustig, D. L. Donoho, J. M. Santos, and J. M. Pauly (2008) Compressed sensing mri. IEEE signal processing magazine 25 (2), pp. 72. Cited by: §I, §II-A, §IV-A.
  • [35] M. Lustig, D. Donoho, and J. M. Pauly (2007) Sparse mri: the application of compressed sensing for rapid mr imaging. Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine 58 (6), pp. 1182–1195. Cited by: §I, §II-A, §IV-A, §VI.
  • [36] C. J. Maddison, A. Mnih, and Y. W. Teh (2016)

    The concrete distribution: a continuous relaxation of discrete random variables

    .
    arXiv preprint arXiv:1611.00712. Cited by: §III-B, §III-B.
  • [37] A. Mousavi and R. G. Baraniuk (2017) Learning to invert: signal recovery via deep convolutional networks. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2272–2276. Cited by: §II-C.
  • [38] A. Mousavi, A. B. Patel, and R. G. Baraniuk (2015) A deep learning approach to structured signal recovery. In 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp. 1336–1343. Cited by: §II-C.
  • [39] R. Otazo, D. Kim, L. Axel, and D. K. Sodickson (2010) Combination of compressed sensing and parallel imaging for highly accelerated first-pass cardiac perfusion mri. Magnetic resonance in medicine 64 (3), pp. 767–776. Cited by: §IV-A.
  • [40] D. Perdios, A. Besson, M. Arditi, and J. Thiran (2017) A deep learning approach to ultrasound image recovery. In 2017 IEEE International Ultrasonics Symposium (IUS), pp. 1–4. Cited by: §I, §II-C.
  • [41] J. Provost, C. Papadacci, J. E. Arango, M. Imbault, M. Fink, J. Gennisson, M. Tanter, and M. Pernot (2014) 3D ultrafast ultrasound imaging in vivo. Physics in Medicine & Biology 59 (19), pp. L1–L13. Cited by: §I.
  • [42] E. Y. Sidky and X. Pan (2008) Image reconstruction in circular cone-beam computed tomography by constrained, total-variation minimization. Physics in Medicine & Biology 53 (17), pp. 4777. Cited by: §I.
  • [43] T. L. Szabo (2004) Diagnostic ultrasound imaging: inside out. Academic Press. Cited by: §IV-C3, §V-C.
  • [44] M. Tanter and M. Fink (2014) Ultrafast imaging in biomedical ultrasound. IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control 61 (1), pp. 102–119. Cited by: §I.
  • [45] Z. Tian, X. Jia, K. Yuan, T. Pan, and S. B. Jiang (2011) Low-dose ct reconstruction via edge-preserving total variation regularization. Physics in Medicine & Biology 56 (18), pp. 5949. Cited by: §I, §II-A.
  • [46] J. Tsao, P. Boesiger, and K. P. Pruessmann (2003) K-t blast and k-t sense: dynamic mri with high frame rate exploiting spatiotemporal correlations. Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine 50 (5), pp. 1031–1042. Cited by: §II-A.
  • [47] R. J. Van Sloun, H. Belt, K. Janse, and M. Mischi (2018) Learning doppler with deep neural networks and its application to intra-cardiac echography. In 2018 IEEE International Ultrasonics Symposium (IUS), pp. 1–4. Cited by: §IV-B2.
  • [48] R. J. van Sloun, O. Solomon, M. Bruce, Z. Z. Khaing, H. Wijkstra, Y. C. Eldar, and M. Mischi (2018) Super-resolution ultrasound localization microscopy through deep learning. arXiv preprint arXiv:1804.07661. Cited by: §VI.
  • [49] R. van Sloun, A. Pandharipande, M. Mischi, and L. Demi (2015) Compressed sensing for ultrasound computed tomography. IEEE Transactions on Biomedical Engineering 62 (6), pp. 1660–1664. Cited by: §I.
  • [50] N. Wagner, Y. C. Eldar, A. Feuer, G. Danin, and Z. Friedman (2011) Xampling in ultrasound imaging. In Medical Imaging 2011: Ultrasonic Imaging, Tomography, and Therapy, Vol. 7968, pp. 796818. Cited by: §I, §II-A.
  • [51] D. Wildes, W. Lee, B. Haider, S. Cogan, K. Sundaresan, D. M. Mills, C. Yetter, P. H. Hart, C. R. Haun, M. Concepcion, et al. (2016) 4-d ice: a 2-d array transducer with integrated asic in a 10-fr catheter for real-time 3-d intracardiac echocardiography. IEEE Transactions on Ultrasonics, Ferroelectrics, and frequency Control 63 (12), pp. 2159–2173. Cited by: §I.
  • [52] B. Xu, N. Wang, T. Chen, and M. Li (2015) Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.00853. Cited by: §IV-B3, §IV-C3.