I Introduction
Advanced medical imaging techniques require transfer and storage of large amounts of data. Due to limited bandwidth and storage capacity, the raw sensor data must be compressed prior to its transfer to the backend system. Data compression, undersampling, and subsequent reconstruction techniques have been an active area of research for medical imaging modalities such as computed tomography (CT) imaging [42, 12, 14, 45], ultrasound CT imaging [49], ultrasound imaging [31, 50], and magnetic resonance imaging (MRI)[35, 34]. In this paper, we propose a framework for learning a taskdriven subsampling and reconstruction method that permits reduction of sensor data rates, while retaining the information required to perform a given (imaging) task.
Among diagnostic imaging options, ultrasound imaging is an increasingly used modality, owing to its portability, costeffectiveness, excellent temporal resolution, minimal invasiveness, and radiationfree nature. Compact, portable, and wireless ultrasound probes are emerging[6], enabling ‘pocketsized’ devices. Also transducers are becoming miniaturized, which facilitates e.g. inbody imaging for interventional applications. As a consequence, available bandwidth is limited due to either wireless data transfer or data being transferred over a thin catheter in case of inbody applications. At the same time, emerging ultrafast 3D ultrasound imaging techniques[41, 44] cause data rates to drastically grow, which in turn poses even higher demands on the probetosystem communication. Given these challenges, ultrasound imaging serves as an excellent candidate for evaluating the effectiveness of the framework that we will introduce.
Commonly used techniques to reduce data rates in 2D and 3D echography applications are microbeamforming[29, 51] and slowtime^{1}^{1}1In ultrasound imaging a distinction is made between slowtime and fasttime: slowtime refers to a sequence of snapshots (i.e., across multiple transmit/receive events), at the pulse repetition rate, whereas fasttime refers to samples along depth. multiplexing. The former compresses data from multiple (adjacent) transducer elements (i.e. channels) into a single focused line, thereby virtually reducing the number of receive channels. While effective, this impairs the attainable resolution and image quality. The latter only communicates a subset of the channel signals to the backend of the system for every slowtime transmission. This comes at the cost of reduced frame rates.
Compressed sensing (CS) permits low data rate sensing (below the Nyquist rate) with strong signal recovery guarantees under specific conditions [7, 8, 10, 9, 20]. In CS, a sparse signal is to be recovered from measurements that are taken at a subNyquist rate through a sensing matrix : , with : , .
should preserve distance between distant signal vectors, i.e. it should satisfy the restricted isometry property (RIP)
[11, 8].Proven (RIPcompliant) designs for take randomlyweighted linear combinations of input vector elements [10, 20]. Unfortunately, such designs often impose challenges regarding practical implementability. For example, in ultrasound imaging, sensing weighted combinations of slowtime frames would require an, often unfeasible and undesirably, large temporal signal support (including past and future values), and measuring linear combinations of channel signals imposes strong connectivity challenges. Alternatively, sampling a random subset of Fourier coefficients was also shown to be RIPcompliant [10, 20]. Whenever measuring in the Fourier domain is possible (e.g. in MRI), such partial Fourier measurements alleviate the above challenges.
After sensing, signal recovery in CS is typically achieved through proximal gradient schemes, such as the Iterative Shrinkage and Thresholding algorithm (ISTA)[19]. Although proximal gradient schemes are effective tools for solving nondifferentiable convex optimization problems, in practice, their performance is greatly dependent on tuning of the thresholding parameter and their timeconsuming iterative nature makes them less suitable for realtime applications. Recently, a number of deep learning approaches have been proposed for fast signal or image reconstruction in CS [40, 28]
, showing that deep neural networks can serve as a powerful alternative to conventional recovery techniques.
Inspired by both the challenge of finding adequate contextspecific sensing matrices, and the given deep learning approaches for signal recovery, we present a deep learning solution that jointly learns a context and taskbased subsampling pattern and a corresponding signal reconstruction method. This approach is referred to as LeArning SubSampling and RecoverY (LASSY). Efficient learning by error backpropagation is enabled through the adoption of the GumbelSoftmax distribution
[25], that circumvents the inherently nondifferentiable nature of sampling. We demonstrate LASSY’s effectiveness for signal recovery from both partial Fourier measurements and subsampled invivo ultrasound radiofrequency (RF) data.The remainder of this paper is organized as follows, we start by providing some related work in Sec. II, followed by the general framework of LASSY in Sec. IIIA. Sections IIIB and IIIC respectively elaborate on the subsampling strategy and signal recovery method of LASSY. The training strategy is described in Sec. IIID. Section IVA demonstrates LASSY on a common Fourier domain subsampling problem. Its applications in ultrasound imaging are subsequently described in Secs. IVB and IVC. Results are given in Sec. V, which are discussed in Sec. VI. Final conclusions are drawn in Sec. VII.
Ii Related work
In this section we briefly list recent applications of conventional CS techniques for medical imaging that subsample the data. We then give promising applications of sparse arrays. These examples highlight the potential relevance for learning a taskdriven subsampling pattern across a number of applications. The recent developments in deep learning for CS, that we discuss lastly, show stateoftheart methods for learningbased data compression.
Iia Compressed sensing in medical imaging
Several CS approaches have been introduced for various medical imaging applications. In MRI, CS is applied by randomly subsampling the Kspace [34, 35]
, i.e. the 2D spatial Fourier transform of the image. The authors of
[46] extend this to subsampling in the Ktime space, while preserving qualitative image reconstructions using their kt BLAST and kt SENSE algorithms for one coil and multiple coils, respectively. Likewise, CS has spurred lowdose Xray CT through image reconstruction from subsampled projection measurements [14, 45], and the authors of [31] show good reconstruction results after subsampling 3D US data over RF lines. In [50, 13], the authors apply CS to ultrasound imaging by passing the RF channel signals through analog sumofsinc filters, permitting sampling of a partial set of Fourier coefficients. Related to this, we demonstrate how LASSY permits learning of partial Fourier coefficients in Sec. VA.IiB Sparse arrays
Significant research efforts have been invested in exploration of adequate sparse array designs [30]. Examples in medical ultrasound imaging are a nonuniform slowtime transmission scheme for spectral Doppler [17] and sparse arrays for reduction of the required number of channels for Bmode imaging^{2}^{2}2In ultrasound imaging, Bmode refers to “brightness mode”, a 2D intensity image at a single point in time., based on sparse periodic arrays [5] or sum coarrays [16]. In Secs. VB and VC, we show how LASSY enables learning of these slowtime and array sampling patterns for ultrasound imaging in a taskbased fashion.
IiC Deep learning for compressed sensing
Recently, a number of deep learning approaches have been proposed for fast signal or image reconstruction in CS [40, 28], showing that deep neural networks can serve as powerful signal or image recovery methods. The authors of[40, 38, 37, 3, 2, 32] extend learning beyond signal recovery, and simultaneously train signal compression methods. However, they all rely on taking (randomly weighted) (non)linear combinations of elements from the input vector, making them challenging to implement in hardware. Instead, LASSY is based on subsampling, which is straightforwardly implementable and applicable across the applications given in Secs. IIA and IIB.
Iii Methods
Iiia General framework
In LASSY, we consider a signal vector that we wish to subsample through a binary subsampling matrix parametrized by , to yield a measurement vector^{3}^{3}3 and can also be higher dimensional. In that case all given formulas are applied on the dimension in which we want to subsample . , with :
(1) 
We subsequently aim to decode into , some function of the original signal vector in which we are interested (i.e. the task):
(2) 
To this end, we adopt a (potentially nonlinear) differentiable function approximator parametrized by a set of parameters :
(3) 
where denotes the recovery of from the subsampled measurements . The function may for instance be a neural network. Matrix is constrained to have a rowwise norm equal to 1, i.e. every row contains exactly one nonzero element. As such, selects a subset of (out of ) elements from input vector .
To permit joint learning of an adequate subsampling pattern for and recovery of through by backpropagation, we will introduce a probabilistic sampling strategy, on which we elaborate in the next section.
IiiB Learning subsampling
Each row of , with
, is defined as a onehot encoding
^{4}^{4}4The onehot encoding, , of a categorical random variable with classes results in a unitvector of length . Exactly one element is nonzero and its index corresponds to the class of the drawn sample.of an independent categorical random variable
(4) 
where is a vector containing
class probabilities. Note that
thus represents the probability of sampling the entry in at the measurement . We reparametrizeusing unnormalized logprobabilities (logits)
, such that(5) 
where is the unnormalized logit of .
To enable sampling from the categorical probability distribution, we leverage the Gumbelmax trick
[23], i.e. sampling is reparametrized into a function of the distribution parameters and a Gumbel noise vector , with , i.i.d.. A realization of is then defined as:(6) 
The subscript WR denotes sampling without replacement, which we implement across to , i.e. the same sample is never selected more than once. This is achieved by dynamically excluding the categories that have already been sampled, and renormalizing the logits of the resulting distribution. Each row can now be defined as:
(7) 
We define as the row of a trainable matrix that contains the unnormalized logits of all distributions. To permit optimization of by backpropagation, we require to exist . Since
is a nondifferentiable operator, we adopt the StraightThrough Gumbel Estimator
[25, 36] as a surrogate for :(8) 
with (row operator) as a continuous differentiable approximation of the onehot encoded operation. We refer to sampling using the function as soft sampling. Its temperature parameter serves as a gradient distributor over multiple entries (i.e. logits) in .
IiiC Signal recovery by deep learning
IiiD Training strategy
We train model parameters and by minimizing the mean squared error (MSE) between the model’s output and the target
, assuming normally distributed prediction errors. To prevent overfitting and exploding gradients, the problem is regularized by adding an
penalty on . Besides, we promote training towards onehot distributions by penalizing convergence towards high entropy distributions using:(11) 
with defined as in (5).
The resulting optimization problem can be written as:
(12) 
with
(13) 
and
(14) 
where the input and target vectors, i.e. and respectively, follow datagenerating distribution . Penalty multipliers and weigh the importance of the different penalties.
The Adam solver with hyperparameters
, , and [27] is used to stochastically optimize (12). In practice, we found that the appropriate learning rates for and were different. As such, two separate learning rates were used, i.e. and , with . The adopted values are reported in Sec. IV, along with the values for the penalty multipliers and , and the number of used iterations for training. We define one iteration as a trainable parameter update using one minibatch of data.The temperature parameter in (10) is initialized at and gradually lowered to during training. The initialization of logits matrix , promotes preservation of the original order of elements in . As such, all elements , with and are initialized according to:
(15) 
with constants and , and i.i.d..
The pseudocode of LASSY is shown in Algorithm 1
. LASSY was implemented in Python using Keras
[15]with a TensorFlow backend
[1]. Training and inference were performed on a Titan XP (NVIDIA, Santa Clara, CA).Iv Validation methodology
Iva Partial Fourier sampling of sparse signals
Many practical CS applications require signal reconstruction from partial Fourier measurements[39, 35, 34], and we therefore first demonstrate LASSY in such a scenario. To that end, we synthetically generate random Ksparse signal vectors , with , which we subsequently Fouriertransform to yield the signal that we aim to partially sample^{5}^{5}5For each experiment the length of the signal (in the dimension to be subsampled) was set to the closest integer multiple of the subsampling factor, e.g. 126 for factor 6.. Here, the measurement , with , is a subsampled set (learned by ) of Fourier coefficients in , and the task is to recover the sparse signal, , from measurement .
We compare the reconstruction performance of using this taskbased learned subsampling pattern with performances of using an untrained fixed uniform and a random subsampling strategy. The latter is typically adopted in CS[10, 20].
We adopt a specific recovery network architecture that is inspired by the proximal gradient ISTA scheme[19]; it unrolls the iterative solution of ISTA as a 2layer feedforward neural network with trainable (thresholding) parameters[22]. To prevent dying gradients during backpropagation, we replace the conventional softthresholding operators in this learned ISTA (LISTA) method by a sigmoidbased softthresholding operator [4].
We train for 96,000 iterations across minibatches of 16 randomly generated Fouriertransformed data vectors. The learning rates and are set at and , and the penalty multipliers and at 0.0 and , respectively.
IvB Slowtime subsampling in ultrasound imaging
IvB1 Data acquisition and preprocessing
Sequential (slowtime) ultrasound data were acquired from an invivo openchest intracardiac echography measurement of the right atrium of a porcine model. To that end, a 48element linear array miniTEE s73t transducer with a pitch of 0.151 mm was used in combination with a Verasonics Vantrage system (Kirkland, WA). The center frequency for transmission and reception was 4.8 MHz and a 13angle diverging wave scheme was used. The sampling rate of the received RF data was 19.2 MHz and coherently compounded beamformed frames (each with 68 scanlines) were collected at a frame rate of 474 Hz. These RF data frames were then demodulated into their inphase and quadrature (IQ) components, and subsequently normalized between 1 and +1. Two such complete acquisitions were performed, of which one was used for training and one served as a holdout test set.
IvB2 Tasks
Using the data acquired according to the procedure described in the previous section, we employ LASSY to learn a subsampling pattern for a sequence of IQ scanlines across slowtime and subsequently learn a specific task. We define two different tasks. First, we aim to recover the envelope of the beamformed RF signal in order to produce a standard graylevel ultrasound image. Here, the target is the magnitude of the (fully sampled) complex IQ data . Second, we explore LASSY for learningbased tissue motion estimation (i.e. Doppler recovery [47]) from the subsampled IQ scanlines across slowtime. In this case, the target is computed using the wellknown Kasai autocorrelator [26]. We expect the two tasks to yield very distinct sampling patterns; where envelope construction is performed independently per frame, Doppler shifts are obtained by measuring phase shifts across the slowtime sequence.
IvB3 Recovery neural network architecture
For recovery of from the subsampled IQ scanlines in
, we employ a deep convolutional neural network
[21]. The first 2 layers are 1D convolutional layers with respectively 256 and 128 features and window length = 5, assuming translational invariance across the fasttime dimension. Across slowtime, neurons are fully connected, since a similar invariance in this dimension may be lost after (possibly irregular) subsampling. After 2 such layers, spatial structure across both dimensions is assumed to be retained, and 4 2D convolutional layers with kernel sizes
and respectively 32, 64, 32, and 1 feature(s), are added. We use leaky rectified linear unit (leaky ReLU) activation functions (
) across all convolution layers, except the last, which has no activation function [52].IvB4 Training
For both tasks, the networks are stochastically optimized using the Adam solver, with settings as described in Sec. IIID, and learning rates and . We train for 320,000 iterations with minibatches consisting of 16 randomly selected patches. Each patch contains 128 sequential slowtime samples of 256 fasttime IQ samples for a single radial scanline. The logits of the categorical distributions in matrix are initialized according to (15). Penalty multipliers and are set at and , respectively.
IvC Channel subsampling in ultrasound imaging
IvC1 Data acquisition and preprocessing
The same imaging setup as described in Sec. IVB1 was used to demonstrate LASSY for subsampling across the 48channel array, prior to beamforming. To facilitate the subsequent receive beamforming stage, we first predelay the channel signals for 68 different scanlines (with steering angles in ) [24]. Taking into account the transmit delay (i.e. the timeofflight (TOF) between the virtual point source behind the array and the focus point in our diverging wave transmission scheme), and the receive delay (i.e. the TOF of the backscattered wave between the focus point and the array element location, indexed by ), the total delay function for the central wave transmit is defined as[24]:
(16) 
in which
(17) 
and
(18) 
Focal depth is denoted by , is the distance between the surface of the transducer array and the virtual point source behind the array, and are respectively the pitch and total number of channels of the array, and denotes the speed of sound in soft tissue. The adopted values for these parameters are: mm, mm, mm, , and m/s.
After computing 68 delayed signals per channel, we obtain a 4D dataset spanning slowtime frames, fasttime samples, channels, and radial scanlines. Note that precomputing these delays is only done to accelerate training, and can in practice be performed after array subsampling. Finally, the predelayed RF channel signals were demodulated into their inphase and quadrature (IQ) components, and thereafter normalized between 1 and +1.
IvC2 Tasks
We again distinguish two tasks, envelope reconstruction and tissuemotion (Doppler) estimation. Both target datasets are generated by first beamforming the fully sampled channel data, and then subsequently processing this as described in Sec. IVB2.
IvC3 Recovery neural network architecture
For recovery of from the subsampled channel data in , we leverage a convolutional neural network. The network’s first 4 layers are 2D convolutional layers with kernels and respectively 64, 128, 64, and 48 features. Convolutions take place across the fast and slowtime dimension, i.e. the channels are fully connected. Each of the convolutional layers is followed by a leaky ReLU activation function ()[52]. The network’s last layer is a fully connected layer across the (subsampled) channel dimension, which acts as a weighted summation and therefore shares similarities with the array apodization used in typical DAS beamforming [43].
IvC4 Training
For both envelope and Doppler reconstruction, the networks are stochastically optimized using Adam optimizer, with its settings as described in Sec. IIID. Learning rates and are set at and respectively, and we train for 160,000 iterations. Randomly selected minibatches are used for training, each consisting of 16 patches with spanning 32 slowtime frames, 64 fasttime samples, 48 channels, and one radial scanline. Trainable matrix is initialized according to (15) and the penalty multipliers and are set at and , respectively.
V Results
Va Partial Fourier sampling of sparse signals
Figure 2 displays sparse signal recovery from partial Fourier measurements for a uniform, random, and learned subsampling pattern (subsampling factor ) using LASSY. A quantitative evaluation of the recoveries for different subsampling factors is given in Fig. 3, showing that in all cases the MSE was lowest when using LASSY’s learned subsampling pattern. Uniform subsampling performed poorly due to aliasing, resulting in a repeated prediction pattern (see Fig. 2abottom). However, a (CSinspired) random sampling pattern approached the performance of LASSY; interestingly, the learned pattern also exhibits (pseudorandom) irregular sampling (see Fig. 2ctop), and showed to be RIPcompliant.
VB Slowtime subsampling in ultrasound imaging
Figure 4 demonstrates envelope (ae) and Doppler (fj) reconstruction from uniform and learned slowtime subsampling patterns. Interestingly, LASSY’s learned patterns for both tasks are very distinct. For envelope reconstruction, the learned pattern exhibited an almost perfectly uniform sampling pattern. As such, the resulting reconstructions (see Fig. 4b and 4d) were found to be similar. This was consistent across all tested subsampling factors, displaying increased blurring of the graylevel images for higher subsampling factors in both methods. Their MSEs are compared in Fig. 5a.
Unlike envelope reconstruction, Doppler recovery was greatly hampered by uniform subsampling for (Fig. 5b). Increasing the subsampling factor did not only lead to blurring, but strongly impaired Doppler estimation due to slowtime aliasing. Interestingly, LASSY yields a very distinct subsampling pattern (Fig. 4hbottom), exhibiting an ‘ensemble’type of sampling for . Similar patterns were clearly visible for the other tested subsampling factors as well.
The learned ‘ensemble’style subsampling pattern efficiently captures high frequency slowtime signals due to tissue displacements (Doppler shifts) within ensembles, and relatively low frequency information (changes in Doppler shifts over time) among these ensembles. Consequently, LASSY’s performance degraded less for increasing subsampling factors, compared to a uniform subsampling strategy.
Using the trained network for inference on the test set (256 slowtime frames, containing 68 scanlines and 2048 fasttime samples) took on average 1.29 s (SD ms). Accordingly, the reconstruction network allows a reconstruction speed of 198 subsampled frames per second.
VC Channel subsampling in ultrasound imaging
Figure 6 displays the envelope (ae) and Doppler (fj) reconstructions after channel selection and subsequent processing for each of these tasks, respectively. The results of using a learned subsampling pattern by LASSY are compared to those obtained by fixed uniform undersampling of the channel array. Using LASSY for learning slowtime subsampling patterns yielded near onehot distributions for each of the measurements, whereas this was not the case for channel subsampling. As such, each realization of (see (9)) was slightly different. Figures 6c and 6h show histograms of the selected channels for realizations obtained in a MonteCarlo fashion. A relative occurrence of 1 indicates that the specific channel was selected for each of the realizations of . The depicted sampling pattern below the histogram is one example of such a realization.
It can be seen that both for envelope and Doppler reconstruction, the center channels were found to carry most information for reconstruction. Interestingly, the relativeoccurrence histogram for envelope reconstruction is wider than the one for Doppler reconstruction, indicating the need for a larger aperture of the transducer array in case of envelope reconstruction. Since a larger aperture imposes higher lateral resolution, the wider histogram for envelope reconstruction perfectly relates to the fact that lateral resolution is typically higher for graylevel images than for Doppler images.
Common practice is to design channel arrays in ultrasound probes that have a pitch which is half the signal’s wavelength in order to prevent grating lobes in the filed of view [43]. Increasing the pitch between channels by uniformly subsampling the channel array thus caused grating lobes to appear in the graylevel images, indicated by the white dashed lines in Fig. 6btop. The relative angle of the grating lobes (with respect to the main beam) can be calculated as [43]:
(19) 
where (mm) is the (original) pitch of the array, and (mm) is the wavelength of the signal.
Figures 7a and 7b respectively show the MSE values for both envelope and Doppler reconstruction using different subsampling factors. In both cases we can see that the MSE gradually increases for higher subsampling factors for both uniform subsampling and learned subsampling using LASSY. However, for all factors LASSY’s reconstruction outperformed reconstruction when using a uniform subsampling pattern.
Running inference on patches from the test set revealed an average reconstruction time of 36.7 ms (SD ms) for IQ data from 12 channels, steered towards 68 scanlines with 2048 fasttime samples at one point in (slow)time, implying a frame reconstruction rate of 27 frames per second.
Vi Discussion
Recent technological trends in medical imaging have spurred the demand for imaging pipelines that rely on less data without compromising image quality, temporal resolution, or more generally, diagnostics. We here consider the notion of taskdriven sampling, in which sampling schemes are optimized not to recover the sensor signals themselves, but to fulfill a specific imaging task.
In this paper we proposed LASSY, a framework that permits joint learning of a context and taskspecific subsampling pattern and an adequate reconstruction method. We demonstrated that these learned subsampling patterns yield improved reconstruction results compared to nonlearned patterns, and are indeed specific to the imaging task. As opposed to other recently introduced learned compressed sensing techniques, LASSY learns to subsample rather than to take full linear measurements that face practical implementation challenges. Subsampling permits straightforward implementation of the learned sampling pattern into sensing applications, with examples being array element selection, slowtime ultrasound pulsing schemes, (nonuniform) analogtodigital converters (ADC) and partial Fourier measurements.
In ultrasound imaging, we specifically applied LASSY for slowtime pulse scheme design and the array channel selection problem. Besides data reduction, the former reduces the amount of transmit events, which has the additional advantage of drastically reducing power consumption. Reduced power consumption also benefits battery life for wireless applications, and reduces heat generation of ADCs, which is particularly relevant for inbody applications.
The applications, or tasks, that we considered within the ultrasound imaging domain were anatomical (graylevel) imaging and tissuemotion (Doppler) imaging. LASSY yielded distinct sampling patterns for each task, with e.g. tissuemotion estimation spurring a pattern that uses compact groups of slowtime samples with a short interpulse time. We expect that other ultrasound imaging applications, such as superresolution ultrasound localization microscopy (ULM), can benefit similarly from learned and dedicated sampling schemes. In ULM, millions of highly sparse pointscatterers (intravascular microbubbles) are to be detected and localized across thousands of frames at ultrafast imaging rates
[18]. Consequently, data rates are extremely high. Recently, deep neural networks have been proposed for fast ULM recovery [48], and one can envisage the use of LASSY to learn adequate sampling patterns that reduce data rates in this context.Generally, the learned subsampling patterns outperformed uniform subsampling schemes. In one particular example, this was not the case, namely when subsampling across slowtime by only a factor 2 (see Fig. 5). Interestingly, considering that the (fully sampled) Doppler shifts yielded a maximum relative frequency that was just below 0.5, uniformly undersampling by a factor 2 did not introduce aliasing and still permitted adequate reconstruction. This was however not the case for Doppler prediction using uniform subsampling patterns with higher factors; Doppler reconstruction was greatly impaired due to aliasing.
We expect that improvements of LASSY (for all subsampling factors) can be realized by better finetuning of the training hyperparameters. These include the learning rate and learning rate schedulers, the penalty multipliers, and the initialization of the logits in . In addition, the ratio between the learning rates and was found to have great influence on performance. Extensive finetuning of these parameters was out of the scope of this research however.
While the focus of this work was on the development of a framework that permits backpropagationbased learning of (hard) sampling, additional improvements can be expected when further optimizing the recovery neural networks, making them more dedicated to the task. For instance, for image recovery after channel subsampling, recent work on adaptive beamforming by deep learning can be considered [33].
Beyond the ultrasound applications considered here, future work may include learning subsampling and reconstruction for compressed sensing MRI [35], where measurements are inherently performed by sampling the spatial Fourier domain. MRI thus shares strong similarities with signal reconstruction from partial Fourier measurements (shown in Sec. VA), making it an excellent candidate for LASSY. Also investigating LASSY’s use for sparse view CT imaging is of interest, potentially permitting reduction of the amount of transmit events, and therewith exposure to harmful radiation.
Vii Conclusions
In this paper we have presented LASSY, a probabilistic framework that permits joint optimization of a taskbased subsampling scheme and a signal recovery method by deep learning. We have demonstrated its effectiveness for sensing partial Fourier coefficients of sparse signals and a number of ultrasound imaging applications, showing that the proposed method indeed learns sampling schemes that are dedicated to a given task. As such, LASSY opens up a wide range of new opportunities; beyond ultrasound imaging, we foresee its application in other medical imaging domains (e.g. MRI and CT) and, more generally, in compressed sensing problems.
References

[1]
(2016)
Tensorflow: a system for largescale machine learning
. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), pp. 265–283. Cited by: §IIID.  [2] (2016) A deep learning approach to blockbased compressed sensing of images. arXiv preprint arXiv:1606.01519. Cited by: §IIC.
 [3] (2016) Compressed learning: a deep neural network approach. arXiv preprint arXiv:1610.09615. Cited by: §IIC.
 [4] (2008) Smooth sigmoid wavelet shrinkage for nonparametric estimation. In 2008 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 3265–3268. Cited by: §IVA.
 [5] (2002) Sparse 2d arrays for 3d phased array imagingdesign methods. IEEE Transactions on Ultrasonics, Ferroelectrics, and frequency Control 49 (8), pp. 1073–1086. Cited by: §IIB.
 [6] (2009) Design of lowcost portable ultrasound systems. In 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 792–795. Cited by: §I.
 [7] (2006) Stable signal recovery from incomplete and inaccurate measurements. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences 59 (8), pp. 1207–1223. Cited by: §I.

[8]
(2005)
Decoding by linear programming
. IEEE Transactions on Information Theory 51 (12), pp. 4203–4215. Cited by: §I.  [9] (2006) Nearoptimal signal recovery from random projections: universal encoding strategies?. IEEE Transactions On Information Theory 52 (12), pp. 5406–5425. Cited by: §I.
 [10] (2006) Compressive sampling. In Proceedings of the International Congress of Mathematicians: Madrid, August 2230, 2006: invited lectures, pp. 1433–1452. Cited by: §I, §I, §IVA.
 [11] (2008) The restricted isometry property and its implications for compressed sensing. Comptes rendus mathematique 346 (910), pp. 589–592. Cited by: §I.
 [12] (2008) Prior image constrained compressed sensing (piccs): a method to accurately reconstruct dynamic ct images from highly undersampled projection data sets. Medical physics 35 (2), pp. 660–663. Cited by: §I.
 [13] (2014) Fourierdomain beamforming: the path to compressed ultrasound imaging. IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control 61 (8), pp. 1252–1267. Cited by: §IIA.
 [14] (2010) Compressed sensing based conebeam computed tomography reconstruction with a firstorder method a. Medical physics 37 (9), pp. 5113–5125. Cited by: §I, §IIA.

[15]
(2015)
Keras: deep learning library for theano and tensorflow
. URL: https://keras. io/k 7 (8), pp. T1. Cited by: §IIID.  [16] (2018) Sparse convolutional beamforming for ultrasound imaging. IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control 65 (12), pp. 2390–2406. Cited by: §IIB.
 [17] (2018) Sparse doppler sensing based on nested arrays. IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency control 65 (12), pp. 2349–2364. Cited by: §IIB.
 [18] (2018) Ultrasound localization microscopy and superresolution: a state of the art. IEEE Transactions on Ultrasonics, Ferroelectrics, and frequency Control 65 (8), pp. 1304–1320. Cited by: §VI.
 [19] (2004) An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences 57 (11), pp. 1413–1457. Cited by: §I, §IVA.
 [20] (2012) Compressed sensing: theory and applications. Cambridge University Press. Cited by: §I, §I, §IVA.
 [21] (2016) Deep learning. MIT Press. Note: http://www.deeplearningbook.org Cited by: §IVB3.
 [22] (2010) Learning fast approximations of sparse coding. In Proceedings of the 27th International Conference on International Conference on Machine Learning, pp. 399–406. Cited by: §IVA.
 [23] (1954) Statistical theory of extreme values and some practical applications. NBS Applied Mathematics Series 33. Cited by: §IIIB.
 [24] (2011) Highframerate echocardiography using diverging transmit beams and parallel receive beamforming. Journal of medical ultrasonics 38 (3), pp. 129–140. Cited by: §IVC1.
 [25] (2017) Categorical reparametrization with gumbelsoftmax. stat 1050, pp. 17. Cited by: §I, §IIIB, §IIIB.
 [26] (1985) Realtime twodimensional blood flow imaging using an autocorrelation technique. IEEE Transactions on sonics and ultrasonics 32 (3), pp. 458–464. Cited by: §IVB2.
 [27] (2014) Adam: a method for stochastic optimization. ICLR. Cited by: §IIID.

[28]
(2016)
Reconnet: noniterative reconstruction of images from compressively sensed measurements.
In
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
, pp. 449–458. Cited by: §I, §IIC.  [29] (1993July 20) 2d phased array ultrasound imaging system with distributed phasing. Google Patents. Note: US Patent 5,229,933 Cited by: §I.
 [30] (2017) Maximally economic sparse arrays and cantor arrays. In 2017 IEEE 7th International Workshop on Computational Advances in MultiSensor Adaptive Processing (CAMSAP), pp. 1–5. Cited by: §IIB.
 [31] (2015) Compressed sensing reconstruction of 3d ultrasound data using dictionary learning and linewise subsampling. IEEE Transactions on Medical Imaging 34 (12), pp. 2467–2477. Cited by: §I, §IIA.
 [32] (2018) Convcsnet: a convolutional compressive sensing framework based on deep learning. arXiv preprint arXiv:1801.10342. Cited by: §IIC.
 [33] (2019) Deep learning for fast adaptive beamforming. In ICASSP 20192019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1333–1337. Cited by: §VI.
 [34] (2008) Compressed sensing mri. IEEE signal processing magazine 25 (2), pp. 72. Cited by: §I, §IIA, §IVA.
 [35] (2007) Sparse mri: the application of compressed sensing for rapid mr imaging. Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine 58 (6), pp. 1182–1195. Cited by: §I, §IIA, §IVA, §VI.

[36]
(2016)
The concrete distribution: a continuous relaxation of discrete random variables
. arXiv preprint arXiv:1611.00712. Cited by: §IIIB, §IIIB.  [37] (2017) Learning to invert: signal recovery via deep convolutional networks. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2272–2276. Cited by: §IIC.
 [38] (2015) A deep learning approach to structured signal recovery. In 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp. 1336–1343. Cited by: §IIC.
 [39] (2010) Combination of compressed sensing and parallel imaging for highly accelerated firstpass cardiac perfusion mri. Magnetic resonance in medicine 64 (3), pp. 767–776. Cited by: §IVA.
 [40] (2017) A deep learning approach to ultrasound image recovery. In 2017 IEEE International Ultrasonics Symposium (IUS), pp. 1–4. Cited by: §I, §IIC.
 [41] (2014) 3D ultrafast ultrasound imaging in vivo. Physics in Medicine & Biology 59 (19), pp. L1–L13. Cited by: §I.
 [42] (2008) Image reconstruction in circular conebeam computed tomography by constrained, totalvariation minimization. Physics in Medicine & Biology 53 (17), pp. 4777. Cited by: §I.
 [43] (2004) Diagnostic ultrasound imaging: inside out. Academic Press. Cited by: §IVC3, §VC.
 [44] (2014) Ultrafast imaging in biomedical ultrasound. IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control 61 (1), pp. 102–119. Cited by: §I.
 [45] (2011) Lowdose ct reconstruction via edgepreserving total variation regularization. Physics in Medicine & Biology 56 (18), pp. 5949. Cited by: §I, §IIA.
 [46] (2003) Kt blast and kt sense: dynamic mri with high frame rate exploiting spatiotemporal correlations. Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine 50 (5), pp. 1031–1042. Cited by: §IIA.
 [47] (2018) Learning doppler with deep neural networks and its application to intracardiac echography. In 2018 IEEE International Ultrasonics Symposium (IUS), pp. 1–4. Cited by: §IVB2.
 [48] (2018) Superresolution ultrasound localization microscopy through deep learning. arXiv preprint arXiv:1804.07661. Cited by: §VI.
 [49] (2015) Compressed sensing for ultrasound computed tomography. IEEE Transactions on Biomedical Engineering 62 (6), pp. 1660–1664. Cited by: §I.
 [50] (2011) Xampling in ultrasound imaging. In Medical Imaging 2011: Ultrasonic Imaging, Tomography, and Therapy, Vol. 7968, pp. 796818. Cited by: §I, §IIA.
 [51] (2016) 4d ice: a 2d array transducer with integrated asic in a 10fr catheter for realtime 3d intracardiac echocardiography. IEEE Transactions on Ultrasonics, Ferroelectrics, and frequency Control 63 (12), pp. 2159–2173. Cited by: §I.
 [52] (2015) Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.00853. Cited by: §IVB3, §IVC3.