# Model-Driven Deep Learning for Joint MIMO Channel Estimation and Signal Detection

In this paper, we investigate the model-driven deep learning (DL) for joint MIMO channel estimation and signal detection (JCESD), where signal detection considers channel estimation error and channel statistics while channel estimation is refined by detected data and takes the signal detection error into consideration. In particular, the MIMO signal detector is specially designed by unfolding an iterative algorithm and adding some trainable parameters. Since the number of trainable parameters is much fewer than the data-driven DL based signal detector, the model-driven DL based MIMO signal detector can be rapidly trained with a much smaller data set. Furthermore, the proposed signal detector can be extended to soft-input soft-output detection easily. Based on numerical results, the model-driven DL based JCESD scheme significantly improves the performance of the corresponding traditional iterative detector and the signal detector exhibits superior robustness to signal-to-noise ratio (SNR) and channel correlation mismatches.

## Authors

• 8 publications
• 34 publications
• 81 publications
• 54 publications
• ### Deep Learning-Aided Iterative Detector for Massive Overloaded MIMO Channels

The paper presents a deep learning-aided iterative detection algorithm f...
06/28/2018 ∙ by Masayuki Imanishi, et al. ∙ 0

• ### A Model-Driven Deep Learning Network for MIMO Detection

In this paper, we propose a model-driven deep learning network for multi...
09/25/2018 ∙ by Hengtao He, et al. ∙ 0

• ### ComNet: Combination of Deep Learning and Expert Knowledge in OFDM Receivers

In this article, we propose a model-driven deep learning (DL) approach t...
10/22/2018 ∙ by Xuanxuan Gao, et al. ∙ 0

• ### Learning to Modulate for Non-coherent MIMO

The deep learning trend has recently impacted a variety of fields, inclu...
03/09/2019 ∙ by Ye Wang, et al. ∙ 1

• ### Learning to Search for MIMO Detection

This paper proposes a novel learning to learn method, called learning to...
09/17/2019 ∙ by Jianyong Sun, et al. ∙ 0

• ### Trainable Projected Gradient Detector for Massive Overloaded MIMO Channels: Data-driven Tuning Approach

The paper presents a deep learning-aided iterative detection algorithm f...
12/25/2018 ∙ by Satoshi Takabe, et al. ∙ 0

• ### Trainable Projected Gradient Detector for Sparsely Spread Code Division Multiple Access

Sparsely spread code division multiple access (SCDMA) is a promising non...
10/23/2019 ∙ by Satoshi Takabe, et al. ∙ 0

##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## I Introduction

Multiple-input multiple-output (MIMO) technology can dramatically improve the spectral efficiency and link reliability and has been applied to many wireless communication systems. To obtain the benefits of MIMO [2], efficient channel estimation and signal detection algorithms, which balance performance and complexity, are essential in receiver design and have arouse a series of research [3, 4, 5, 6]. Among existing detectors, maximum likelihood (ML) detection can achieve the optimal performance. However, its complexity increases exponentially with the number of decision variables. Some suboptimal linear detectors, such as zero-forcing (ZF) and minimum mean-squared error (MMSE) detectors, are with reduced computational complexity, but have a huge performance degradation compared with the ML detection.

With excellent performance and moderate complexity, iterative detectors, based on approximate message passing (AMP) [7] and expectation propagation (EP) [8] algorithms, have been proposed for MIMO detection [5, 6]

. The AMP-based detector approximates the posterior probability on a dense factor graph by using the central limit theorem and the Taylor expansion, which can achieve Bayes-optimal performance in the large-scale systems when the elements of the channel matrix are with independent and identically sub-Gaussian distribution. The EP-based detector

[6, 8] is derived by approximating the posterior distribution with factorized Gaussian distributions and can achieve Bayes-optimal performance when the channel matrix is unitarily-invariant 111A matrix is unitarily-invariant if , and are mutually independent, and , are Haar-distributed. The independent and identically distributed (i.i.d.) Gaussian matrix is a typical unitarily-invariant matrix. and with a large scale. However, for practical small-size (e.g., or ) MIMO systems, the performance of these iterative detectors is still far from Bayes-optimal solution and has serious deterioration with correlated MIMO channels and imperfect channel state information (CSI) [9].

Owing to strong learning ability from the data, deep learning (DL) has been successfully introduced to computer vision, automatic speech recognition, and natural language processing. Recently, it has been applied in physical layer communications

[10, 11, 12], such as channel estimation [13, 14, 15], CSI feedback [16], signal detection [17, 18, 19, 20, 25, 21, 22, 23, 24], and channel coding [26, 27]. In particular, a five-layer fully connected deep neural network (DNN) is embedded into an orthogonal frequency-division multiplexing (OFDM) system for joint channel estimation and signal detection (JCESD) by treating the receiver as a black box and without exploiting domain knowledge [17]. However, training such a black-box-based network requires a lot of training time in addition to a huge data set. On the other hand, model-driven DL constructs the network topology based on known domain knowledge and has been successfully applied to image reconstruction [29], sparse signal recovery [30, 31, 32, 33], and wireless communications recently[1, 11].

For MIMO detection, a specifically designed network, named DetNet, has been proposed in [18] by unfolding the iteration of a projected gradient descent algorithm and adding considerable trainable variables. DetNet has comparable performance with the AMP-based detector and is more robust to ill-conditioned channels [18]. To further reduce the number of learnable parameters and improve convergence, the approaches in [19] and [20] use DL techniques for the belief propagation and message passing detector, respectively. In [25], a DL-based sphere decoding algorithm is proposed, where the radius of the decoding hypersphere is learned by DNN. The performance achieved by this algorithm is very close to the optimal ML detection. However, most of the existing DL-based detector assume accurate CSI at the receiver and ignore the channel estimation error.

Motivated by existing works, we develop a model-driven DL framework for joint MIMO channel estimation and signal detection in this article, where the iterative detector is improved with a few number of trainable variables to adapt to various channel environments and incorporate channel estimation error at the same time. In particular, a model-driven DL network, named OAMP-Net2, is proposed for signal detection. The structure of the detector is obtained by unfolding the OAMP detector, which is similar to our early work in [1], and inspired by TISTA network[31] but adding more trainable parameters. Furthermore, data-aided scheme is utilized to further improve channel estimation. The trainable parameters are optimized by DL technique to adapt to various channel environments and take channel estimation error into consideration. The main contributions of this paper are summarized as follows:

• Different form the existing DL-based MIMO detector [18, 19, 21, 20, 22, 23, 25] with perfect CSI, we address joint MIMO channel estimation and signal detection, which improves the performance of MIMO receiver by considering the characteristics of channel estimation error and channel statistics and using the estimated payload data to refine the channel estimation.

• Compared with the existing DL-based MIMO detector [18, 19, 20, 25], our proposed detector can provide soft-output information for decoder and absorb the soft information. In addition, only a few trainable parameters are required to be learned, which can reduce the demand for computing resources and training time significantly.

• Based on our numerical results, the OAMP-Net2 has considerable performance gain compared with the OAMP detector. Furthermore, OAMP-Net2 has strong robustness to signal-to-noise (SNR) and channel correlation mismatch.

Notations—For any matrix , , , and denote the transpose, conjugated transpose, and trace of , respectively. In addition,

is the identity matrix,

is the zero matrix. A proper complex Gaussian with mean

and covariance

can be described by the probability density function,

 NC(z;μ,Θ)=1det(πΘ)e−(z−μ)HΘ−1(z−μ).

The rest of this paper is organized as follows. After introducing the JCESD architecture in Section II, we develop channel estimator in Section III and propose the model-driven DL detector in Section IV. Then, numerical results are presented in Section V. Finally, Section VI concludes the paper.

## Ii Joint Channel Estimation and Signal Detection

In this section, we consider a MIMO system with transmit and receive antennas. After presenting the JCESD architecture, we introduce the signal detection with channel estimation error and data-aided channel estimation.

We assume that channel matrix does not change in a time slot. In each time slot,

pilot vectors

for , are first transmitted, which are followed data vectors, . The received signal vectors are for and for corresponding to the pilot and data vectors, respectively. We can also express them into matrix forms as , , , and .

### Ii-a JCESD Architecture

As in Fig. , we consider a turbo-like JCESD architecture for MIMO systems in this paper, which shares the same spirit as iterative decoding. In JCESD, channel estimator and signal detector exchange information iteratively until convergence [34, 35, 36, 37]. In the first iteration, pilot-only based channel estimation is performed. In the subsequent iterations, data-aided channel estimation is employed with the help of the detected data.

The input of the JCESD architecture is the pilot signal matrix, , received signal matrix corresponding to the pilot matrix, , corresponding to the data matrix, in each time slot. In the -th iteration, is the estimated data matrix, and and are used to computer the covariance matrix for equivalent noise in signal detector and channel estimator, respectively. The final output of the signal detector is finally detected data matrix , where is the total number of iterations.

Compared with the conventional receiver design where the channel estimator and signal detector are designed separately, this architecture can improve the performance of the receiver by considering the characteristics of channel estimation error in addition to channel statistics when performing signal detection and using the estimated payload data for channel estimation, as we will illustrate subsequently.

### Ii-B Signal Detection with Channel Estimation Error

In the MIMO system, the received data signal vector corresponding to the -th data vector can be expressed by

 yd[n]=Hxd[n]+nd[n], (1)

where is the additive white Gaussian noise (AWGN) vector. Note that (1) can also be expressed into matrix from as

 Yd=HXd+Nd, (2)

where is the AWGN matrix in the data transmission stage. Denote the estimated channel as

 ^H=H+ΔH, (3)

where is the channel estimation error. If the estimated channel is used for signal detector, the signal detection problem can be formulated as

 yd[n] =Hxd[n]+nd[n] =(^H−ΔH)xd[n]+nd[n] =^Hxd[n]+nd[n]−ΔHxd[n] =^Hxd[n]+^nd[n], (4)

where is the equivalent noise in signal detector which includes the contribution of channel estimation error and original additive noise. is assumed to be Gaussian distribution, and the covariance matrix can be obtained by considering the statistical properties of the channel estimation error and detailed calculated process is shown in Appendix A.

### Ii-C Data-Aided Channel Estimation

In this section, we introduce the data-aided channel estimation approach. Conventional pilot-only based channel estimation is performed and transmitted symbols are detected in the first iteration. Then, the detected symbols are feedback to the channel estimator as additional pilot symbols to refine the channel estimation.

In the channel training stage, pilot matrix is transmitted, similar to (1), the received signal matrix corresponding to the pilot matrix can be expressed as

 Yp=HXp+Np, (5)

where is the AWGN matrix and each column for . The estimated data matrix can be expressed as

 ^Xd=Xd+Ed, (6)

where is the signal detection error matrix. In data-aided channel estimation stage, estimated are feedback to channel estimator as additional pilot. Then, the received signal matrix corresponding to can be expressed as

 Yd =HXd+Nd =H(^Xd−Ed)+Nd =H^Xd+(Nd−HEd) =H^Xd+^Np, (7)

where is the equivalent noise for additional pilot part . The statistical information of the -th column of , for , where is calculated in Appendix A and will be utilized in data-aided channel estimation stage. Then, we denote as received signal matrix corresponding to overall transmitted signal matrix. Based on (5) and (II-C), we have

 Y =(Yp Yd) =(HXp+Np,H^Xd+^Np) =H(Xp,^Xd)+(Np,^Np) =HX+N, (8)

where , can be interpreted as the equivalent pilot signal and noise in data-aided channel estimation stage, respectively.

### Ii-D Model-Driven DL for JCESD

In Section II-A, we have introduced the principle of the JCESD architecture. Compared with other DL-based JCESD architecture [17, 15], which uses a large number of data to train the black-box-based network, we construct the network architecture by employing model-driven DL and domain knowledge. In particular, we employ the LMMSE channel estimator and construct the model-driven DL detector. As in [1], the proposed model-driven DL detector is obtained by unfolding the existing iterative detector and adding several trainable parameters, which fully exploiting the domain knowledge. This architecture is promising as it inherits the superiority of traditional approach and uses DL technique to improve the performance. We will introduce the channel estimator and signal detector in detail in Sections III and IV, respectively.

## Iii LMMSE Channel Estimator

To facilitate the representation of the channel estimation problem, we apply matrix vectorization to (5) and rewrite it as

 yp=Aph+np, (9)

where , , and . We denote as the matrix Kronecker product and as the vectorization operation. In pilot-only based channel estimation stage, the LMMSE estimate of is given by

 ^hp=RhhAHp(ApRhhAHp+σ2INpNr)−1yp, (10)

where is the channel covariance matrix. Based on the property of LMMSE estimate, is a Gaussian random vector. The channel estimation error vector is also a Gaussian random vector with zero-mean and the covariance matrix can be computed as

 (11)

In data-aided channel estimation stage, considering data feedback to channel estimator, the LMMSE channel estimation is given by

 ^h=RhhAH(ARhhAH+Rnn)−1y. (12)

where , , and . The covariance matrix of the channel estimation error vector can be computed as

 RΔh=Rhh−RhhAH(ARhhAH+Rnn)−1ARhh. (13)

The covariance matrix contains the equivalent noise power of the actual pilot and additional pilot part , which is calculated in Appendix A.

## Iv Model-Driven DL Detector

In this section, we develop a model-driven DL detector, called OAMP-Net2, which has been partly presented in our early work [1]. After introducing the principle of the OAMP detector, we present the structure of OAMP-Net2 in detail. Afterwards, computational complexity and extension to soft-input and soft-output detection are also investigated.

### Iv-a OAMP Detector

The OAMP algorithm has been proposed to solve sparse linear inverse problems in compressed sensing [38] and can be utilized for MIMO detection in Algorithm 1. The goal of the OAMP algorithm is to recover the transmitted signal vector from the received signal vector . The principle of the algorithm is to decouple the posterior probability into a series of in an iterative way, when estimated channel and received signal are available.

The OAMP detector is presented in Algorithm 1 and mainly contains two modules: linear estimator (14) and nonlinear estimator (15

). The error variance estimators (

16) and (17) are the average variance of the two error vectors and , where and are used to measure the accuracy of the output in the linear and nonlinear estimators, respectively. They are defined as

 v2t=E[∥qt∥22]Nt,τ2t=E[∥pt∥22]Nt, (18)

and can be computed by employing (16) and (17).

#### Iv-A1 Linear Estimator

The matrix in linear estimator (14) can be the transpose of , the pseudo inverse of or the LMMSE matrix. From [38], the optimal one is

 Wt=Nttr(^Wt^H)^Wt, (19)

where is the LMMSE matrix given by

 ^Wt=v2t^HH(v2t^H^HH+R^nd^nd)−1, (20)

where is expressed in (16), and is the covariance matrix of the equivalent noise in signal detector, which includes the contribution of the channel estimation error and original additive noise , and has been discussed in Section II-B. The matrix is called de-correlated when , where , and thus ensures the entries of are uncorrelated with those of and mutually uncorrelated with zero-mean and identical variances.

#### Iv-A2 Nonlinear Estimator

The nonlinear estimator in the OAMP detector is constructed by MMSE estimate of , which is with respect to the equivalent AWGN channel

 rt=xd,t+wt, (21)

where . As the transmitted symbol is from the discrete constellation set , corresponding MMSE estimate for each element of the estimated symbol vector is given by

 ^x(i)d,t+1=E{xi|ri,τt}=∑sisiNC(si;ri,τ2t)p(si)∑siNC(si;ri,τ2t)p(si), (22)

where is the prior distribution of the transmitted symbol and is given by

 p(xi)=∑j∈M1√Mδ(xi−sj). (23)

From (14), (15), (17) and (21), we observe that and are the prior mean and variance in MMSE estimator that control the accuracy and convergence of estimated result . The OAMP detector uses an iterative manner to obtain and , and the step-size for the update of and will influence the final performance. Instead of using complicated analytical method to find an optimal step-size, we will use a DL approach for providing an appropriate step-size to update and and constructing learnable nonlinear estimator to improve the detection performance.

### Iv-B OAMP-Net2 Detector

In this section, we introduce the OAMP-Net2 detector, which is developed by unfolding the OAMP detector. After presenting the network architecture, we elaborate the linear and nonlinear estimators, error estimators, and learnable variables in OAMP-Net2 detector.

#### Iv-B1 Network Architecture

The structure of the OAMP-Net2 is illustrated in Fig. 2, which is obtained by unfolding the OAMP detector and adding several trainable parameters. The input of the OAMP-Net2 is the received signal vector, , and the estimated channel matrix, , while the final output is .

The network consists of cascade layers and each has the same architecture but different trainable parameters. For the -th layer of the OAMP-Net2, the input is the estimated signal from the ()-th layer. Signal detection is performed as follows

 rt=^xd,t+γtWt(yd−^H^xd,t), (24)
 ^xd,t+1=ηt(rt,τ2t), (25)
 v2t=∥y−^H^xd,t∥22−tr(R^nd^nd)tr(^HH^H), (26)
 (27)

The critical difference between the OAMP and OAMP-Net2 detectors is there are learnable variables in each layer. By optimizing these parameters in the training process, the detection performance can be improved. When , and , the OAMP-Net2 is reduced to the OAMP detector. The matrix in the OAMP-Net2 is a revised structure of in the OAMP detector with learnable parameter to regulate the error variance . We will introduce the linear estimator, , nonlinear estimator, , error estimators, and , and learnable variables, , respectively.

#### Iv-B2 Linear and Nonlinear Estimator Modules

The linear estimator module in (24) is a revised structure of (14) in the OAMP algorithm by adding corresponding learnable parameter . The learnable parameter can be interpreted as the step-size for the update of , and is in each iteration in the OAMP detector. The de-correlated matrix is given in (19). The nonlinear estimator in OAMP-Net2 is constructed by the divergence-free estimator

 ηt(rt,τ2t)=ϕt(E{xd|rt,τt}−ξtrt), (28)

where is the MMSE estimate of with the equivalent AWGN channel (21), and has the same expression with (22). Compared with the MMSE estimator (22) in the OAMP detector, the divergence-free estimator (28) considers the contribution of the linear estimator and learnable parameters . The MMSE estimator (22) can be interpreted as a special case of (28) by setting and .

#### Iv-B3 Error Variance Estimators

The error variance estimators in (26) and in (27) play important roles in providing appropriate variance estimates required for the linear and nonlinear estimators in the OAMP-Net2 detector. For error variance , we adopt the same estimator with the OAMP detector in (26). For error variance , we construct the estimator based on two assumptions on error vectors and in [38] and incorporate learnable variables (). The detailed derivation for the two variance estimators and is provided in Appendix B. Furthermore, we substitute by for a small positive constant to avoid stability problem.

#### Iv-B4 Learnable Variables

The learnable variables are optimized in the training process for OAMP-Net2. The Bayes-optimal property of the OAMP algorithm has been proven in [39], but it is derived in the large system with unitarily-invariant matrix . In fact, the performance of the OAMP detector is far from the Bayes-optimal performance in practical finite-dimensional MIMO systems, especially when there are strong spatial correlation and channel estimation error. These observations motivate us to improve the original iterative detector with several trainable parameters to adapt to various channel environments.

Similar to the TPG detector [21] and OAMP-Net[1], the OAMP-Net2 uses two learnable parameters to adjust the linear and nonlinear estimators, and to provide appropriate stepsize for the update of the mean and variance in the MMSE estimator. On the other hand, the linear estimator (24) is related to gradient descent algorithm and its convergence behavior and performance are determined by appropriate step-size of moving to the search point. The optimal step-size can be learned from the data for the update of the prior mean in the MMSE estimator. Furthermore, the parameter has the similar function for the error variance , which can compensate for the channel estimation error and regulate the to provide appropriate value for the update of the prior variance in the MMSE estimator.

The parameters (, ) in the nonlinear estimator play important roles in constructing an appropriate divergence-free estimator, which has been discussed in [38]. In precise, the divergence-free estimator (28) can be applied in the OAMP detector, but the and are related to the prior distribution of the original signal and difficult to calculate. Therefore, MMSE estimator (22) is considered for simplicity in the OAMP detector. In our early work [1], we set and in OAMP-Net and use MMSE estimator (22). By contrary, we adaptively learn two parameters and in OAMP-Net2 to construct the nonlinear estimator satisfying the divergence-free property.

### Iv-C Computational Complexity

The computational complexity required for the OAMP-Net2 is in each iteration. Similar to the OAMP detector and OAMP-Net, the computational complexity is dominant by the matrix inverse in . As is relatively small (e.g., or ) on small-size MIMO systems, the matrix inverse operation is always acceptable. From Fig. 2, the total number of trainable variables is equal to since each layer of the OAMP-Net2 contains only four trainable variables . By contrast, trainable variables are required to train in OAMP-Net. Furthermore, the number of trainable variables of the OAMP-Net2 and OAMP-Net are independent of the number of antennas and , and only determined by the number of layers . This is an advantageous feature for large-scale problems, such as massive MIMO detection. With only few trainable variables, the stability and speed of convergence can be improved in the training process.

### Iv-D Soft-Input and Soft-Output

As many modern digital communication systems need to produce a probabilistic estimation of the transmitted data given the observations to probabilistic channel decoder. A significant issue is whether the MIMO detector can use the soft-information from the decoder and produce the soft-output. Different from the DetNet in [18] that only can provide the soft-output, our proposed detector are the soft-input and soft-output receiver and therefore can achieve the turbo equalization. We only provide the principle of the OAMP-Net2-based turbo receiver and the specific experimental results are outside the scope of this paper and will be conducted in the future.

From (28), the OAMP-Net2 can decouple the joint posterior probability into a series of marginal posterior probability . The marginal posterior probability is used to produce the soft output log-likelihood ratios (LLR), which is given by

 LA(bj,k)=log∑S+jP(xj|rt,τt)∑S−jP(xj|rt,τt), (29)

where is the -th bit in the transmitted symbol , and and denote the subsets of the constellation symbols with the -th bit being and , respectively. After interleaved and delivered to the channel encoder, extrinsic LLR can be computed and given to the OAMP-Net2 detector as updated prior information. The detector and decoder iteratively exchange information until convergence.

### Iv-E Practical Implementation

The developed detector can be divided into two stages. In the offline training stage, we obtain the optimized parameters,

, for different SNRs and channel correlation coefficients based on TensorFlow platform and GPUs’ powerful computing ability. The optimized parameters will be stored to detect the modulated symbols in the deployment stage. The OAMP-Net2 detector can be interpreted as a new iterative detector after training. The incorporated learnable parameters can adapt to practical channel and compensate for channel estimation error to improve the detection performance. Although aforementioned implementation process is in an offline manner, the proposed detector can also be implemented by online training to adapt to the fluctuations in the channel conditions owing to the superiority of its low demand for training data and computational resources.

## V Numerical Results

In this section, we provide numerical results to show the performance of the proposed model-driven detector for MIMO detection. First, we elaborate the implementation details and parameter settings. Then, the performance of the OAMP-Net2 is presented under i.i.d. and correlated Rayleigh MIMO channels with perfect CSI. Afterwards, we investigate the performance of the OAMP-Net2-based JCESD architecture. Finally, the robustness of the OAMP-Net2 to SNR and channel correlation is demonstrated.

### V-a Implementation Details

In our simulation, OAMP-Net2 is implemented in Tensorflow by using a PC with GPU NVIDIA GeForce GTX 1080 Ti. The SNR of the system, defined as

 SNR=E∥Hxd∥22E∥nd∥22, (30)

is used to measure the noise level. We assume the same SNR in pilot and data transmission stage. Each element of the i.i.d. Rayleigh MIMO channel is while the correlated Rayleigh MIMO channel is described by the Kronecker model,

 Hc=R1/2RGR1/2T, (31)

where and are the spatial correlation matrix at the receiver and the transmitter, respectively, and are generated according to the exponential correlation model [40] with the same correlation coefficient , and is the i.i.d. Rayleigh fading channel matrix. Furthermore, to implement OAMP-Net2 in the Tensorflow, we consider equivalent real-valued representation for all aforementioned problem models.

The training data consists of a number of randomly generated pairs . For each pair , channel is randomly generated from the i.i.d. or correlated Rayleigh MIMO channel model. The data is generated from the -QAM modulation symbol with being the modulation order. We train the network with epochs. At each epoch, the training set contains different samples , and different validation samples. The validation sets are used to choose the best network for each epoch in the training stage. In the test stage, we generate the data to test the network until the number of bit errors exceed

. The network is trained using the stochastic gradient descent method and Adam optimizer. The learning rate is set to be

and the batch size is set to . To train the learnable variables well, the learning rate is set to be and validation samples is in high SNR regime (SNR = or above). Except for special instructions, we choose the loss as the cost function in all experiment settings, which is defined as

 l2(Ω)=1D∑d(i)∈D∥x(i)d−^xd,T+1(y(i))∥22, (32)

where is the training set of a mini-batch and .

In general, the optimal variables may be different for different SNRs. Therefore, we train the OAMP-Net2 for each SNR separately, which means that the training and test data are generated with the same fixed discrete SNR value. For the correlated Rayleigh MIMO channel, we train the network for each correlation coefficient , respectively. Furthermore, the network needs to be retrained if different modulation symbols are utilized. We also consider the robustness of the OAMP-Net2 to SNR and channel correlation mismatches.

### V-B Perfect CSI

In this section, we study the performance of OAMP-Net2 for MIMO detection with perfect CSI. After analyzing its convergence, the detection performance of the OAMP-Net2 under i.i.d. and correlated Rayleigh MIMO channels is investigated.

#### V-B1 Convergence analysis

First, we analyze the convergence of the OAMP-Net2 and OAMP detectors. Fig. 3 illustrates the bit-error rate (BER) performance versus the number of layers (iterations) under dB and dB with QPSK and -QAM constellations. From Fig. 3, both the OAMP-Net2 and OAMP detectors converge within four layers (iterations) for all the cases and OAMP-Net2 can obtain more performance gain when SNR = dB, which means the OAMP-Net2 detector improves OAMP detector significantly in high SNR regime. Furthermore, compared with -QAM, the performance gap between OAMP and OAMP-Net2 is larger for QPSK. Based on the above observations, we set OAMP and OAMP-Net2 detectors four layers in following simulation when SNR dB, and ten layers when SNR = dB or above except for specific instructions.

#### V-B2 Performance Comparison

Fig. 4 compares the BER performance of the OAMP, OAMP-Net [1], TPG [21], and OAMP-Net2 detector. Both and MIMO systems are investigated. The TPG detector [21] is only developed for QPSK with two trainable parameters in each layer and has similar architecture to OAMP-Net. From the figure, the OAMP-Net, TPG, and OAMP-Net2 detectors outperform the OAMP detector in all setting, which demonstrates the deep learning can improve the OAMP detector by learning the optimal parameters. But the performance gain is limited for -QAM. Compared with OAMP-Net in [1], the proposed network OAMP-Net2 has a significant performance improvement in all setting and outperforms the TPG detector in QPSK. Specifically, compared with the OAMP detector in MIMO with QPSK, the OAMP-Net can obtain dB performance gain while the gain for OAMP-Net2 is approximately dB when BER= is required. Since the OAMP-Net2 can learn the optimal parameter to construct the nonlinear estimator satisfying the divergence-free property. For high-dimensional MIMO systems in Fig. 5, similar conclusions can be obtained.

#### V-B3 Correlated MIMO Channel Performance

Fig. 6 illustrates the BER of the OAMP-Net2 and the OAMP detector under correlated Rayleigh MIMO channel with . We set the OAMP and OAMP-Net2 detectors ten layers with different orders of modulation. From the figure, all algorithms have performance degradation compared with the i.i.d. Rayleigh MIMO channel, But the OAMP-Net2 detector still have the performance improvement over the OAMP detector in all setting and can obtain more performance improvement in MIMO system. Specifically, if we target for BER= with QPSK modulation, BER performance improves about dB for MIMO. By contrast, it is approximately dB for MIMO. But the performance gain of the OAMP-Net2 detector decreases with the increase of the modulation order. For example, if we target for BER= in MIMO, SNR improves about dB for QPSK modulation. By contrast, it decreases to approximately dB for -QAM .

### V-C JCESD Performance

In the above, all detectors are investigated with accurate CSI. In this section, we consider OAMP-Net2-based JCESD architecture for a MIMO system. Both OAMP and OAMP-Net2 has four layers. In order to avoid gradient vanishing, we use the summation of -loss over all layers to train the model, which is defined as

 l2(Ω)=L∑l=1T∑t=11D∑d(i)∈D∥x(i)d−^x(l)d,T(y(i))∥22, (33)

and the learning rate is set to be .

We consider orthogonal pilot in channel training stage, where the pilot matrix is chosen by selecting

columns of the discrete Fourier transformation (DFT) matrix

. Fig. 7 shows the BERs of the OAMP and OAMP-Net2 detectors in the JCESD architecture, where means no data feedback to channel estimator and means the detected data are feedback twice to channel estimator. From the figure, the OAMP-Net2 detector outperforms the OAMP detector in all settings significantly. Specifically, if we target for BER= with -QAM, BER performance improves about dB for and improves about dB for . Furthermore, we observe that OAMP-Net2 detector without data feedback () outperforms OAMP detector with data feedback twice (), which demonstrates learning some trainable variables can compensate for the channel estimation error to improve the equivalent SNR in detection stage. For QPSK modulation, the performance gain are dB and dB for and , respectively, when we target for BER=. In addition, marginal performance is obtained by data feedback for the OAMP-Net2 detector with QPSK because learnable parameters have strong ability to compensate for channel estimation error. Therefore, no data feedback is needed in this case.

### V-D Robustness

In this section, we analyze the robustness of the OAMP-Net2 against the SNR and correlation mismatches. As aforementioned numerical results are performed with known SNR and channel correlation, an interesting question is whether the OAMP-Net2 trained under fixed SNR and channel correlation can still obtain good performance on the whole SNR and correlation regime. Because of limited data and computing resources are available for online training, verifying the robustness of offline-trained network is particularly meaningful.

Fig. 8 presents the BER performance of OAMP and OAMP-Net2 with SNR and correlation mismatches in MIMO system. The network is trained in the correlated Rayleigh MIMO channel (31) with channel correlation coefficient and SNR=dB, and tested with mismatched channel correlation coefficient and SNR, which are shown in the figure. From the figure, the trained network with correlation mismatch still outperforms the OAMP detector significantly in all setting, which demonstrates the OAMP-Net2 have strong robustness to mismatch. Interestingly, the trained network even outperforms the OAMP detector with in high SNR regime (SNR=- dB). Since the learnable variables successfully compensates for the disadvantages of channel correlation. Compared with perfect SNR333 Perfect SNR means that network in the training and test stage have the same SNR., the OAMP-Net2 detector with SNR mismatch has little performance deterioration except for SNR = dB or above, when we target .

## Vi Conclusion

We have developed a novel model-driven deep learning framework for joint MIMO channel estimation and signal detection. The signal detector network is obtained by unfolding the OAMP detection algorithm. The OAMP-Net2 detector inherits the superiority of the Bayes-optimal signal recovery algorithm and DL technique, and thus present excellent performance. The network are easy and fast to train because only few adjustable parameters are required to be optimized. Simulation results demonstrate that significant performance gain can be obtained by learning corresponding optimal parameters from the data to improve the detector and compensate for the channel estimation error. Furthermore, the signal detection network exhibit strong robustness to SNR and channel correlation mismatches.

## Appendix A: derivation for Covariance matrices

In order to derive the covariance matrix of the equivalent noise in signal detection stage, we should evaluate the statistical properties of the for , which can be expressed as

 Cov[zd[n]]=E[zd[n](zd[n])H]. (34)

As the covariance matrix for each symbol index is same and the data symbol is independent between different antennas, we omit the index in and have

 E[zd,izHd,k]=⎧⎪ ⎪⎨⎪ ⎪⎩0,∀i≠kNt∑j=1σ2Δhi,j,i=k (35)

where is the -th element of the , is the -th element of the channel estimation error matrix , and is -th element of the . The is the variance of -th element in channel estimation error matrix which can be obtained from in (11) or in (13). By considering the contribution of original additive noise, the covariance matrix can be obtained as

 ^Vd=diag(Nt∑j=1σ2Δhi,j+σ2,…,Nt∑j=1σ2Δhi,j+σ2). (36)

The covariance matrix will be utilized in the OAMP-Net2 detector as the covariance matrix of the equivalent noise .

Next, we compute the covariance matrix of the equivalent noise in data-aided channel estimation stage using similar approach. For each time index , We denote and , where is the -th column of the signal detection error matrix . In a similar way, we have

 E[zi,nzHk,n]=⎧⎪ ⎪⎨⎪ ⎪⎩0,∀i≠kNt∑j=1σ2ej,n/Nr,i=k (37)

because and is the variance for -th element of the . Therefore, the covariance matrix is given by

 ^Vp[n]=(Nt∑j=1σ2ej,n+σ2)INt. (38)

The covariance matrix will be utilized in channel estimator to evaluate the covariance matrix of the equivalent noise where . By considering the different covariance matrix of the actual pilot and additional pilot, we have

 Rnn=[σ2INpNr~V] (39)

where denotes the covariance matrix of the noise for actual pilot while

 ~V=⎡⎢ ⎢ ⎢⎣^Vp[1]⋱^Vp[Nd]⎤⎥ ⎥ ⎥⎦ (40)

denotes the covariance matrix of the equivalent noise for the additional pilot .

## Appendix B: derivation for variance estimators

To derive the expressions for the error variance estimators and in OAMP-Net2, we use similar method to [31]. We should indicate that and in OAMP-detector are based on the following two assumptions about error vectors and [38].

Assumption 1: consists of i.i.d. zero-mean Gaussian entries independent of .

Assumption 2: consists of i.i.d. zero-mean Gaussian entries independent of and .

From the similar viewpoint, we obtain the error variance estimators and (26) and (27 in OAMP-Net2 using Assumption 1 and 2, and considering the effect of the learnable variables and . For convenience, we will use , , and to refer , , and , respectively. Based on assumption 2, consists of i.i.d. zero-mean Gaussian entries which is independent of and , which means that

 E[(^xt−x)H