I Introduction
Millimeterwave (mmWave) communications is a key technology for 5G, which can address the bandwidth shortage problem in current mobile systems [1, 2, 3, 4, 5]. The largescale antenna array is needed to compensate for the severe path loss and penetration loss at the mmWave wavelengths [6, 7]. However, the substantial increase in the number of antennas leads to nontrivial practical constraints. The traditional fulldigital multipleinput and multipleoutput (MIMO) beamforming which requires one dedicated radio frequency (RF) chain per antenna element is prohibitive in mmWave systems due to the unaffordable hardware cost and power consumption of a large number of antenna elements [8, 9]. By separating the whole beamformer into a lowdimensional baseband digital one and a highdimensional analog one implemented with phase shifters, the hybrid analog and digital beamforming (HBF) architecture has been shown to dramatically reduce the number of RF chains while guaranteeing a sufficient beamforming gain [10, 9, 11, 12, 13, 15, 14].
Ia Related Works and Motivations
Compared with the traditional fulldigital beamforming design, in HBF, besides the difficulty of the joint optimization over the four beamforming variables (the transmit and receive analog and digital beamformers), the constant modulus constraints of the analog beamformers due to the phase shifters make the problem highly nonconvex and difficult to solve [9, 16, 19]. Most existing works overcome the difficulty by first decoupling the original problem into hybrid precoding and combining subproblems and then focusing on the constant modulus constraint in solving the subproblems. One effective and widely used approach is to regard the HBF design as a matrix factorization problem and to minimize the Euclidean distance between the hybrid beamformer with a fulldigital beamformer [9, 17, 18]. To solve this matrix factorization problem, in [9], the authors exploited the spatial structure of the mmWave propagation channels and proposed spatially sparse precoding and combining algorithms via the orthogonal matching pursuit (OMP) method. In [18], a manifold optimization (MO) based HBF algorithm, as well as some lowcomplexity algorithms, was proposed. Besides the matrix factorization approach, another idea for HBF design is to tackle the original problem directly. In [19, 20], the closedform solution of digital beamformers was first derived according to the original objective, followed by several iterative algorithms for the analog ones with the constant modulus constraint.
All the above works, as well as most of the other previous studies, design the HBF with the objective of maximizing the spectral efficiency. By recalling the joint precoding and combining designs in conventional fulldigital MIMO systems, besides spectral efficiency, the mean square error (MSE) is another important metric [21, 22, 23, 24]. One direct motivation to consider MSE is that a practical system is normally constrained to some particular modulation and coding scheme instead of the Gaussian code [22], and thus MSE is a direct performance measure to characterize the transmission reliability. Furthermore, it has been shown that the variants of the MSE such as sumMSE, minmax MSE, modified MSE, weighted MSE, etc., are related to other important performance measures (e.g., signal to interference plus noise ratio (SINR) and symbol error rate) [21, 22, 23, 24, 25]. For example, it has been shown in [21, 22] that the MSE is related to the SINR and SER (BER) metrics in the beamforming design for the fulldigital MIMO systems with multiple data streams. Thus, it is of great interest to take MSE as an alternative optimization objective for HBF. Actually, even in some existing HBF designs with the spectral efficiency as the objective, the hybrid receive combining matrices were optimized by minimizing the MSE instead [9, 19, 20, 28]. Moreover, in [26, 30, 31], it was illustrated that precoding design based on the minimum MSE (MMSE) criterion can also achieve good performance in spectral efficiency.
There have been some works on the HBF design using the MMSE criterion for mmWave systems. In [26], the authors focused on the hybrid MMSE precoding at the transmitter side and proposed an OMPbased algorithm. To improve the system performance, in our previous work [27], we tackled the MMSE precoding problem directly and proposed an algorithm based on the general eigendecomposition (GEVD) method. In [17], the authors replaced the hybrid MMSE precoding problem by the one of factorizing the optimal fulldigital MMSE precoder. In their later work [28], the hybrid MMSE combiner was further considered with a similar approach to that in [9, 16], aiming at minimizing the weighted approximation gap between the hybrid combiner and a fulldigital combiner. However, all of these works considered the narrowband scenario and cannot be straightforwardly extended to the broadband scenario, which is more relevant for mmWave communication systems.
IB Contributions and Paper Organization
In this paper, we investigate the joint transmit and receive HBF optimization for broadband pointtopoint mmWave systems, aiming at minimizing the modified MSE [24]. Besides the aforementioned challenges in the joint optimization of the four beamforming variables and the constant modulus constraint on the analog beamformers, it is also worth noting that in the broadband scenario, yet another challenge is that the digital beamformers should be optimized for different subcarriers while the analog one is invariant for the whole frequency band. Aiming at these challenges in the MMSE based HBF design for broadband mmWave MIMO systems, the contributions in this paper can be summarized as follows.

Instead of factorizing the optimal fulldigital beamformer in the indirect HBF design approach [9, 17, 18], we optimize the hybrid beamformers by directly targeting the MMSE objective for better performance. Different from the conventional MMSE based HBF designs [17, 26, 28] which only considered the narrowband scenario, we propose a general HBF design approach for both the narrowband and broadband mmWave MIMO systems. In particular, we decompose the original sumMSE minimization problem into the transmit hybrid precoding and receive combining subproblems, and show that the two subproblems can be unified in almost the same formulation and solved through the same procedure. The alternating minimization method is adopted to solve the overall HBF problem, for which a novel initialization method is proposed to reduce the number of iterations. Furthermore, following the approach of extending the sumMSE minimization problem to the weighted sumMSE minimization (WMMSE) problem and connecting it to the spectral efficiency maximization problem in the narrowband scenario [28], we show that in the broadband scenario the proposed MMSE based HBF algorithms can be generalized to the ones for maximizing the spectral efficiency.

To deal with the constant modulus constraint in the analog beamforming optimization, we apply the manifold optimization (MO) method [18, 33]. In contrast to the application of the MO method in [18] for minimizing the Euclidean distance between the hybrid beamformer and the target fulldigital beamformer, in this study, the MO method is applied to directly minimize the sumMSE and the new contribution is to derive the more complicated Euclidean conjugate gradient of the sumMSE with some skilled derivations so that the Riemannian gradient can be computed. This provides a direct approach with guaranteed convergence to solve the MMSE HBF problem instead of the indirect approach in [18].

To avoid the high complexity in the MOHBF algorithm, we propose several lowcomplexity algorithms. In the narrowband scenario, we show that the analog beamforming matrix can be optimized columnbycolumn with the GEVD method. In the broadband scenario, we derive both upper and lower bounds of the original objective and then propose two eigendecomposition (EVD) based HBF algorithms. Compared with the existing algorithms based on the OMP method [26, 28, 17], the proposed algorithms directly tackle the original sumMSE objective without the restriction of the space of feasible solutions and thus result in better performance.
The rest of the paper is organized as follows. For the ease of presentation, we start with the narrowband scenario and introduce the system model along with the HBF problem formulation in Section II. In Section III, we present the basic idea and the optimization procedure, and propose the MOHBF and GEVDHBF algorithms. In Section IV, we extend the problem formulation and design procedure to the broadband scenario, and propose three HBF algorithms. In Section V, we extend the MMSE based HBF design to the WMMSE one for maximizing the spectral efficiency. We discuss the convergence property and analyze the computational complexity for all the proposed HBF algorithms in Section VI. We demonstrate various numerical results in Section VII. Finally, we conclude the paper in Section VIII.
Throughout this paper, boldfaced upper case letters, boldfaced lower case letters, and lightfaced lower case letters are used to denote matrices, column vectors, and scalar quantities, respectively. The superscripts
, , and represent matrix (vector) transpose, complex conjugate, and complex conjugate transpose, respectively. denotes the Euclidean norm of a vector. , and denote the trace and the Frobenius norm of a matrix. denotes the conjugate gradient of a function. denotes the expectation operator. denotes the absolute value or the magnitude of a complex number. denotes the th entry of a matrix .Ii System Model and Problem Formulation
Iia System Model
For the ease of presentation, we first consider a pointtopoint narrowband mmWave MIMO system with HBF as in Fig. 1, where data streams are sent and collected by transmit antennas and receive antennas, respectively. Both the transmitter and receiver are equipped with RF chains, where . The original symbol vector, denoted by with , is firstly precoded through an digital beamforming matrix , and then an analog beamforming matrix which is implemented in the analog circuitry using phase shifters. From the equivalent baseband representation point of view, the precoded signal vector at the transmit antenna array can be represented as . Without loss of generality, the normalized transmit power constraint is set to .
Similar to that in [9, 19], the mmWave propagation channel is characterized by a geometrybased channel model with clusters and rays within each cluster. Considering the mmWave system with a halfwave spaced uniform linear array (ULA) at both the transmitter and the receiver, the channel matrix can be represented as
(1) 
where denotes the complex gain of the th ray in the th propagation cluster, and and denote the normalized responses of the transmit and receive antenna arrays to the th ray in the th cluster, respectively, where and denote the angles of arrival and departure.
With a similar HBF at the receiver, i.e., an analog combiner followed by an digital baseband combiner , we finally have the processed signal as
(2) 
where denotes the additive noise vector at the
receive antennas satisfying the complex circularly symmetric Gaussian distribution with zero mean and covariance matrix
, i.e., . Similar to existing works on the HBF design (e.g. [10, 18, 19, 20]), in this paper, it is assumed that perfect channel state information (CSI) is available at both the transmitter and receiver and that there is perfect synchronization between them.IiB Problem Formulation
In this work, we take the modified MSE [24] as the performance measure and optimization objective for the joint transmit and receive HBF design, which is defined as
(3) 
where is a scaling factor to be jointly optimized with the hybrid beamformers. By substituting (2) into (3) and after some mathematical manipulations, we have
(4)  
where , are defined as the overall hybrid transmit and receive beamformers, respectively. Notice that since the analog beamformers are assumed to be implemented with phase shifters which only adjust the phases of the input signals, the elements of analog beamformers should satisfy the constant modulus constraint, namely for and , and for and . With the derived MSE expression in (4), the transmit power constraint and the constant modulus constraint of the phase shifters, the HBF optimization problem in the narrowband scenario can be formulated as
(5) 
It is worth noting that there are mainly three reasons or advantages for introducing the scaling factor and taking the modified MSE as the objective function. First, as the joint transmit and receive HBF problem will be decoupled into the hybrid precoding and combining subproblems, adjusting achieves a better performance for the precoding optimization by considering the noise effect (which is also referred to as the transmit Wiener filter) [24]. Second, is also helpful in dealing with the total transmit power constraint and thus simplifies the precoding optimization procedure [30, 31]. Finally, by introducing , the hybrid precoding and combining subproblems can be unified and solved in the same way aiming at the same modified MSE objective. These advantages will be elaborated in more details in the following sections.
Iii Hybrid Beamforming Design Based on The MMSE Criterion
Since the HBF problem in (5) involves a joint optimization over five variables, along with nonconvex constraints, it is unlikely to find the optimal solution. A suboptimal but efficient way to overcome the difficulties is to separate the original problem into two subproblems corresponding to the optimization for the hybrid transmit precoder and receive combiner, respectively, and solve each independently [9, 19, 20, 28]. Taking this approach, we propose several HBF algorithms in the following two subsections. Finally, we develop the whole alternating minimization algorithm for the HBF optimization based on the MMSE criterion.
Iiia Hybrid Transmit Design
This section focuses on the hybrid precoder design (including ) in (5) by fixing the receive combining matrices and . As shown in [24, 26, 31], the original precoder can be separated as , where is an unnormalized baseband precoder. With this separation, the precoder optimization problem can be formulated as
(6) 
where denotes the equivalent channel of the concatenation of the air interface channel and the hybrid receive combiner. Our optimization approach is to first derive the optimal digital precoding matrix and the scaling factor by fixing , then derive the resulting objective as a function of , and finally optimize by further minimizing the objective with the constant modulus constraint. Due to the transmit power constraint, it can be proved by contradiction that the optimal solution must be achieved with the maximum total transmit power, i.e., the optimal is given by
(7) 
Then according to the KarushKuhnTucker (KKT) conditions, the closedform solution of the optimal is given by
(8) 
where is defined for notational brevity. Substituting the optimal and into (6) and after some mathematical derivation, the resulting MSE is given by^{1}^{1}1Note that the above derivations benefit from the introduction of . To show this, it can be checked that if we remove from (6) (or just set in (6)), it is highly challenging to get a closedform expression of via the KKT conditions and further get a closedform expression of the MSE as a function of for the optimization of the analog precoder.
(9) 
The optimizing problem in (6) is now reduced to the following one for the optimization of
(10) 
Here we propose two algorithms for optimizing the analog precoding matrix with the constant modulus constraint, which are based on the MO and GEVD methods, respectively.
IiiA1 Analog Precoder Design Based on the MO Method
To deal with the constant modulus constraint, the MO method [18][33] can be applied to obtain a local optimal . The basic idea is to define a Riemannian manifold for with the consideration of the constant modulus constraint, and iteratively update this optimization variable on the direction of the Riemannian gradient (i.e., a projection of the Euclidean conjugate gradient onto the tangent space of a point on the Riemannian manifold) in a similar way to that in the conventional Euclidean gradient descent algorithm (the details can be referred to [18]). However, the application of the MO method is not straightforward, and the most difficult part is the derivation of the conjugate gradient in the Euclidean space, in order to obtain the associated Riemannian gradient. It should be mentioned that for the scalar function associated with a complexvalued variable , the conjugate gradient [32] is defined as . By defining in (9) for notational brevity, we have the following lemma for the conjugate gradient.
Lemma 1.
The conjugate gradient of the function with respect to is given by
(11)  
Proof: According to some basic differentiation rules for complexvalue matrices [32], the differential of can be expressed as
(12)  
where denotes the differential with respect to while taking as a constant matrix during the derivation of the conjugate gradient . The second equality in (12) holds due to the properties of and .
On the other hand, we can directly compute from (9). According to some differentiation rules for differentiating a matrix’s trace and inverse, we express as
(13) 
It can be further derived that
(14)  
where
(15) 
By substituting (15) and (14) into (13) and using again , we have
(16)  
With the derived Euclidean conjugate gradient, the manifold optimization can be applied to solve the problem with the constant modulus constraints [33]. The overall MOHBF algorithm is summarized in Algorithm 1, where the iteration index is denoted in the subscript of . In particular, the detailed operation in the 4th step is given as follows. First, project the Euclidean gradient onto the tangent space to obtain the Riemannian gradient. Second, search a point in the tangent space along the Riemannian gradient and use the ArmijoGoldstein condition to determine the step size. Finally, retract the searched point back to the manifold.
IiiA2 Analog Precoder Design Based on the GEVD Method
The above algorithm for optimizing the analog precoding matrix in (10) is essentially a gradient based algorithm, where the computational complexity is proportional to the number of iterations and is related to the form of the objective function and the stop condition. In this part, we propose a lowcomplexity algorithm based on GEVD. According to [19], for largescale MIMO systems, it can be approximated that based on the fact that the optimized analog beamforming vectors for different streams are likely orthogonal to each other. With this approximation, (9) can be simplified as
(17) 
With this simplified form, it can be shown that the analog precoding matrix can be optimized columnbycolumn. Specifically, define as the remaining submatrix of after removing the th column . Further define . Then, using the fact that for a fullrank matrix and a rankone matrix , the MSE expression in (17) can be written as
(18) 
where and are both Hermitian matrices. It is seen from (18) that the MSE expression is separated into two terms which are related to and , respectively. By fixing , becomes a function on in the second term in (18). As both and are Hermitian and is positive definite, according to [35], the optimal in the sense of maximizing the last term in (18) or minimizing the whole term in (18
) is the eigenvector associated with the maximum generalized eigenvalue between
and , which can be obtained via the GEVD operation. To further take the constant modulus constraint into account, a simple but effective way is to only extract the phase of each element in the generalized eigenvector. By applying the above GEVD and phase extraction operations for each column and repeating them for the whole matrix until the stop condition is satisfied, we finally get the optimized analog precoding matrix. The overall GEVDHBF algorithm is summarized in Algorithm 2.IiiB Hybrid Receive Combiner Design
By fixing the updated precoding matrices and along with the scaling factor , the optimizing problem in (5) can be reduced to the following one for the hybrid receive combiner
(19) 
where and . Similarly, by differentiating the objective function of (19) with respect to and setting the result to zero, we have the optimal as follows
(20) 
IiiC Alternating Minimization for Hybrid Beamforming
IiiC1 Alternating Optimization
A joint hybrid precoding and combining design based on the MMSE criterion can be developed by iteratively and alternatively using the hybrid precoding design in Section IIIA and the hybrid combining design in Section IIIB. Specifically, during the th iteration, first for the optimization of the hybrid precoder, by updating the problem in (6) with the optimized combiners and in the th iteration, the hybrid precoding matrices , and the scaling factor are optimized via the MOHBF or the GEVDHBF algorithm. Similarly, with the new hybrid precoder, the hybrid combining optimization problem in (19) is then updated and solved via the same algorithm. This alternating optimization is repeated until a stop condition is satisfied.
IiiC2 Stopping Condition
To distinguish the iteration of the alternating minimization between the transmitter and receiver beamforming optimization and the iteration in the optimization of the analog beamformer (i.e., Algorithms 1 and 2), we refer to the former as the outer iteration and the latter as the inner iteration. The stopping condition of these two iterations can be set as either the number of iterations exceeding a specified value , or the relative difference between the MSE values of two consecutive iterations becoming smaller than a specified value . For example, considering a typical system configuration in Section VII, according to the observation in simulations, good performance can be achieved when we set for both the inner and outer iterations for the alternating MOHBF algorithm and set for the inner iteration and for the outer iteration for the alternating GEVDHBF algorithm.
IiiC3 Beamforming Initialization
It is worth noting that the number of iterations for the alternating optimization highly depends on the initialization of the beamformers. One simple idea is to randomly generate a hybrid combiner (precoder) for the optimization of the precoder (combiner) at the beginning of the alternating minimization. This method does not require extra information, but may need a lot of iterations to converge to a local optimal point with certain performance loss. As the concatenation of the hybrid beamformer will gradually approach the fulldigital one during the iterations, we propose to take a fulldigital beamformer as the initialization for better convergence. Specifically, the optimal fulldigital precoder based on the MMSE criterion proposed in [21] can be used here. Assuming without loss of generality that the alternating optimization starts from the transmit beamforming problem in (6), we assume that there is a virtual fulldigital combiner at the receiver in the initialization step. Namely, we initialize the concatenation of the hybrid combiner, , as the optimal fulldigital combiner in [21] and substitute it into (6) for the precoder optimization in the first iteration. We refer to the proposed initialization method as the virtual fulldigital beamformer (VFD) method. It is worth noting that as the VFD initialization method assumes a virtual fulldigital beamformer at one side, which generally cannot be directly implemented using the HBF structure, at least one outer iteration is needed to obtain the hybrid beamformers for both sides. As the VFD initialization does not require the alternating optimization to obtain the fulldigital beamformer, its additional complexity is much lower compared with that of the main HBF algorithms. Simulations in Section VII will show that with the VFD method, the convergence speed improves significantly with even some MSE performance improvement, in comparison with random initialization.
Iv Hybrid BeamForming Design for Broadband MmWave Systems
Due to the large available bandwidth of mmWave systems, frequency selective fading will be encountered. Therefore, in this section we generalize the previous hybrid beamformer design to broadband mmWave systems. In particular, we point out that similar to the narrowband scenario, the optimization of precoder and combiner in the broadband scenario can also be unified and solved through the same procedure. Thus, we focus on the hybrid transmit precoder design in the broadband scenario and propose three algorithms.
Iva System Model in the Broadband Scenario
To overcome the channel frequency selectivity, we assume that the orthogonal frequency division multiplexing (OFDM) technology is applied so that the channel fading on each subcarrier can be regarded as being flat. To facilitate the following system design, the broadband mmWave MIMO channel model with halfwavelength spaced ULAs at both the transmitter and the receiver in [20] is adopted here, where the channel matrix at the th subcarrier, for with being the total number of subcarriers, is given by
(22) 
where the other parameters are defined in the same way as that in (1). It is worth noting that although the geometrybased spatial channel model is applied in simulations, all proposed HBF algorithms are compatible for other general models.
As shown in Fig. 2, the processed signal vector at the th subcarrier after the hybrid receive combining can be expressed as
(23) 
where and denote the transmitted symbol vector and the additive noise vector at the th subcarrier, respectively, and denote the digital precoder and combiner at the th subcarrier, respectively, and and denote the analog precoder and combiner, respectively. It is worth noting that in the broadband scenario, the digital beamformers and must be optimized for different subcarriers while the analog precoder or combiner is invariant for all subcarriers due to the postIFFT or preFFT processing. Similar to [18, 20], we assume equal power allocation among subcarriers at the transmitter, namely . To deal with the new difficulty in the HBF design for broadband mmWave MIMO systems, we take the sumMSE of all the subcarriers and all the streams as the objective function. That is, define , where denotes the modified MSE on the th subcarrier and is given by
(24)  
where , , and is a scaling factor for the th subcarrier as similar to that in the narrowband scenario. Then, the optimization problem in the broadband scenario can be formulated as
(25) 
By comparing the problem in (25) with that in (5) for the narrowband scenario, it can be found that they have almost the same form except that the digital beamformers need to be optimized for different subcarriers in (25). Thus, the alternating minimization principle is also applicable here. In particular, it can be shown that in the broadband scenario the two subproblems associated with (25) for the optimization of precoding and combining can also be solved through the same procedure. Therefore, in the following we focus on the hybrid transmit precoder design.
IvB Broadband Hybrid Transmitter Design
Analogous to that in Section IIIA, the original precoder is also separated as , where is an unnormalized precoder for the th subcarrier. It can also be proved by contradiction that the optimal is given by . Then, by fixing the hybrid receive combiner and based on the KKT conditions, the optimal can be derived as a function of . That is,
(26) 
where and . Now the original problem in (25) is reduced to the one for as follows
(27) 
where
(28) 
IvB1 Analog Precoding Based on the MO Method
Although the objective function for analog precoding optimization in the broadband scenario is more complicated than that in the narrowband scenario, the MO method can still be applied here. Using some differentiation rules for complexvalued matrices, the conjugate gradient of the function with respect to can be expressed as
(29) 
where can be derived as follows according to Lemma 1
(30)  
with defined for notational brevity. Thus, the Riemannian gradient can be computed by projecting the above Euclidean gradient onto the tangent space of the Riemannian manifold [18]. According to the property of the gradient descent method, with a proper selection of the step size, is guaranteed to converge to a feasible local optimal solution via MO.
IvB2 Analog Precoding Based on the EVD Method
Note that since the objective function in the broadband scenario is the sumMSE of all the subcarriers, the variant channel matrices and digital beamformers at different subcarriers prevent us from rewriting the original problem in an GEVDavailable formulation as that in Section IIIA2. Nevertheless, the approximation of can still be utilized here for developing other lowcomplexity algorithms. The basic idea is to ignore the constant modulus constraint in (27) temporarily and add a new constraint of . Then, we have the following new problem
(31) 
It turns out that the above optimization problem is still difficult to solve. Therefore, we devote to derive its lower bound first with the help of the following lemma.
Lemma 2.
A lower bound of the objective in (31) is given by
(32) 
Proof: For notational brevity, we define and have
(33)  
where denotes the th eigenvalue of and is positive because is positive definite. The inequalities (a) and (b) in (33) both come from the Jensen’s inequality, with equality of (a) satisfied if and equality of (b) satisfied if , respectively. Substituting the definition of , the proof is completed.
Then, instead of the objective function in problem (31), we devote to minimize its lowerbound, which is equivalent to maximizing . After omitting the constant terms, the optimization problem can be rewritten as
(34) 
It can be proved that the optimal is times the isometric matrix containing the eigenvectors associated with the largest eigenvalues of [34], which can be obtained through EVD. To further make the constant modulus constraint satisfied, similar to that in Section IIIA, we just extract the phase of each element of the optimal . The algorithm is referred to as the EVDLBHBF algorithm, where LB denotes the abbreviation for lower bound. In the following, we propose a better algorithm, where instead of minimizing a lower bound of (31) in EVDLBHBF an upper bound is derived for minimization.
Lemma 3.
For an positive definite and Hermitian matrix and an arbitrary
paraunitary matrix
, i.e., , define the eigenvalues of and in descending order as and , respectively. Then we have .Proof: According to CourantFisher minmax theorem [36],
where is a nonzero vector, denotes a dimension subspace of and
is a new subspace after a linear transform of
to . Similarly, as can be proved to be the th largest eigenvalue of , we have(35) 
Then we have . Since is positive definite, by Jensen’s inequality, holds for any nonzero vector . Thus, the proof is completed.
Then, denoting as , and using Lemma 3, the objective in (31) can be further upper bounded as
(36)  
where (a) follows from the relationship between a matrix’s trace and eigenvalues and (b) follows from Lemma 3. Using the matrix inversion lemma [36], we have , where . The optimization problem in (31) can be converted to the one minimizing its upper bound in (36), or equivalently
(37) 
Similar to that for (34), the solution can be obtained through the EVD and phase extraction operations. To distinguish, we refer to this algorithm as the EVDUBHBF algorithm, where UB denotes the abbreviation for upper bound.
IvB3 Analog Precoding Based on the OMP Method
By combining the OMPMMSEHBF algorithm for narrowband multiuser mmWave MIMO systems [26] and the OMPbased HBF algorithm aiming at maximizing the spectral efficiency for broadband mmWave MIMO systems [29], we come up with the lowcomplexity OMPMMSE precoding algorithm for broadband mmWave systems. Specifically, by restricting the search range of within a set of basis vectors , the hybrid beamforming problem can be rewritten as
(38) 
where denotes the zero norm of a matrix. and is an matrix having nonzero rows which constitute as defined in [9, 26, 29]. With the readily derived closedform solution of the digital precoder in (26), the algorithm developed based on the OMP method can be applied to choose the columns of that are most strongly correlated with the residual error to form the analog precoder.
V Extend to Spectral Efficiency Based on the WMMSE Criterion
Spectral efficiency is another important performance metric for the HBF design [9, 18, 20]. Based on the fulldigital beamforming design approach in [25], the authors in [28] investigated the HBF design with the WMMSE criterion and connected it to the one for sumrate maximization. However, their HBF algorithm is based on the OMP method, which has a limited feasible set for the analog beamformers, and is only for the narrowband scenario. In this section, following the design approach in [25, 28], we first show that in the narrowband scenario our proposed HBF algorithms in Section III can be extended to the ones for achieving better spectral efficiency than the OMP based algorithm. We also extend the sumMSE minimization problem to the WMMSE problem and connect it to the spectral efficiency maximization problem in the broadband scenario. It is shown that the proposed broadband HBF algorithms in Section IV can be generalized to the ones for maximizing the spectral efficiency. Simulation results in Section VII will show that the WMMSE HBF algorithms proposed in this section provide better or comparable spectral efficiency than the conventional ones [18, 19, 20, 28].
First start from the narrowband scenario. Assuming that the transmitted symbols follow a Gaussian distribution, the achievable spectral efficiency is then given by , where and denote the hybrid precoder and combiner, respectively. Inspired by [25, 28], a suboptimal but efficient HBF design for maximizing the spectral efficiency can be connected to the following WMMSE problem
(39) 
where is an weighting matrix to be optimized, and denotes the modified MSE matrix. According to [25, 28], a threestep procedure is applied to solve (39). In the first step, is optimized by fixing and in (39). That is,
(40) 
where . By comparing (40) with (19), it can be shown that the optimal has exactly the same form as that in (20). After substituting the optimal