Non-orthogonal multiple access (NOMA) has been recently considered as a promising solution for the next generation of wireless communication networks (i.e., 5G) to improve the spectral efficiency and user fairness [1, 2, 3]. The principle of NOMA relies on utilizing the power domain to serve multiple users at the same time/frequency/code . Specifically, in a NOMA network with more than two users, a base station (BS) transmits the messages superposed with different power levels to users [4, 5, 6]. The power allocation is carried out based on the users’ channel conditions. In principle, the successive interference cancellation (SIC) scheme is employed. First, a user detects the messages of other users with weaker channel conditions (i.e., weak users) and then removes these components from its observation. Second, it decodes its own messages by treating the messages of the other (i.e., the users with stronger channel conditions, so-called strong users) as noises. Therefore, in comparison with conventional multiple access schemes, the NOMA can give a better user fairness because the users with weak channel conditions can be appropriately served.
With the development of wireless networks, information security is a critical challenge due to the broadcast nature of wireless transmissions [7, 8]. Recently, physical layer security (PLS) has been emerging as a prominent candidate for improving the secrecy performance of wireless systems [9, 10, 11, 12]. The key idea of the PLS relies on exploiting the characteristics of wireless channels to guarantee secure communications . This approach is different from traditional security solutions, such as cryptographic protocols in the upper layers . In the PLS, it is well-known that the perfect secrecy is achieved when the quality of the legitimate channel is higher than that of illegitimate channel [7, 8, 9, 10, 11, 12]. To obtain further significant improvements, many PLS-based transmission methods have been proposed, based on the applications of multi-input multi-output (MIMO) [14, 15], artificial noises , jamming , or beamforming techniques .
Nowadays, the approach of the PLS in NOMA systems is becoming an interesting topic [19, 20, 21, 22, 23]. In , the secrecy sum rate of a single-input single-output (SISO) NOMA system has been studied and the closed-form optimal solution of power allocation has been derived, based on the users’ quality of service (QoS) requirements. Taking the PLS of large-scale NOMA networks into account, the work  has employed stochastic geometry methods to locate users and eavesdroppers in the system. Furthermore, a protected zone around the source has been introduced to enhance the information security. To characterize the secrecy performance of the proposed system, new exact and asymptotic expressions of the secrecy outage probability have been derived. In , the application of multiple antennas, artificial noises (AN) and a protected zone to NOMA in large-scale networks have been studied for the purpose of improving the secrecy performance. The obtained results regarding the secrecy outage probability indicated that a significant secrecy performance gain could be achieved by generating the AN at the BS and invoking the protected zone. In addition, the PLS in a two-user multi-input single-output (MISO) NOMA system was examined in  where NOMA is performed based on the users’ QoS requirements, instead of their channel conditions. In this context, the authors considered a scenario that the eavesdropper detects two-user data using SIC. Specifically, it first treats the message of the user with a low QoS as noise to decode the message of the user with high QoS, and then subtracts this component from its observation before decoding the message of the remaining user. Moreover, the use of a transmit antenna selection (TAS) scheme for improving secrecy performance was considered. In , a new secrecy beamforming (SBF) scheme for MISO NOMA systems was studied. Specifically, the proposed SBF scheme efficiently exploited AN to protect the confidential information of two NOMA assisted legitimate users (LUs). It is clear that the TAS technique has been demonstrated as a low-complexity and power-efficient solution for secure NOMA networks [22, 24].
Nevertheless, the previous works on PLS in NOMA systems [19, 20, 21, 22, 23] have not covered the following issues. First, they have considered the scenario of SISO or MISO systems with Rayleigh fading channels. Thus, the applications of MIMO which can significantly improve the secrecy performance and the consideration of Nakagami- fading channel, which possibly bringing more insight analysis, have not been addressed yet. Second, the works [19, 20, 21] have been carried out under the ideal assumptions as follows: (i) the strong user always successfully decodes the message of the weak user, and (ii) the eavesdroppers have the powerful detection abilities to distinguish a multi-user data stream without the interference, generated by superposed transmit signals (i.e., worst-case eavesdropping scenario (WcES)). Unlike these studies, the authors in  have investigated the scenario that the eavesdropper is affected by the interference generated by superposed signals when carrying out a multi-user data detection as discussed above. However, they have assumed that the message of the user with high QoS is already decoded successfully at the eavesdropper when analyzing the secrecy performance of the user with low QoS. Meanwhile, the work  has considered both transmission outage111In a two-user NOMA system, the transmission outage occurs at the strong user when it does not successfully decode the message of the weak user or its own message. and secrecy outage for analyzing the secrecy performance. However, the WcES was still examined. In fact, although the WcES is an effective approach to characterize the secrecy performance, this can overestimate the eavesdropper’s multi-user decodability.
Motivated by the aforementioned issues, in this paper, we proposed a new TAS-based secrecy communication protocol for two-user MIMO NOMA networks over Nakagami-m fading channels, under the following scenario. In our system model, a source transmits information, following the principle of NOMA, to two LUs222In order to focus on designing a new secrecy communication protocol, a two-user NOMA network is studied in this paper. In particular, our obtained results can be easily used for further calculations in downlink NOMA systems with multiple users (more than two users) by applying the hybrid multiple access techniques (i.e., the combination between NOMA and conventional OMA schemes) as studied in [25, 26]., so-called near and far users. Also, a passive eavesdropper wants to extract the information from the source to LUs channels. On the transmitter side, the source broadcasts the mixed messages to the two LUs by using TAS. On the receiver side, it is assumed that maximal ratio combining (MRC) is employed at the two LUs and the eavesdropper to maximize the signal quality. Particularly, regarding the eavesdropper’s multi-user decodability, we considered the cases that the eavesdropper can successfully and unsuccessfully decode the far user’s message to analyze the secrecy performance of the near user. On this basis, two solutions of TAS to design a secure communication protocol, namely Solutions I and II, were proposed to maximize the received signal power at the near and far users, respectively. To analyze the secrecy performance of the two solutions, we derived the exact closed-form expressions of the secrecy outage probability (SOP) for the two LUs and the overall system. Also, we provided the asymptotic expressions of the SOP and investigate the secrecy diversity. Moreover, we then compared the secrecy performance in various scenarios adopting the two solutions, such as SISO versus MIMO, and compared our solutions with the previous works [21, 22, 23] to evaluate the benefits of our proposed protocol. Accordingly, to validate the analytical results, Monte Carlo simulation was employed.
Therefore, the contributions of this paper are summarized as follows:
Analyzing the secrecy performance of the NOMA network in which the impact of MIMO, Nakagami- fading, and the eavesdropper’s multi-user decodability are addressed.
Proposing two new TAS solutions to improve the security for the considered NOMA network.
Deriving the closed-form expressions of the SOP at the LU sides and also for the overall network to evaluate the secrecy performance of our solutions.
Providing the asymptotic expressions of the SOPs and investigating the secrecy diversity order.
The remainder of the paper is organized as follows: The system model is presented in Section II. The transmit antenna selection scheme is described in Section III. The secrecy performance analysis of the considered system is shown in Section IV. The numerical results and discussions are depicted in Section V. Finally, Section VI shows our conclusion.
Ii System Model
Consider a downlink MIMO NOMA network consisting of a source (the base station) denoted by , a near user denoted by , a far (cell edge) user denoted by , and a passive eavesdropper denoted by , as depicted in Fig. 1. In this system, the source , the near user , the far user , and the eavesdropper are equipped with , , , and antennas, respectively.
Let (, , ) denotes the channel fading coefficient from antenna at to antenna at . Similarly, () denotes the channel fading coefficient from antenna at source to antenna at . In our work, legitimate and eavesdropping channels are modeled as mutually independent and identically distributed (i.i.d) Nakagami- fading ones with parameters and , and squared means and , respectively. In addition, the distance and path loss exponent of and channels are denoted by and , and and , respectively.
Employing the NOMA technique, simultaneously communicates with two LUs and . Further, selects an antenna among antennas to broadcast information to and by applying a TAS technique. The antenna selection schemes will be clarified in the next section. At the receiver side, maximal ratio combining (MRC) is used at , , and .
Suppose that antenna at is selected for transmission, the channel gain of link can be expressed as
denotes the channel coefficient vector oflink.
Given the above discussion, the overall communication process of the system can be mathematically depicted as follows. Following the principle of NOMA, transmits the superposed message to and , where and denote the intended messages for and , respectively. Also and denote power allocation coefficient for and , respectively. Thus, the received signal at () is given by
where represents the signal processing operation at with MRC, stands for an additive white Gaussian noise (AWGN) at user . According to the principle of NOMA, we assume that , and . Therefore, the instantaneous signal-to-interference-and-noise ratio (SINR) at to detect is
where denotes the average transmit signal-to-noise ratio (SNR) associated with the LUs.
At , a SIC receiver is used to decode which is then removed from the observation before detecting . Thus, the instantaneous SINR at to detect can be given by
and the instantaneous SNR at to detect is written as
At , the signal received from can be expressed as
where represents the signal processing operation at with MRC, stands for the AWGN at . We assume that SIC receiver is applied at to decode , then subtract this element from the received signal to detect . Thus, the instantaneous SINR at to detect can be expressed as
and the instantaneous SNR at to detect is written as
where is the average transmit SNR associated with .
Iii Proposed TAS Protocol and Preliminaries
In this section, we propose two solutions, namely Solutions I and II, for designing a TAS-based secure protocol for the MIMO NOMA network.
Iii-a Two Proposed TAS Solutions
For convenience, let , , and .
Iii-A1 Solution I and Formulations of Legitimate Channels
In solution I, an antenna at is selected for transmission with the aim of finding the best channel condition of link.
Given this context, the selected antenna, denoted by , can be mathematically represented as follows:
With this setting, the CDF of has the following form 
where and .
where , , and .
For , the CDF of is given by 
where and .
Iii-A2 Solution II and Formulations of Legitimate Channels
Different from Solution I, in Solution II, an antenna at is chosen to broadcast information with the purpose of providing the best channel gain of link. Accordingly, the chosen antenna at in this case can be expressed as
Thus, the CDF of can be given as
and the CDF of is
where , , and .
Iii-A3 Formulations of Eavesdropping Channels
For , the PDF and the CDF of in both Solutions I and II are given by 
where and .
Iii-B Preliminary Analysis of SINRs with Solutions I and II
Under Nakagami- fading, the CDF of in Solution I has the following form
where and .
Under Nakagami- fading, the CDF of in Solution II is expressed as
Under Nakagami- fading, the PDF of is given by
Iv Secrecy Performance Analysis
In this section, to validate the two proposed solutions, the secrecy performance regarding the SOP obtained at and , and the SOP of the overall system is analyzed.
This subsection presents the definitions of secrecy capacity and SOP for secure communication in the considered MIMO NOMA system.
First, let and denote the capacities of legitimate channel and illegitimate channel to detect the signal , respectively. Thus, according to , the secrecy capacity achieved at can be defined as
where and .
Second, the SOP obtained at is defined as
where denotes the SNR threshold for correctly decoding , represents the target data rate of , and represents the secrecy rate threshold at .
To analyze the SOP at user , its formulation needs to be further expressed for more insights. Thus, according to (27), can be expressed as in Lemma 4.
The SOP at user can be given as
where , , and are, respectively, given by
where and .
See Appendix A. ∎
Third, the SOP at has the following form
where and represents the secrecy rate threshold at .
Fourth, in the system, we define the overall SOP as the probability that the secrecy outage event occurs at either or , i.e.,
Iv-B Secrecy Performance Analysis of Solution I
Iv-B1 Exact Secrecy Outage Probability Analysis
In this case, the SOP at and are derived through Theorem 1 and Theorem 2 as follows.
Under Nakagami- fading, the SOP of user in Solution I is approximately expressed as
where , , , , , and is a complexity-accuracy trade-off parameter.
See Appendix B. ∎
See Appendix C. ∎
From (30), the overall SOP of the system in Solution I is expressed as
Iv-B2 Asymptotic Secrecy Outage Probability Analysis
Using the series representation of given by [27, Eq. 1.211]
the asymptotic CDF of and when are written as
where , , and .
The secrecy diversity order at in Solution I, , is defined as