has been considered as an important part of various military and civilian communication systems, such as electronic warfare, radio surveillance and spectrum awareness. As known, classical signal identification methods used in the past are based on complex collections of feature extraction methods, such as cyclostationarity, high–order cumulants and complex hierarchical decision trees. Furthermore, it should be noted that classical methods cannot be generalized over all signal types and they suffer from dynamic nature of the propagation channel and cannot be adopted easily if a new wireless communication technology emerges. On the other hand,DL has been proposed as a useful method for such classification problems and recently have been applied to this domain intensively. However, these methods should also provide strong performance against the wireless impairments in that particular domain thus, robust AMC methods based on DL techniques should be investigated to achieve dependable, efficient and resilient classification performance under realistic wireless communication channel conditions.
I-a Related Work
Signal identification systems often use likelihood based (LB) and feature based (FB) techniques. Although, LB
methods make the probability of correct classification maximum, they suffer from high computational complexity. Also, they are not robust to model mismatches, such as channel coefficient estimates and timing offsets[13, 3, 17].
On the other hand, in FB approaches, it is required to find a feature which can distinguish the signal from others. However, single feature mostly is not sufficient to classify signals in a large set. In literature, the higher order statistics, wavelet transform, and cyclic characteristics are mainly proposed features for signal identification. For instance, the wavelet transform is utilized in the identification of frequency shift keying (FSK) and phase shift keying (PSK) signals 
. The higher order statistics such as higher order cumulants and moments which are another feature used inAMC [9, 16]. In addition to these features,  utilizes instantaneous amplitude, phase and frequency statistics in order to make modulation classification. Howbeit, it is explicitly known that these features hamper to perform well in real–world conditions such as multipath channel fading, frequency, and timing offsets. Although the most powerful FB approach, cyclostationarity–based features are resistant to mismatches compared to other features , it suffers from high computational complexity.
Machine learning–based approaches have been recently adopted to AMC. For example, CNN, convolutional long short term memory fully connected deep neural network fast Fourier Transform
convolutional long short term memory fully connected deep neural network(CLDNN) and long short term memory (LSTM) can be said as the most popular deep neural network architectures for AMC.  proposes using CNN with in–phase/quadrature (I/Q) data and
fast Fourier Transform(FFT) for AMC and interference identification in industrial, scientific and medical (ISM) band. It is shown that recurrent neural networks can be utilized for AMC under Rayleigh channel with uncertain noise condition . In addition to proposing CLDNN for AMC,  compares it to other existing models under different subsampling rates and different number of samples. Furthermore, it aims to reduce training time for online learning by utilizing subsampling and principal component analysis (PCA). LSTM is proposed in , but it does not allow online learning and has long enough training time to require very high computing capacity. The RadioML2016.10a dataset111It is available on http://opendata.deepsig.io/datasets/2016.10/RML2016.10a.tar.bz2  is widely used in the literature. However, a system that works under real–conditions should be designed to operate under different channel conditions. Due to the dynamic nature of propagation channel and severe multipath effects, the existing available datasets cannot fulfill to provide the desired real–world conditions. RadioML2018.01a introduced in  includes over–the–air recordings of digital and analog modulation types. However, it cannot provide information about the channel parameters since this data set is based on measurement. Therefore, this dataset cannot allow generating information about how the channel conditions affect the performance of the model trained on the dataset. Furthermore, it has not serious diversity because it is created in the laboratory environment where there is no significant change in the channel parameters such as fading and number of taps. In this case, there is a need for a data set that includes both actual channel conditions and controlled channel parameters. It is also necessary to design a DL model that can work under real channel conditions.
The main contributions of this study are two fold and can be summarized as follows:
First, aforementioned discussions show that currently, there is no comprehensive, inclusive, and controlled dataset that integrates the severe multipath effects for the real–world channel conditions. Therefore, we first introduce a new and more challenging modulation dataset, HisarMod2019.1 . This new public dataset provides wireless signals under ideal, static, Rayleigh, Rician (), and Nakagami–m () channel conditions with various numbers of channel taps. Thus, it becomes possible to observe more realistic channel conditions for the proposed DL–based AMC methods.
More importantly, a new CNN model with optimal performance in terms of accuracy and training time under more realistic conditions is proposed. The proposed method exhibits higher performance under both in HisarMod2019.1 dataset and existing RadioML2016.10a dataset when compared to the available classifiers. The new CNN consists of four convolution and two dense layers. In addition to its high performance, the model has lower training complexity when compared to the available techniques, thus, the training process is relatively short.
Ii HisarMod2019.1: A New Dataset
In order to increase the diversity in signal datasets, we create a new dataset called as HisarMod2019.1, which includes classes and different modulation families passing through different wireless communication channel. During the generation of the dataset, MATLAB 2017a is employed for creating random bit sequences, symbols, and wireless fading channels.
The dataset includes modulation types from different modulation families which are analog, FSK, pulse amplitude modulation (PAM), PSK, and quadrature amplitude modulation (QAM). All modulation types are listed in Table I. In the dataset, there are signals, which have the length of I/Q samples, for each modulation type. To make HisarMod2019.1 similar to RadioML2016.10a for fair comparison, there are different signal–to–noise ratio (SNR) levels in between -20dB and 18dB. As a result, the dataset covers totally signals. When generating signals, oversampling rate is chosen as and raised cosine pulse shaping filter is employed with roll–off factor of .
Furthermore, the dataset consists of signals passing through different wireless communication channels which are ideal, static, Rayleigh, Rician (), and Nakagami–m (). These channels are equally likely distributed over the dataset; therefore, there are 300 signals for each modulation type and each SNR level. Ideal channel refers that there is no fading, but additive white Gaussian noise (AWGN). In the static channel, the channel coefficients are randomly determined at the beginning and they remain constant over the propagation time. The signals passing through Rayleigh channel are employed to make the system resistant against non line–of–sight (NLOS) conditions. On the other hand, Rician fading with shape parameter, , of is utilized owing to the fact that the dataset covers a mild fading. In addition to these channel models, the distribution of received power is selected as Nakagami–m with shape parameter, , of for the rest of the signals in the dataset. As a result, the dataset includes signals with different fading models. Noting that the number of multipath channel taps are equally likely selected as and which are adopted from ITU–R M1225 .
|Modulation Family||Modulation Types|
Iii The Proposed Cnn Model
In this paper, a CNN1]. The proposed CNN model involves four convolution and pooling layers terminated by two dense layers. The rectified linear unit (ReLU
) activation function, which is defined as
is employed in each convolution layer. In (1), , , , and are the input and output of the function, weight, and bias, respectively. In this model, it is chosen that the model gets narrower in terms of the number of filters in each convolution layer through the end of the feature extraction part of the model. Our experience with many different configurations indicated that the models that get narrower in each following convolutional layer provides better results in terms of classification and reduce training time. Indeed, for the optimal performance, we employed filters in the first layer while the last layer had filters. The first dense layer is formed by neurons and ReLU activation function. The dense layer is followed by a softmax activation function which computes the probabilities for each class as
where and are any element of classes and the number of classes, respectively. Moreover, the adaptive moment estimation (ADAM) optimizer is used to estimate the model parameters with the learning rate of . The CNN model architecture is depicted in Fig. 1. Furthermore, the layout for the proposed CNN model is given in Table II. During the training process, we use early stopping to terminate the process if the validation loss converges to a level enough. As a result, the model is preserved to be overfitted. As seen in Fig. 1
, there is a layer, which adds noise at each epoch; thus, it also prevents the model to overfit. The power of noise is determined according to the desiredSNR level.
In the training and test stages, we employ four NVIDIA Tesla V100 graphics processing units by operating them in parallel. It is seen that the proposed CNN model is too light compared to CLDNN  and LSTM . For example, the proposed CNN model has million trainable parameters, whereas CLDNN has million trainable parameters for HisarMod2019.1 dataset. Furthermore, CNN model takes one–quarter time of LSTM per epoch.
Iv Classification Results
The proposed model is tested in both the HisarMod2019.1 and the RadioML2016.10a datasets. The test results are provided below.
Iv-a HisarMod2019.1 Dataset Classification Results
As detailed in Section II, the HisarMod2019.1 covers different modulation types. It is not that easy to handle so many signal types in the fading environment. It is expected that they are confused each other due to the deterioration in their amplitude and phase. Thus, in this study, we use an approach like the data binning method by labeling signals with respect to their modulation families such as analog, FSK, PAM, PSK, and QAM. The hierarchical approach is depicted in Fig. 2. Firstly, we aim to classify signals in terms of modulation families. Then, each modulation type can be identified in the family subset. One should keep in mind that this study focuses on the classification of the modulation families not the order of each modulation type for the HisarMod2019.1 dataset. The dataset is split as , , and for training, validation, and test sets, respectively.
As stated before, the early stopping is employed in the training stage. The first layer of the CNN adds noise to data according to the SNR level. As a result, the model becomes more robust to overfitting.
The model gives meaningful results at SNR levels higher than dB. It might be said that the model makes a random choice between modulation families at low SNR values. Considering the nature of wireless communications, the model performs well for the expected SNR values. The dataset is also employed with the CLDNN model. It is noted that we employ the CLDNN and LSTM models as detailed in  and  without any adjustment. Also, the proposed CNN model shows better performance than the existing CLDNN and LSTM models in HisarMod2019.1 dataset. For example, it exceeds accuracy at 8dB SNR; however, CLDNN performs with the same accuracy at 16dB SNR. While CLDNN does not achieve accuracy, our model exceeds this level at 14dB and higher. The maximum accuracy values for the proposed CNN model and state of the art CLDNN model are and , respectively. Surprisingly, LSTM cannot show acceptable classification results; however, it performs well in RadioML2016.10a. At SNR values, the results are not meaningful in terms the classification accuracy since the false alarm rate gets higher. Fig. 3 denotes the accuracy values for CNN, CLDNN, and LSTM models at the SNR values in between [-20dB, 18dB]. Fig. 4 and Fig. 5 show the confusion matrices for the proposed CNN model and CLDNN model, respectively. Both of them have difficulty in the identification of QAM signals. On the other hand, LSTM recognizes signals as analog modulated signals regardless of the received signal type. Hence, the confusion matrices are not provided for the LSTM model.
Iv-B RadioML2016.10a Dataset Classification Results
RadioML dataset is heavily used in modulation classification studies and it is a well accepted dataset by the literature. Therefore, in order to show the robustness of our proposed CNN model, we also test our model in RadioML dataset to observe its performance. In this section, RadioML2016.10a dataset is employed. It consists of synthetic signals with modulation types. The modulation types covered by the dataset are listed as: AM–DSB, WBFM, GFSK, CPFSK, 4–PAM, BPSK, QPSK, 8–PSK, 16–QAM, and 64–QAM. Details for the generation and packaging of the dataset can be found in .
Here, the dataset is split into two parts (i.e. training and test) with equal number of signals. After training procedure, the models are tested with the rest of the signals. According to test results, the proposed CNN model shows higher performance than the CLDNN model at the SNR levels higher than -2dB. LSTM performs slightly better than CNN. The CLDNN is able to reach the maximum accuracy of . On the other hand, the proposed CNN model performs with the maximum accuracy of even though it is not originally designed for the RadioML2016.10a dataset. Although LSTM reaches up to accuracy, its computational complexity is extremely high. Fig. 3 denotes the accuracy values with respect to SNR levels. The confusion matrices for the classification results of the proposed CNN model are depicted in Fig. 6. It is observed that the model recognizes almost all signals as 8–PSK at low SNR levels. Fig. 6
shows the confusion matrix of the minimumSNR value of which the model performs over accuracy. As can be seen from Fig. 6, the model gives poor results in modulation types other than 4–PAM. The proposed model achieves very high performance in all modulation types, except WBFM at 6dB and above.
Initial observations suggest that the proposed model can work with high performance both in a diverse dataset, HisarMod2019.1, and RadioML2016.10a which is a frequently used dataset.
V Concluding Remarks
In this study, we present a diverse new dataset, which consists of multipath fading signals with different number of channel taps, and a CNN model for AMC. The first stage of hierarchical classification architecture, which is the classification of modulation families, is realized with the proposed CNN model on this dataset and the results compared with the CLDNN model proposed in the literature. The results show that the proposed CNN model performs significantly better than CLDNN. Furthermore, the performance of the proposed CNN model on the RadioML2016.10a dataset is examined. It is demonstrated that the proposed CNN model is both faster and more accurate than the CLDNN model. As a future work, we will investigate the classification of modulation orders assuming that the modulation family is identified. Finally, extensive search conducted for optimal model in this study shows that starting with an extensive set of filters, and then reducing their numbers down step by step provides better results in terms of accuracy. This phenomenon will be investigated thoroughly and technical discussions will be provided in terms of explainable AI terminology.
-  (2015) Keras. Note: https://keras.io Cited by: §III.
-  (2015) Signal identification for emerging intelligent radios: classical problems and new challenges. IEEE Instrum. Meas. Mag. 18 (2), pp. 11–18. Cited by: §I-A.
-  (2009) On the likelihood-based approach to modulation classification. IEEE Trans. Wireless Commun. 8 (12), pp. 5884–5892. Cited by: §I-A.
-  Cited by: 1st item.
-  (1999) Identification of digital modulation types using the wavelet transform. In IEEE Mil. Commun. Conf. (MILCOM), Vol. 1, pp. 427–431. Cited by: §I-A.
-  (2018) Robust modulation classification under uncertain noise condition using recurrent neural network. In IEEE Glob. Commun. Conf. (GLOBECOM), pp. 1–7. Cited by: §I-A.
-  (1997) Guidelines for evaluation of radio transmission technologies for IMT-2000. External Links: Cited by: §II.
-  (2018) End-to-end learning from spectrum data: a deep learning approach for wireless signal identification in spectrum monitoring applications. IEEE Access 6, pp. 18484–18501. Cited by: §I-A.
-  (2006) A novel modulation classification method based on high order cumulants. In Intl. Conf. on Wireless Commun., Net. and Mobile Computing (WiCOM), pp. 1–5. Cited by: §I-A.
-  (1995) Automatic analogue modulation recognition. Signal Processing 46 (2), pp. 211–222. Cited by: §I-A.
-  (2016) Radio machine learning dataset generation with GNU radio. In Proceedings of the GNU Radio Conference, Vol. 1. Cited by: §I-A, §IV-B.
-  (2018) Over-the-air deep learning based radio signal classification. IEEE J. Sel. Topics Signal Process. 12 (1), pp. 168–179. Cited by: §I-A.
-  (2000) Likelihood ratio tests for modulation classification. In IEEE Mil. Commun. Conf. (MILCOM), Vol. 2, pp. 670–674. Cited by: §I-A.
-  (2018) Deep learning models for wireless signal classification with distributed low-cost spectrum sensors. IEEE Trans. on Cogn. Commun. Netw. 4 (3), pp. 433–445. Cited by: §I-A, §III, §IV-A.
-  (2019) Fast deep learning for automatic modulation classification. arXiv preprint arXiv:1901.05850. Cited by: §I-A, §III, §IV-A.
-  (2000) Hierarchical digital modulation classification using cumulants. IEEE Trans. Commun. 48 (3), pp. 416–429. Cited by: §I-A.
-  (2010) Software-defined radio equipped with rapid modulation recognition. IEEE Trans. Veh. Technol. 59 (4), pp. 1659–1667. Cited by: §I-A.