With the trending of cyber physical system (CPS), computing and connectivity have been present in every aspect of human lives . Interconnected devices in a network cooperate in the distribution of computing in wireless network, facilitating the decentralized applications, such as seismic monitoring and industry control . However, due to the restricted computing ability and energy consummation limitation, complicated cryptography could not be implemented on devices of low hardware configuration in CPS. Besides, the bugs of software, operating systems and firmwares that running on them lead them to be vulnerable . Malwares like Stunex  and Duqu  have caused a lot of concerns. Thus, how to ascertain the devices of a CPS are trustworthy is becoming an important and open question.
To mitigate increasing attacks targeting CPS devices, a lot of countermeasures has been proposed. Remote attestation is an efficient to verify the security of the individual CPS devices and has been a hot research topic. The main idea of remote attestation is the system management sends attest requests to the remote devices and make decisions according to the received security reports from the devices. Generally speaking, remote attestation could be sorted into three categories: software-based, hardware-based and hybrid. Software-based methods  calculate the checksum of memory and essentially register and expect a time-limited response for trustworthy devices without the help of additional hardware. Hardware-based methods depend on secure hardware such as TPM  and TrustZone  to provide secure execution environment. The hybrid methods require the minimal collection of hardware and software components that result in secure remote attestation [9, 10, 11, 12]. However, most existing methods are not scalable for a large number of devices and could only measure static integrity.
Resource usage analysis is another approach to verify the trustworthiness of CPS devices. The usage of resources such as CPU, memory and network of the devices is used in resources usage methods. The inside states of devices could thus be inferred, which is useful for system management. [13, 14] analyzed the user behaviors by tracking the energy consumptions. However, they are only targeting individual device. [15, 16] proposed to monitor HPC systems by collecting detailed resource usage information to detect anomalous behaviors. But they didn’t consider the co-occurrence of resources usage among the HPCs. Besides, collecting and transmitting detailed information might not be practical for devices of low hardware configuration which have limited computing and bandwidth resources. In a CPS system, interconnected devices work together in running decentralized applications. The devices will collectively decide a holistic view of the entire system Unlike desktop computers or servers, a CPS only runs very limited number of applications simultaneously. Thus, the resource usage patterns of a CPS could be characterized by analysis the resource usage patterns of decentralized applications that would run on the CPS devices.
. Machine learning tasks are typically classified into two broad categories: supervised learning and unsupervised learning. Supervised learning is presented with example inputs and their desired outputs and the goal is to learn a general rule that maps inputs to outputs. However, unsupervised learning is given no labels for the inputs and the goal is to discover the hidden patterns of the inputs. used regression, a supervised learning method, to predict the energy consumption of cellphones. 
used binary decision tree, also a supervised learning method, to classify the the behaviors of devices.
In this paper, we choose the network traffic pattern as a study case to demonstrate the effectiveness of the proposed method, while the methodology may similarly apply to classification and anomaly detection based on other resource metrics. We first apply texture feature analysis to extract the spatial information of the network resource usage of the CPS devices a time point. Then we study the temporal information of the extracted texture features in time series. The spatial and temporal features together of the network resource usage determine the communication patterns of the CPS network. At last, a supervised learning method, -NN classification, is used to classify and recognize the communication patterns. The phenotype method is experimented on four real-world decentralized applications. With proper length of sampled continuous network resource usage, the overall classification accuracy is about 99%. Additionally, the classification error is used to detect the anomaly network resource usage. We simulate the anomaly network resource usage that equals to 10%, 20% and 30% of the normal network resource usage. The experiment results show the proposed anomaly detection method is efficient in detecting each intensity of anomaly network resource usage.
The contributions of this work are summarized below:
To our best knowledge, this work is the first to phenotype a whole CPS network instead of individual devices.
This work presents a novel method to characterize the network resource usage of four real-work decentralized applications based on image processing and machine learning.
This work also propose a straightforward and efficient method to detect abnormal network resource usage of CPS network.
The rest of this paper is structured as follows. Section II shows the preliminaries of the proposed approaches. Section III proposes the method to phenotype the CPS network base on network resource usage analysis and the method to detect anomaly network traffic. Section IV shows and discusses the experiment results. Section V concludes this paper.
Ii-a System Model
Many CPS use a cloud to visualize and analyze data collected from the CPS devices, as shown in Fig. 1. System management is provided with information about the states of all the CPS devices for security operation, such as shutting down or restarting the CPS network because of the detected abnormal behaviors.
In this model, the system management is provided with real-time network resource usage information through the cloud. From the perspective of sensors, the network resource usage application or process locally running on every CPS device works as a sensor, which monitoring the network resource usage of it. The set of sensors collectively decide a holistic view of the whole CPS network. The CPS devices might be equipped with heterogeneous hardware and software configuration and those devices are interconnected. However, generally, those devices are resource-constrained and only very limited number of applications run on the CPS at the same time. The applications running on the CPS are decentralized and the devices are scheduled to collect data from their built-in sensors or transfer/receive data for their neighboring devices. According to the different applications running on the CPS, the CPS devices present application-specific communication feature in when to transfer/receive data and the size of data that are transferred/received. Hence, we propose to phenotype the CPS network by analyze the communication feature of the applications that would run on the CPS.
As as Fig. 2, there are 100 devices in an example CPS network. Those devices are interconnected with each other directly or indirectly wirelessly. The discussion of the rest of the paper is based on this example CPS network. Note that the proposed phenotype method and anomaly detection method is not depend on the topology of the CPS network.
Ii-B Threat Model
Denial of service (DoS) attack and botnet attack are two common attacks against CPS network . A DoS attack could even drain the limited battery life and computational capabilities of the CPS devices, which could cause them to be slow to response to other devices. A heavy DoS attack could lead the devices to be completely irresponsible, which is trivial to detect. A botnet attack is to control the CPS devices to carry out distributed Dos attacks to remote target servers. An aggressive botnet attack could consume a lot of network resource and cause the applications running on the CPS network to progress very slowly or halt, which is also conspicuous to the system management. Hence, in this paper, we consider a more sophisticated attack that brings in additional network traffic among the CPS devices but would not affect the progress of the applications running on the CPS network. The attack could be either a light DoS attack or light botnet attack. Both the attacks introduce anomaly network traffic to the CPS network.
For the ease of discussion, let’s introduce the definitions for this paper.
Network throughput, the real-time sum of speeds of transferring and sending data of a CPS device;
Communication matrix, a matrix comprised of the network throughput of all the CPS device at a time;
Communication picture, a picture created by visualizing a communication picture;
Texture feature snippet, a slice of the time-series texture features;
Coefficient matrix, each element of which is the Pearson Correlation Coefficient between two time-series texture feature of a texture feature snippet.
Communication pattern, partial elements of a coefficient matrix.
Iii Phenotyping the CPS network and detecting anomaly network traffic
Fig. 2 shows that the 100 interconnected devices in the example CPS network. The communication between every device and other devices is scheduled by the decentralized applications running on the CPS network. Each application has its own unique schedule. Based on this observation, we propose to phenotype the CPS network by analyzing the network traffic of the decentralized applications running on it.
Iii-a Spatial analysis of the communication image
By reading the network throughput of every device in Fig. 2 and organize them as the devices are ordered in the figure, we get an communication image by visualizing the network throughput of every device as a pixel. As shown in Fig. 3, an example communication image of CPS network in Fig. 2. There are 100 pixels in the figure and stands for the th row of the picture and stands for the th column of the picture. , the value of the th row and th column pixel, is the network throughput of the th device. Note that if the topology of the CPS network is squared as that in Fig. 2
, we could still get a rectangular communication picture by padding with some zero-value pixels, which would not affect the following proposed methods.
Each communication image represents the network resource usage of the whole CPS network at a time sample. According to the different applications running on it at that time, the corresponding communication images are different. In this paper, we utilize texture feature analysis [19, 20, 21] on the communication images. Texture analysis is an important element of human vision and has been apply to problems of texture classification, discrimination and segmentation . According to , there are many different kinds of texture features.
In this paper, we propose to transform the communication image into multiple single values to reduce computing complexity. Hence we choose the statistics based feature extraction methods. Statistics based feature extraction methods are classified into three classes: first-order statistics, second-order statistics and higher-order statistics. First-order statistics only consider the individual pixels. Second-order statistics depend on the co-occurrence of neighboring pixels. Higher-order statistics are to deal with nonlinearities and non-Gaussianity. In our case, the interconnected devices of the CPS network work collaboratively so that the network throughput of neighboring devices are related to each other. Hence, this paper chooses GLCM (gray level co-occurrence matrix) texture features, the most popular second-order statistical texture features. We select five GLCM texture features: energy, entropy, contrast, IDM (Inverse Difference Moment) and DM (Directional Moment). To get the GLCM texture features, we need to first convert the communication images into GLCM.
Iii-A1 Convert the communication images into GLCM
Shown as in Table I are the corresponding intensity (gray-level) values of every pixel of Fig. 2. Note that we normalize the network throughput of the devices to be within the range of to for the ease of demonstrating how to convert the communication image into GLCM. Let’s denote the length of the range as and in this example equals to .
Table II is the resultant GLCM. and stand for the th row and th column of the GLCM matrix, respectively. Each element in the GLCM is the sum of the number of times that the pixel with value occurred in the specified spatial relationship to a pixel with value in the communication image . There are four different kinds of relationship between two adjacent pixels: horizontal, vertical, positive diagonal and anti-diagonal, as shown in Fig. 3(a), 3(b), 3(c) and 3(d), respectively. Take the horizontal relationship for example: as shown in Table I, all the horizontal adjacent pixel pairs that is (1,2) are circled. Hence it is easy to see , circled in Table Table II, equals to 6.
As we can see, a GLCM is a by matrix. The larger is, more information of the communication picture is preserved in the GLCM. However, larger brings in more computing complexity.
Iii-A2 Calculate GLCM texture features
As discussed in previous section, we can get four GLCMs from a communication picture from the perspectives of four different adjacent relationship between two pixels in the communication picture. For each GLCM, we apply below five equations .
IDM (Inverse Difference Moment):
DM (Directional Moment):
Energy, entropy, contrast, IDM and DM measure the intensity of pixel pair repetitions, the randomness of pixels, the extent of a pixel and its neighbors, local homogeneity of the communication image and alignment of it in terms of the angle, respectively.
Let’s denote the number of GLCM texture features we get from one communication picture. For each GLCM from the perspective of one adjacent relationship between two pixel, we get GLCM texture features. Thus, we get GLCM texture features in total, i.e. .
Iii-B Temporal analysis of GLCM texture features
We have previously mentioned that each communication image stands for the network resource usage of the CPS network at a time sample. The GLCM texture features of a single communication are not enough to represent the application running on the CPS devices. We consider the communication feature as the transition of the texture features, i.e., the communicate feature is showed by both the texture feature values and the time-series trends of them.
As shown in Fig. 5, we present the time-series trends of the texture feature values of an example application for only one adjacent relationship between two pixels 111In order to show all the five curves in one picture, we normalize the Energy and Entropy by timing them with .. The horizontal axis is time and the vertical axis is texture feature value. We record the example application every second 222The communication images are recorded every second throughout this paper. for 300 seconds, starting about 50 seconds before the application starts and around 50 seconds after it ends. The example application lasts around 200 seconds. The value of the points the curves go through and the time-series trends of the curves representing the communication features of the example application.
Iii-B1 Texture feature snippets
In order to extract more communication features of one decentralized application ruining on the interconnected CPS network, we slice the time-series texture feature trends into texture feature snippets by using a sliding window, as shown in Fig. 6.
Assume the running time of an application is seconds and the communication images are recorded every second. Assume the size of the sliding window is , then every texture feature snippet consists of texture feature values. The sliding window move once a second, hence we have snippets for this application.
Iii-B2 Coefficient matrix
We observe that each time the CPS network runs a certain application, all the time-series texture feature trends are stochastic. However, the similarities between each two time-series texture feature trends are stable. Hence we propose to use the similarities between each two time-series texture feature trends as the communication patterns. Pearson Correlation Coefficient 
is common method to calculate the similarity of two vectors. As shown in Equation6, and are two time-series texture features, is the Pearson Correlation Coefficient between and .
Let’s denote the th time-series texture feature of the texture feature snippet as . By computing the Pearson Correlation Coefficient between every two time-series texture feature trends, we could get a times coefficient matrix as below:
The coefficient matrix is a symmetric matrix and all of the elements in the diagonal line, (), equal 1. Thus we could use partial elements ( and ) to represent the coefficient matrix. The size-reduced coefficient matrix is comprised of only elements. We represent the size-reduced coefficient matrix as the communication pattern. For each application, we have communication patterns. Note that computing the communication patterns from the communication images does not depend on the number of devices in the CPS network. Hence the proposed phenotyping method is scalable.
Iii-C Classification of the communication patterns
Iii-C1 Communication pattern labeling
We denote as the number of different applications that would run on the CPS network. For application , we transform the network throughput into communication patterns, ( ), and label them as . In this paper, we use -NN (-nearest neighbors algorithm) to classify the communication patterns. -NN is a very popular supervised machine-learning classification method 
. In order to further reduce the computing complexity, we perform principal components analysis on the attributes before training the-NN model.
Iii-C2 Recognition of Communication Patterns
Algorithm 1 shows how to recognize the communication patterns. Firstly, the algorithm is given a communication pattern under recognition, . Then it searches the labeled communication patterns to find communication patterns with shortest Euclidean distance with and store them in set . The next step is find the label that labels most communication patterns in . Label is the result of recognition.
Iii-D Anomaly detection
Training the -NN model must be done under safe environment, i.e., all the CPS devices are trustworthy. The anomaly detection period could be done under unsafe environment. Recollect the threat model in II-B that the attack would not affect the progress of the applications running on the CPS network. Since the tasks of the CPS network are scheduled by the system management, the system management know what application is running on the CPS network at any time. Once the network resource usage is compromised, the recognized application label would be different from expected label by the system management. Assume the recognition accuracy of the trained -NN model for application is , we set the accuracy threshold of it as . For application in period of anomaly detection, if the recognition accuracy is less than , we consider the network resource usage of the CPS network as abnormal and otherwise, we consider it as normal.
Iv Experiment Results
In this section, we study four real-world applications, aggregation, broadcast, consensus and collection. In CPS network, these four decentralized applications are very popular and they have different communication styles. In aggregation, each leaf device sends data to its father devices. The father devices process the received data and send same-size data to their father. As to broadcast, starting with root device, data are sent from father devices to their children devices. For consensus, each device sends data to its neighbors and calculate received data and send data and calculate received data again and again till convergence is reached. The Distributed Gradient Descent (DGD) is very similar to consensus. Each device do certain number of iterations of sending data to its neighbor and calculating received data. The difference is for DGD, each device is synchronized with its neighboring devices, but for consensus, each device is asynchronous with its neighboring devices.
The experiments are based on CORE Simulator. CORE Simulator is a network emulation tool that uses virtualization provided by FreeBSD jails, Linux OpenVZ containers, and Linux namespaces containers . We set up 100 devices on it, as shown in Fig 2. We run every application on CORE Simulator for 200 times and the network throughput are recorded every second. We also run the CORE Simulator with executing none application as base reference for 200 times and record the network throughput every second as well. The normalization range is empirically set as 9. In order to shorten the training time of -NN model, we use principal component analysis to reduce the dimensions of the feature data. We choose to preserve 95% of the information of the communication patterns, reducing the original 210 dimensions to only 11 dimensions. The running time of these applications and the average network throughput of every CPS device throughput the running time of every application are shown as in Table III.
|Application||T (s)||Average network throughput (kbps)|
In our experiment, we randomly choose 80% of the communication patterns of every application as training data and label them respectively. The rest 20% of the communication patterns of every application are used to test the accuracy of the proposed recognition method.
Iv-a Accuracy of proposed method
This section presents the accuracy of the proposed method charactering the applications running on the CPS network. As shown in Fig. 7, the horizontal axis strands for the size of the sliding window and the vertical axis stands for recognition accuracy. As we can see, the proposed method has achieved high accuracy. When is 50s, the recognition accuracy of the broadcast is 68%, which is lowest among all the applications. With the increasing, the recognition accuracy of all the applications become higher. When increase to 190s 333Note that we don’t evaluate the sliding window being 200s because the running time of aggregation and broadcast are 195s, less than 200s., the recognition accuracy of all the applications are higher than 95%. However, also controls the granularity of the communication patterns. Smaller communication patterns could more details of the feature of the network resource usage. Hence, we have to compromise between the recognition accuracy and the granularity of communication patterns.
Iv-B Anomaly detection
In this section, we show our proposed method is capable of detecting abnormal network traffic. As mentioned in section II-B, both of DoS attack and botnet attack could bring in extract network traffic to almost every CPS device. Thus we simulate the anomaly network traffic by adding different intensity of random network throughput to every device when the CPS devices are running. We choose anomaly network throughput that equal to 10%, 20% and 30% of the normal average throughput for every application. For each intensity of anomaly network throughput, We run every application for 40 times. We set as 100s since the trained -NN model archives good balance between the recognition accuracy and the granularity of communication patterns when the is 100s. We set the threshold parameter as 95%. The results are shown as in Fig. IV.
|With 10% anomaly||0.78||0.75||0.53||0.35|
|With 20% anomaly||0.60||0.50||0.40||0.30|
|With 30% anomaly||0.46||0.25||0.32||0.27|
As we can see, for each intensity of anomaly network traffic, the recognition accuracy of aggregation, broadcast, Consensus and DGD is evidently lower than the corresponding thresholds, i.e., the proposed detection method succeeds in detecting the anomaly network traffic that caused by DoS attack or botnet attack.
This paper proposes to develop a network phenotyping mechanism based on network resource usage patterns and identify compromised network traffic. The proposed phenotyping method is based on transforming the network resource usage into time-series texture feature values, thus the computing complexity is reduced. We also apply both spatial and temporal analysis of the network resource usage. The phenotyping method achieves every high accuracy of recognizing the communication patterns of four real-world decentralized applications. Additionally, the paper proposes a detection method to verify whether the network resource usage of the CPS network is normal. The detection method is straightforward and efficient. In the future, we plan to combine the network resource usage with other resource usage to better phenotype the CPS network. Besides, we only consider one application is running on the CPS network at a time, which is not always the case. Hence, we also plan to characterize the CPS network that multiple applications run on it at the same time.
-  “Gartner Says 8.4 Billion Connected ”Things” Will Be in Use in 2017, Up 31 Percent From 2016,” 2017. [Online]. Available: http://www.gartner.com/newsroom/id/3598917
-  D. Miorandi, S. Sicari, F. De Pellegrini, and I. Chlamtac, “Internet of things: Vision, applications and research challenges,” Ad Hoc Networks, vol. 10, no. 7, pp. 1497–1516, 2012. [Online]. Available: http://dx.doi.org/10.1016/j.adhoc.2012.02.016
-  O. Arias, K. Ly, and Y. Jin, “Security and Privacy in IoT Era,” Smart Sensors at the IoT Frontier, pp. 351–378, 2017.
-  R. Langner, “Stuxnet: Dissecting a cyberwarfare weapon,” IEEE Security and Privacy, vol. 9, no. 3, pp. 49–51, 2011.
-  P. G. B. L. Bencsáth, B. and M. Félegyházi, “Duqu: A Stuxnet-like malware found in the wild,” CrySyS Lab Technical Report, vol. 14, pp. 1–60, 2011.
-  A. Seshadri, A. Perrig, M. Luk, L. Van Doom, E. Shi, and P. Khosla, “Pioneer: Verifying code integrity and enforcing untampered code execution on legacy systems,” Operating Systems Review (ACM), vol. 39, no. 5, pp. 1–16, 2005.
-  Group-Trusted-Computing, “TPM Library Specification (TPM 2.0 specifications),” 2014. [Online]. Available: https://trustedcomputinggroup.org/tpm-library-specification/
-  J. Winter, “Trusted Computing Building Blocks for Embedded Linux-based ARM Trustzone Platforms,” Proceedings of the 3rd ACM Workshop on Scalable Trusted Computing, pp. 21–30, 2008.
-  A. Francillon, Q. Nguyen, K. B. Rasmussen, and G. Tsudik, “A minimalist approach to Remote Attestation,” in Design, Automation & Test in Europe Conference & Exhibition (DATE), 2014. New Jersey: IEEE Conference Publications, 2014, pp. 1–6. [Online]. Available: http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6800458
-  K. Eldefrawy, G. Tsudik, A. Francillon, and D. Perito, “SMART : S ecure and M inimal A rchitecture for ( Establishing a Dynamic ) R oot of T rust,” In Network and Distributed System Security Symposium, vol. 12, pp. 1–15, 2012.
-  P. Koeberl, S. Schulz, A.-R. Sadeghi, and V. Varadharajan, “TrustLite: A Security Architecture for Tiny Embedded Devices,” Proceedings of the European Conference on Computer Systems (EuroSys), pp. 1–14, 2014. [Online]. Available: http://dl.acm.org/citation.cfm?id=2592798.2592824
-  F. Brasser, B. El Mahjoub, A.-R. Sadeghi, C. Wachsmann, and P. Koeberl, “TyTAN: Tiny trust anchor for tiny devices,” Proceedings of the 52nd Annual Design Automation Conference on - DAC ’15, pp. 1–6, 2015. [Online]. Available: http://dl.acm.org/citation.cfm?doid=2744769.2744922
-  L. Liu, G. Yan, X. Zhang, and S. Chen, “VirusMeter: Preventing Your Cellphone from Spies,” in Recent Advances in Intrusion Detection: 12th International Symposium, RAID 2009, Saint-Malo, France, September 23-25, 2009. Proceedings, 2009, pp. 244–264.
-  J. M. Kang, S. S. Seo, and J. W. K. Hong, “Usage pattern analysis of smartphones,” in APNOMS 2011 - 13th Asia-Pacific Network Operations and Management Symposium: Managing Clouds, Smart Networks and Services, Final Program, 2011.
-  T. Evans, W. L. Barth, J. C. Browne, R. L. Deleon, T. R. Furlani, S. M. Gallo, M. D. Jones, and A. K. Patra, “Comprehensive resource use monitoring for HPC systems with TACC stats,” in Proceedings of HUST 2014: 1st International Workshop on HPC User Support Tools - Held in Conjunction with SC 2014: The International Conference for High Performance Computing, Networking, Storage and Analysis, 2014.
-  N. Sorkunlu, V. Chandola, and A. Patra, “Tracking System Behaviour from Resource Usage Data,” arXiv preprint, 5 2017. [Online]. Available: http://arxiv.org/abs/1705.10756
L. Caviglione, M. Gaggero, J. F. Lalande, W. Mazurczyk, and M. Urbański, “Seeing the unseen: Revealing mobile malware hidden communications via energy consumption and artificial intelligence,”IEEE Transactions on Information Forensics and Security, 2016.
-  A. Stavrou, J. Voas, and I. Fellow, “DDoS in the IoT,” Computer, vol. 50, pp. 80–84, 2017.
-  “Texture Feature Extraction,” in Perspectives on Content-Based Multimedia Systems, ser. The Information Retrieval Series. Boston: Springer US, 2002, vol. 9, pp. 69–91. [Online]. Available: http://link.springer.com/10.1007/b116171
-  A. Materka and M. Strzelecki, “Texture Analysis Methods – A Review,” Methods, vol. 11, pp. 1–33, 1998.
T. Zhao, J. Zhang, F. Li, and K. J. Marfurt, “Characterizing a turbidite system in Canterbury Basin, New Zealand, using seismic attributes and distance-preserving self-organizing maps,”Interpretation, vol. 4, no. 1, pp. SB79–SB89, 2 2016.
-  A. Baraldi and F. Parmiggiani, “Investigation of the textural characteristics associated with gray level cooccurrence matrix statistical parameters,” IEEE Transactions on Geoscience and Remote Sensing, vol. 33, no. 2, pp. 293–304, 1995.
-  R. M. Haralick, K. Shanmugam, and I. Dinstein, “Textural Features for Image Classification,” IEEE Transactions on Systems, Man, and Cybernetics, vol. SMC-3, no. 6, pp. 610–621, 1973. [Online]. Available: http://ieeexplore.ieee.org/document/4309314/
-  J. Benesty, J. Chen, Y. Huang, and I. Cohen, “Pearson Correlation Coefficient,” in Noise Reduction in Speech Processing. Springer Berlin Heidelberg, 2009, pp. 1–4.
-  N. S. Altman, “An Introduction to Kernel and Nearest-Neighbor Nonparametric Regression,” The American Statistician, vol. 46, no. 3, pp. 175–185, 8 1992. [Online]. Available: http://www.tandfonline.com/doi/abs/10.1080/00031305.1992.10475879
-  J. Ahrenholz, “Comparison of CORE network emulation platforms,” Proceedings - IEEE Military Communications Conference MILCOM, pp. 166–171, 2010.