A Heuristically Self-Organised Linguistic Attribute Deep Learning in Edge Computing For IoT Intelligence

06/08/2020
by   Hongmei He, et al.
Microsoft
0

With the development of Internet of Things (IoT), IoT intelligence becomes emerging technology. "Curse of Dimensionality" is the barrier of data fusion in edge devices for the success of IoT intelligence. A Linguistic Attribute Hierarchy (LAH), embedded with Linguistic Decision Trees (LDTs), can represent a new attribute deep learning. In contrast to the conventional deep learning, an LAH could overcome the shortcoming of missing interpretation by providing transparent information propagation through the rules, produced by LDTs in the LAH. Similar to the conventional deep learning, the computing complexity of optimising LAHs blocks the applications of LAHs. In this paper, we propose a heuristic approach to constructing an LAH, embedded with LDTs for decision making or classification by utilising the distance correlations between attributes and between attributes and the goal variable. The set of attributes is divided to some attribute clusters, and then they are heuristically organised to form a linguistic attribute hierarchy. The proposed approach was validated with some benchmark decision making or classification problems from the UCI machine learning repository. The experimental results show that the proposed self-organisation algorithm can construct an effective and efficient linguistic attribute hierarchy. Such a self-organised linguistic attribute hierarchy embedded with LDTs can not only efficiently tackle "curse of dimensionality" in a single LDT for data fusion with massive attributes, but also achieve better or comparable performance on decision making or classification, compared to the single LDT for the problem to be solved. The self-organisation algorithm is much efficient than the Genetic Algorithm in Wrapper for the optimisation of LAHs. This makes it feasible to embed the self-organisation algorithm in edge devices for IoT intelligence.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

01/12/2021

Reliable Fleet Analytics for Edge IoT Solutions

In recent years we have witnessed a boom in Internet of Things (IoT) dev...
04/04/2015

Managing Multi-Granular Linguistic Distribution Assessments in Large-Scale Multi-Attribute Group Decision Making

Linguistic large-scale group decision making (LGDM) problems are more an...
07/25/2020

Proactive Tasks Management based on a Deep Learning Model

Pervasive computing applications deal with intelligence surrounding user...
08/13/2020

Learnability and Robustness of Shallow Neural Networks Learned With a Performance-Driven BP and a Variant PSO For Edge Decision-Making

In many cases, the computing resources are limited without the benefit f...
07/29/2020

On the Use of Interpretable Machine Learning for the Management of Data Quality

Data quality is a significant issue for any application that requests fo...
05/25/2020

On Irrelevance of Attributes in Flexible Prediction

This paper analyses properties of conceptual hierarchy obtained via incr...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

The Internet-of-Things (IoT) provides us with a large amount of sensor data. However, the data by themselves do not provide value unless we can turn them into actionable and/or contextualized information. Big data analysis allows us to gain new insights by batch-processing and off-line analysis. Currently, a microprocessor-based sensor node can support many channels (e.g. a Microchip processor can support up to 49 channel inputs datasheet2017 ). Real-time sensor data analysis and decision-making is preferably automated on-board of IoT devices, which will make IoT intelligence towards reality.

Although the computing capability of a microprocessor has improved very much, the ‘curse of dimensionality’ is still a big challenge in data driven machine intelligence, as the computing complexity of designed model function increases as the increasing of input space. Blum and Rivest Blum1992

have proved that training a 2-layer, 3-nodes and n inputs neural network is NP-Complete. Obviously, the big barrier of blocking the applications of deep-learning is the computing complexity, although it shows great attractive on solving complex nonlinear problems. With the strong capability of GPU, deep learning for 2-20 depth networks is successful (e.g. Google AlphaGo). To save the cost, an edge device of IoT systems may not need to equip with GPUs, if the machine intelligence algorithm inside the device is efficient enough. Hence, the performance improvement of computational intelligence is continuous work, especially for creating effective and efficient computing model in edge devices to implement IoT intelligence.

Linguistic decision tree (LDT), a probabilistic tree, has been well used for decision making and classification problems. Given an input space of n attributes, each of which can be described with limit labels. An LDT consists of a set of attribute nodes and a set of edges. An edge, linking from an attribute node, represents a label expression that describes the attribute node (see Section 3.2). However, the branch number of a decision tree exponentially increases as the number of input attributes increases. This shortcoming of an LDT greatly blocks its applications.

A hierarchical model could help overcome the “Curse of Dimensionality” in fuzzy rule-based learning Raju1991 . Campello and Amaral Campello2006 provided a cascade hierarchy of sub-models, using fuzzy relational equations Pedrycs1993 , and unilaterally transformed the cascade models into the mathematically equivalent non-hierarchical one. Lawry and HeLawry2008 proposed a linguistic attribute hierarchy (LAH). It is a hierarchy of LDTs, each of which has a reduced input space, and represents different functions in the problem space. However, as the relationship between inputs and the output in the whole problem space could be strong non-linear and uncertain, different LAHs will have different performance for the problem to be solved. All of the research in Campello2006 ; Lawry2008 neither investigated the performance of the proposed hierarchies, nor studied how the hierarchies can be constructed optimally.

He and Lawry He2009a ; He2009b investigated cascade LAHs, and developed a genetic wrapper algorithm (GWA) to optimise the cascade LAHs. It was shown that a cascade hierarchy can derive much less rules, compared to the single decision tree. However, the accuracy tends to drop as decision threshold increases. Later, they also investigated the optimisation of generic LAHs He2014

. The optimisation of LAHs using the GWA is NP-complete, as the branch number of a decision tree is increasing exponentially as the input attribute number increases, and the convergency of GWA directly affects the speed of optimisation. Hence, the evolutionary algorithm takes very long time to optimise LAHs, even if the evolution takes a small number of iterations. When the attribute number is over 60, the time of evolution process on a PC is not acceptable.

Another issue of hierarchical approaches, including generic deep learning, is the lack of linguistic interpretability, especially neural networks have been viewed as a black box. This is because that intermediate variables, being arbitrarily introduced to interconnect the sub-models, do not present any meanings in a real system, and hence it is difficult to give a clear intuitive interpretation Campello2006 . However, in an LAH, information propagation is completed via the linguistic rules extracted from LDTs. Hence, a transparent hierarchical decision making or classification can be provided.

In a distributed system, we may get partial information from different information resources, based on which, an initial estimate/decision could be done locally. Correspondingly, a collective decision is required. How do these information resources make contribution to the final collective decision? In an LAH, an intermediate variable is equivalent to the initial decision in terms of partial information

He2009a ; He2009b . Hence, the structure of the hierarchy determines the process of decision or fusion process. He et. al. He2015a further proposed an off-line optimisation of sensor arrangement for task-oriented sensor fusion, aligning with the structure of an self-organised wireless sensor network.

Mitra et al.Mitra2002

proposed a fuzzy knowledge-based network, whose structure can be automatically optimised by using linguistic rules of a fuzzy decision tree, thus to help in reducing the complexity of network training for a specific task. An LAH is equivalent to a forward neural network, in which, each neuron is the function of lower layer attributes, but represented by an LDT. The structure of an LAH is different to that of a classic multi-layer feed-forward neural network, in which, all input attributes are in the input layer. In an LAH, input attributes can feed a neuron (e.g. an LDT) at any layer, but for the simplicity of network structure, they cannot repeatedly feed more than one neuron.

The information propagation from bottom to top of an LAH through the LDTs in the LAH provides a new approach to attribute deep learning. In fact, a neural network or any other machine learning models can be used, instead of an LDT in the LAH. For example, a cascade of CMACs has been proposed in He2015b . However, LAH embedded with linguistic decision trees could provide good interpretation for decision making or classification.

He et al. He2017 used an LAH to interpret the process of semantic attribute deep learning for spam detection. The LAH was constructed manually in terms of the semantics of attributes. For IoT intelligence or edge computing, the onboard adaptive sensor fusion is a critical challenge, due to the limit of resources in edge devices (e.g. computing capacity, memory and even power supply). Hence, efficiently automatic constructing a linguistic attribute hierarchy becomes necessary for this purpose.

In this research, a self-organised LAH (SOLAH), embedded with LDTs is proposed for the deep learning of attribute semantics in decision making or classification. A distance correlation-based clustering algorithm is proposed to decompose attributes to several clusters, and a linguistic attribute hierarchy is constructed with the produced clusters in terms of the average distance correlation of cluster members to the goal variable. The LDT, fed by a cluster of attributes with lower distance correlation, will be placed in the lower layer of the LAH. The preliminary experiments are conducted on the SMS spam database from UCI machine learning repository Lichman2013 . A set of databases will be used to validate the performance of the SOLAHs. The experimental results are compared with that of the single LDTs for different databases.

2 Existing work in Deep Learning

Recently, Deep Learning has been put a lot of attention by researchers. It uses multiple processing layers of computational models to learn representations of data with multiple levels of abstraction. Each successive layer is fed by the outputs of previous layer, forming a hierarchy of attributes from bottom to top. Deep learning uses the back-propagation algorithm to learn internal parameters that are used to compute the representation in each layer from the representation in the previous layer, thus to find the hidden structure or information in large data sets. Various deep learning architectures such as deep neural networks, convolutional deep neural networks, deep belief networks and recurrent neural networks have been developed. Deep learning has been successfully applied in the areas of speech recognition

Miao2015 ; ZhangZ2017 ; Zhang2019 , image processing HeK2016 , object detection Girshick2014 ; Ren2017 , drug discovery Gawehn2016 and genomics Park2015 , etc. Especially, deep convolutional nets (ConvNets) have brought about breakthroughs in processing images Chen2016 , video Karpathy2014 , speech Zhang2019 and audio Lee2009 , whereas recurrent nets have shone light on sequential data such as text and speech LeCun2015 .

ConvNets are constructed layer by layer for data processing. A classic application of a ConvNet is in image processing, utilising the properties of a colour image, consisting of the RGB channels of pixel values. Since the early 2000s, ConvNets have been successfully applied for the detection, segmentation and recognition of objects and regions based on images, for example, face recognition

Taigman2014 .

Druzhkov and Kustikova Druzhkov2016

did a survey on deep learning methods for image classification and object detection, covering autoencoders, restricted Boltzmann machines and convolutional neural networks. For example, Szegedy et al.

Szegedy2013

used deep neural networks to solve the problem of object detection for both classifying and precisely locating objects of various classes; Simonyan et al.

Simonyan2013

used deep fisher networks for the classification of large-scale images from ImageNet (

http://image-net.org/); Krizhevsky et al. Krizhevsky2012 trained a large deep convolutional neural network to classify the 1.3 million high-resolution images in the LSVRC-2010 ImageNet training set into the 1000 different classes; and He et al. HeY2014 proposed an unsupervised feature learning framework, Deep Sparse Coding, which extends sparse coding to a multi-layer architecture for visual object recognition tasks.

Acoustic modeling is another area, where large, deep neural networks have been successfully applied. Like ImageNet, the massive quantities of existing transcribed speech data provide rich resources for deep learning. Dahl Dahl2015 provided a brief review of the deep neural net approach to large vocabulary speech recognition (LVSR) in his thesis. Mohamed et al.Mohamed2012

showed that hybrid acoustic models of pre-trained deep neural networks, instead of Gaussian mixture models (GMMs), could greatly improve the performance of a small-scale phone recognition; By using rectified linear units and dropout, Dahl et al.

Dahl2013

further improve the model for a large vocabulary voice search task. A combination of a set of deep learning techniques has led to more than 1/3 error rate reduction over the conventional state-of-the-art GMM-HHM (Hidden Markov Model) framework on many real-world LVCSR tasks

Yu2015 .

The qualitative properties of text data, very different to that of other modality data, provide critical challenges in use of machine learning. Recently Socher et al. Socher2013 developed recursive deep models for semantic compositionality over a sentiment treebank, and improved the accuracy. Natural language understanding is another exploration of deep learning application, which could make a large impact over the next few years LeCun2015 .

The property of compositional hierarchies of some signals is well exploited by deep neural networks through composing lower-level features to abstract the high-level one. For example, an image can be represented by a hierarchy from local edges, motifs, parts, to objects. Similarly, speech and text also have a hierarchy from sounds to phones, phonemes, syllables, words and sentences. In other words, this properties promote the capacities of deep neural networks. However, the interpretability of the decision making process is still an issue. Also, we cannot always explicitly see the semantics of higher-level features in other application domains.

With rapid progress and significant successes in a wide spectrum of applications, deep learning is being applied in many safety-critical environments. Ching et al. Ching2018 forecasted that deep learning enabling changes at both bench and bedside with the potential to transform several areas of biology and medicine, although the limited amount of labelled data for training presents problems as well as legal and privacy constraints on work with sensitive health records. However, deep neural networks (DNNs) have been recently found vulnerable to well-designed input samples called adversarial examples, and adversarial perturbations are imperceptible to human but can easily fool DNNs in the testing/deploying stage Yuan2019 . The transparency of LAH allows the observation of the decision process. He et al. He2017 have investigated the effect of different input attributes on different positions in a linguistic attribute hierarchy. This might indicate a linguistic attribute deep learning could provide good transparency to help people to defend adversarial perturbations to IoT Intelligence, which is out of the scope of this research.

GPU has become the necessary hardware facility for the research on deep learning due to its complexity. Justus et al. Justus2018

analysed various factors that influence the complexity of deep learning, and divided these factors to three categorises: (1) Layer features, such as activate function, optimiser of weights (e.g. Gradient Descent) and number of training samples; (2) layer specific features, including four subcategories: i. Multi- Layer Perception features (e.g. number of inputs, number of neurons and number of layers), ii. Convolutional features (e.g. matrix size, kernel size, stride size, input padding, input depth and output depth, etc.); iii. Pooling features (e.g. matrix size, stride size, input padding), iv. Recurrent features (e.g. Recurrence type, bi-directionality); (3) Hardware Features (e.g. GPU technology, GPU count, GPU memory, GPU memory bandwidth, GPU clock speed, GPU cord count, GPU peak performance, Card Connectivity). The first two categories of factors determine the performance of the function represented by a deep learning model, while the GPU feature directly affect the efficiency of the computing. It is hard to express the polynomial relationship between the computing cost and all the various factors, or use a big O(*) to represent the computing complexity.

3 Linguistic attribute deep learning with a LAH

3.1 A linguistic attribute hierarchy (LAH)

The process of aggregation of evidence in multi-attribute decision making or classification based on attributes can be represented with a functional mapping , where is the goal variable. However, this mapping is often dynamic and uncertain, and it is difficult to find a mathematic equation to precisely describe the mapping function. An attribute hierarchy represents the function with a hierarchy of sub-functions, each of which represents a new intermediate attribute. The set of original attributes is categorised into clusters . When a cluster of attributes is used to make initial decision, an new intermediate attribute is produced by the function of the clustering attributes. One or more intermediate attributes can be combined with another cluster of attributes to make further decision in next level, or multiple intermediate attributes can be directly used to make decision in next level. As there are attribute clusters, at least subfunctions are produced. Namely, for and , as the maximal level of hierarchy with input attributes is the cascade hierarchy with subfunctions. The mapping function is represented by a new function of the intermediate attributes and/or a cluster () of input attributes. The intermediate attributes can be represented by subfunctions of , fed by lower level of attribute set, which could include intermediate attributes and/or another cluster of input attributes. Hence, = = ,…,.

A linguistic decision tree (LDT) can explicitly model both the uncertainty and vagueness in a mapping system, and linguistic rules extracted from the LDT often implies our knowledge of aggregation. When an LAH uses LDTs to represent the functional mappings between parent and child attribute nodes, the semantic information will be transparently propagated through the hierarchy from bottom to top. Assume the goal variable , and a set of labels can be used to describe the goal variable. An introduced intermediate attribute will represent an approximate of the goal variable . Namely, , and the label set can be used to describe . Hence, an LAH can present a hierarchical decision making, and it can provide transparent linguistic interpretation, which helps perform semantic attribute deep learning.

For examples, assume five attributes are fed to an LAH. Fig. 1 (a) shows a cascade LAH, embedded with 4 LDTs, through which, the decision information is cascaded from the bottom model to the top model ; Fig. 1 (b) illustrates a general LAH, embedded with 3 LDTs, through which, the final decision is made by , based on the two intermediate attributes from and in the bottom of the LAH, and the additional attribute . The intermediate attributes and represent an approximate of the goal variable , is decided by and , and is decided by and .

(a) A cascade LAH
(b) A general LAH
Figure 1: Two LAHs with five input attributes

3.2 A Linguistic decision tree based on Label Semantics

An LDT Qin2005a ; LawryJ2008 is a probabilistic tree under the framework of label semantics Lawry2006 . Label semantics introduces two fundamental and interrelated measures: Appropriateness measure () and mass assignment (), where, quantifies how a label is appropriate to describe x based on agent’s knowledge of the current labeling conventions, shown in the collected samples or evidences; quantifies how a particular subset of labels is appropriate to describe . The particular subset of all and only the labels that are appropriate to describe is called focal set, denoted as . Given a set of labels , the focal set can be = , , , , .

An LDT consists of a node set and an edge set . A node is associated to an attribute, and an edge from a node is a focal set, which is appropriate to describe the attribute, indicated by the node. A path from the top node to a leaf of the LDT is called a branch, denoted as . The branch length () is the number of nodes on the branch . Each branch provides a conjunction of focal sets: , with the inference of a set of mass assignments conditional to the branch , denoted as , for each focal set , where, is a set of focal sets that are appropriate to describe different values distributed in the domain of the goal variable . The semantics of an LDT is described in Definition 3.1.

Definition 3.1 (Semantics of an LDT)

The rule derived from a branch in an LDT is presented as:

(1)

where, is the -th focal set on branch , and . Given attribute values . The mass assignment can be obtained using Jeffrey’s rule Jeffrey1990 :

(2)

where is the branch number of the LDT, and

is equivalent to the conditional probability

. denotes the -th attribute on the -th branch. Assume a focal set is appropriate to describe attribute on the branch . The appropriateness measure of on branch is the product of all mass assignments on focal sets that are appropriate to describe the corresponding nodes on the branch, as expressed in Formula (3).

(3)

4 The semantics of a linguistic attribute hierarchy

When an LAH is represented a hierarchy of LDTs, the information is propagated through the LDTs from information sources (e.g. sensors) in the bottom to the decision variable on the top. Therefore, the rules can be derived as conditional expressions through Formula 1 in the label semantics framework. The output variable (either an intermediate attribute or a goal variable) of each LDT in the LAH can be calculated with Formula (2). This provides an approach to quantifying the degree of our belief how each focal set is appropriate to describe the goal, given partial input attributes and/or the previous results of decision or classification (intermediate attributes) in lower level.

Now, we use the LAH in Fig. 1 (b) as an example to demonstrate the upwards information propagation through an LAH. In Fig. 1 (b), two LDTs (, ) are located in the bottom of the LAH. . The mappings , , and are represented in the form of linguistic decision trees (i.e. , and ). provides the function of quantifying our belief of on goal labels in terms of input attributes and , presents the function of quantifying our belief of on goal labels in terms of input attributes and , and offers the function of quantifying our final belief of the goal variable on its labels in terms of lower level believes of and , as well as another input attribute from information source (e.g. sensors). Assume the input attributes are clustered to subsets , which can feed the intermediate attributes or/and the goal variable in an LAH. As an intermediate attribute is the approximate of the goal variable , it can be estimated by a decision making model (e.g. LDT), fed with . Hence, the decision making model can be trained with the samples of (). Given the values of all input attributes , the mass assignments of all intermediate attributes and the goal variable can be estimated with Formula (2) through all the LDTs in the LAH. Namely, the decision information is propagated through the bottom LDTs to the top LDT in the LAH. The semantics of an LAH is defined as Definition 4.1.

Definition 4.1 (Semantics of an LAH)

The semantics of an LAH is the synthetisation of rules extracted from the branches in the LAH, which are allocated by the given sample, . Assume outputs (i.e. intermediate attributes) of LDT() … LDT() are the inputs of LDT(), the rule will be , where can be derived in the form of Formula (1), by an LDT, either directly based on information sources (i.e. the input attributes), or based on the intermediate attributes from the lower level of LDTs.

For the instance of Fig. 1 (b), given all the input attribute values, = , , , , in all samples. For generality, we use to denote the label expression, associated to an edge in an LDT. For the special case, the edges of an LDT are associated to a focal set, represents a focal set . Hence, the semantics of each decision tree can be described as:

For , as it is fed with two attributes, the maximum branch length is 2. Hence, the rule corresponding to the branch can be:

Similarly, for , the rule corresponding to the branch can be:

For , the maximum branch length is 3. Hence, the rule corresponding to the branch can be:

The synthetic semantics of the LAH is:

5 Construction of Linguistic Attribute Hierarchy

The basic idea for the construction of a Linguistic Attribute Hierarchy includes two steps: (1) Decomposition of attributes with a distance correlation based clustering algorithm;
(2) Self-organisation of a linguistic attribute hierarchy in terms of the distance correlation between clustered attributes and the goal variable.

5.1 Attribute decomposition based on their distance correlation

5.1.1 Distance correlation

Distance correlation can be used to statistically measure the dependence between two random variables or vectors, which could have different dimensions. It is zero if and only if the random variables are statistically independent. Assume

be a set of samples. The by distance matrices and present the all pairwise distances for and in all the samples, respectively.

(4)

where denotes Euclidean norm. Then take all doubly centered distances:

(5)

where, is the -th row mean, is the -th column mean, and is the grand mean of the distance matrix for the samples. The notation is similar for the values on the samples. The squared sample distance covariance (a scalar) is simply the arithmetic average of the products and :

(6)

The sample distance variance is the square root of

(7)

The distance correlation of two random variables can be calculated through dividing their distance covariance by the product of their distance standard deviations, as Fomula (

8).

(8)

Distance correlation has the properties:

  • , ;

  • The distance correlation matrix is symmetric (i.e. ), and =1.

  • , if and only if and are independent;

  • and indicates that the linear subspaces spanned by and samples respectively almost surely have an identical dimension Szekely2009 .

5.1.2 Clustering attributes based on distance correlation

As we do not need to consider the distance correlation between an attribute and itself, we set the diagonal of the distance correlation matrix to zeros for the convenience of computing. First, we find the maximum value in the distance correlation matrix . The maximum value is denoted as , the column of the maximum value in the matrix is denoted as . Here we set a value as the range of distance correlation difference in a cluster. Namely, the distance correlation of attributes in a cluster, correlating to attribute will be in . A feasible approach to setting the value of is:

(9)

Usually, is not a zero, and is the preset number of clusters. Secondly, all relevant columns and rows, where the identified cluster members in the cluster are located, are set to zeros. The process is repeated with starting to find next attribute with largest distance correlation in the unvisited attributes for the next cluster, until all elements of are zeros. Algorithm 1 provides the pseudo-code of the clustering algorithm, where the produced clusters are saved in .

1:  Initialise(;
2:  t=0;
3:  while (do
4:       = +1;
5:      ;
6:      ;
7:      ;
8:      );
9:      ;
10:      );
11:      );
12:  end while
Algorithm 1 Distance Correlation Clustering (,)

5.2 Self-organisation of linguistic attribute hierarchy

Once attributes are clustered based on their distance correlation, we can construct a linguistic attribute hierarchy in terms of the distance correlation between clustered attributes and the goal variable. If an attribute has stronger distance correlation with decision variable, this indicates the attribute has stronger linear correlation to the goal variable, namely, the mapping function between the attribute and the goal could be more linear. In other words, it is easier to make decision in terms of the attribute. Our preliminary experimental results show that attributes, which have stronger distance correlation to the goal variable, should be fed to the LDT in higher layer of the linguistic attribute hierarchy. The average of distance correlation between attributes in a cluster and the goal variable can be calculated with Formula (10):

(10)

where, is the attributes in cluster , is the goal variable, and is the size of cluster (i.e. number of attributes in cluster ).

The basic idea of constructing an LAH based on the produced clusters is that:

  • calculating average distance correlation for each produced cluster;

  • Sort all average distance correlations between clusters and decision variable;

  • The LDT fed by the cluster with lower average distance correlation will be at lower level of the LAH;

  • The LDTs fed by the clusters that have similar average values of distance correlation will be at the same level of the LAH. A threshold is set for the assessment of clusters at the same level. The outputs of all LDTs at the same level will be the partial inputs of the next LDT in the next level of LAH;

  • The LDT fed by the cluster with the highest average distance correlation will be on the top of the LAH.

Assume represents the average distance correlation of cluster to the goal variable, and represents the maximum difference of distance correlation values of clusters at the same level to the goal variable. Namely, if

(11)

then the LDTs fed with and will be at the same level in the LAH.

Algorithm 2 provides the pseudo-code of the self-organisation of LAH, where is the vector of distance correlation of all attributes to the goal variable, is the set of clusters produced by the distance correlation-based clustering algorithm above, is the set of all the average distance correlations for cluster set with clusters to the goal variable, is the index of an intermediate attribute, which starts from , the input attribute number, is used to save the indices of intermediate attributes and the corresponding clusters at the same level, is used to save the hierarchy to be constructed. For all clusters, if , save the intermediate attribute index to , and append and the cluster to the hierarchy , else, start a new level, append current , other intermediate attributes saved in and corresponding cluster to the hierarchy , and save current to . If more than one LDTs at the same level in the LAH, and there is no further cluster to be constructed, then the outputs of those LDTs will be the inputs of the top LDT in the LAH. If the LDT constructed with the last cluster does not share the same level with other LDTs, then it will be the top LDT of LAH.

1:  [] = avDistCorr();
2:  ; =||;
3:  [] = sort(); %increasingly
4:  ;;
5:   = 0, = 0;
6:  ;
7:  while (do
8:      +1; +1,
9:      if ((then
10:          ;
11:          );
12:      else
13:          ; % starting a new level
14:          );
15:          ;
16:      end if
17:  end while
18:  if ( > 1) then
19:      ;
20:      +1;
21:  else
22:      ;
23:  end if
Algorithm 2 SOLAH(, , )

5.3 Training of a linguistic attribute hierarchy

In 2009, He and Lawry He2009a ; He2009b first time proposed using the domain and labels of the goal variable to describe intermediate attributes. This made the hierarchy training became possible, and the hierarchical decision making became meaningful. In He2014 , He and Lawry developed a non-recursively post-order traversal algorithm to train a given LAH, where all LDTs are trained with LID3 in a bottom-up way. Intermediate attribute values are estimated with the trained LDTs, and then they are input to next level LDTs. For example, in Fig. 1 (b), , and are trained in turn, where, the intermediate variables and use the corresponding values in all training points, as the goal values of and respectively. After and are trained, the intermediate variables and for the training samples can be estimated by and , respectively. The estimated values of and , the values of , and the values of the goal variable in the training samples are used to train the , and finally the goal variable can be estimated by . Here, we introduce a recursive postorder implementation of the LAH training algorithm (see Algorithm 3).

is a global variable, indicating the LAH to be trained. Initially, is the root of the LAH to be trained, and all leaves in the LAH are tagged as visited, as they are the input attributes from sensor nodes. represents the child set of node , indicating the attributes that feed to the node . is used to save the attributes that are not on the branch from the top root to the current node , and is to remove from set .

  if (node has been visited) then
      return;
  else
      for (=1..do
          (, );
      end for
      =;
      ;
      ;
  end if
Algorithm 3 postorder(,)

5.4 The LID3 algorithm for LDT training

LID3 was used to train an LDT with a given database Qin2005a ; He2009a . It is an update of the classic ID3 algorithm Quinlan1986 , through combining label semantics. The training process is conducted through selecting the attribute that obtains the maximum information gain to extend a branch. The functions used in LID3 have been formulated in Qin2005a ; He2009a , such as information entropy of a branch (), expected entropy () when an attribute node is added to a branch, and information gain (IG). Here, a recursive algorithm implementation of LID3 is proposed (Algorithm 4).

A threshold is setup to stop the branch extension when the conditional probability reaches the threshold . The maximum of branch length is the number of attributes that feed the LDT. Initially, (empty), hence, current node point as well. For each branch, the conditional probabilities for all classes are calculated. The most informative attribute is selected as the next node to be extended to the existing LDT, and all focal sets that are appropriated to describe the attribute in the domain of will be appended to the tree . The process is continued until the maximum conditional probability arrives the specified threshold.

  for (all do
       = conditionProb();
  end for
  if () or  then
      return;
  end if
  for (all do
      ;
  end for
  ;
  
  ;
  ;
  for (all do
      ;
  end for
Algorithm 4 )

6 Experiments and Evaluation

6.1 Experiment methodologies

All databases, used for the experiments, are from UCI machine learning repository Lichman2013 . The experiments are conducted in two stages:

(1) A case study will be done on the benchmark database of Message Spams. This experiment is to demonstrate the use of developed approach for self-organisation of a linguistic attributes hierarchy, given the data.

(2) A set of databases will be tested with the SOLAHs. The performance are used to validate the developed approach to self-organising LAHs by comparing with the performance obtained with the single LDT.

Experimental environment: The experiments are carried out on a laptop with 64-bit Windows 10 and x64-based processor with Intel (R) Core (TM) i5-4210U CPU @1.7GHZ 2.4GHZ, 8GB memory. The LAH training algorithm is implemented in C++.

Discretisation:

For simplicity, all attributes are expressed with three labels, except binary variables, decision or goal variables, which are expressed with the labels as they have. All neighbouring fuzzy intervals are overlapping with 50%.

Mass assignments of attributes: In terms of Label Semantics, if labels are used to describe a continuous variable , then there will be focal sets that are appropriate to describe . For discrete variables, the mass assignment on a focal set that contains only one label is 1, but the mass assignment on a focal set that contains two successive labels is 0.

Ten-fold cross validation: 90% of data is used for training, 10% is used for test. Therefore, data is partitioned to ten parts equally.

Performance measure: The performance is measured with:

(1) Accuracy: , where, is the number of true positive estimates and is the number of true negative estimates, and is the number of samples in the database.

(2) Area under ROC curve (): A ROC curve is used to measure how well the classifier separates the two classes without reference to a goal threshold. The area under the ROC curve has been formalized in He2014 .

(3) Rule number : As defined in He2014 , the rule number of an LAH is the sum of branch numbers, extracted from all LDTs in the LAH. Since a probability threshold was introduced during the process of LDT training, a branch training may stop earlier. Hence, the actual branch number in an LDT may be much less than a full LDT.

(4) The running time : it is the time, spending on the process of ten-fold crossing validation, including training, testing, and overhead for data splitting and exchanging.

6.2 A case study on a benchmark database

To demonstrate the developed approach, the first experiment is carried out on the benchmark database, ’SMSSpamCollection’. The performance for the LAH that was constructed by the proposed approach, the LAH in He2017 , obtained manually, and the single LDT, is evaluated and compared.

6.2.1 The database

The SMSSpamCollection database Almeida2011

, has 5574 raw messages, including 747 spams. The two sets of features, extracted for the research

He2016 ; He2017 , are used for the experiments. Table 1 shows the set of 20 features, and Table 2 presents the set of 14 features.

key word key-word
0 urgent 10 stop
1 congrat 11 click
2 ! 12 Text,Txt
3 WIN/WON 13 sex
4 Offer 14 girl
5 Award 15 cash
6 Prize 16 free
7 Call 17 0p, 1p, …, 9p
8 Reply 18 EURO, GBP, pound,, $, €
9 Send 19 price
Table 1: The set of 20 features He2016
key word key-word
0 urgent 7 call
1 congrat 8 Reply/send
2 ! 9 stop/click
3 WIN/WON 10 text/txt
4 Offer 11 cash/price
5 Award/Prize 12 free/0p…9p
6 sex/girl 13 EURO, GBP, pound,, $, €
Table 2: The set of 14 features He2017

6.2.2 Experimental results on SMS-20

The six clusters obtained by the proposed distance correlation -based clustering algorithm are: :{,,,}, :{,}, :{,}, :{,,,}, :{,}, :{,,,,,}.

has the lowest average distance correlation to the decision variable. Hence, the LDT fed by should be at the bottom of the SOLAH, and produces an intermediate attribute . Here is used to represent an intermediate attribute. with set feeds an LDT in the second level according to the average distance correlation of to decision variable. Sets and have similar average distance correlations to decisional variable. Hence, they sit at the same level in the SOLAH, and their outputs and feed the third level of LDT constructed with set . Sets and have similar average distance correlations to the decision variable. Hence they are at the same level. As they are at the top level, , and the output of the LDT at the third level will be an input of the top LDT. Figs. 2 and 3 illustrate the SOLAH and the best , manually produced in He2017 . The two LAHs have the same levels and the same number of LDTs. But the compositions are different.

Figure 2: The SOLAH on SMS-20
Figure 3: A LAH_m on SMS-20He2017

Table 3 shows the performance for the SOLAH, the in He2017 and the single LDT fed by the whole feature vector (denoted as . It can be seen that the accuracy of is very close to the performance of and , and the performance of the AUC is slightly smaller than that of , but better than that of . However, has the smallest branch number and running time , which is the time of ten-fold crossing validation on a solution, measured in milliseconds (ms).

LAHs A (ms)
, Fig. 2 0.936491 0.946402 197 101766
He2017 , Fig. 3 0.948511 0.949233 267 143718
0.956225 0.895815 775 1339234
Table 3: The performance of solutions

6.2.3 Experimental results on SMS-14

For this experiment, the cluster number is preset to 4. But the clusters obtained by the DCC algorithm are: :{,}, :{,,,}, :{,,,,,,}. Set has the lowest average distance correlation to the decision variable, thus the LDT fed by is placed at the bottom of the SOLAH. The LDT fed by set and the output of the first level at the second level of the SOLAH, and the LDT fed by set sits at the top level of the LAH.

Figs. 4 and 5 illustrate the SOLAH and the with the best , which was obtained by the attribute composition 3 in He2017 . The two LAHs have the same levels, and the attributes are decomposed to 3 clusters. But the compositions of LAHs are different. In the , all LDTs fed by the clusters of input attributes are placed at the bottom of the LAH. Hence a new LDT, fed by all intermediate attributes, which were produced by the bottom level of LDTs, is added on the top of the LAH. has higher level than , and all LDTs fed by the attribute clusters are cascaded.

Figure 4: The SOLAH on SMS-14
Figure 5: A LAH on SMS-14, obtained manually in He2017

Table 4 shows the performance for the SOLAH, the , constructed with composition 3 manually in He2017 and the single LDT (). All solutions in Table 4 obtained similar performance in accuracy and the area under ROC. All performance values of are in between and .

LAHs A (ms)
, Fig. 2 0.955328 0.943391 339 101766
He2017 , Fig. 3 0.951202 0.957466 130 79719
0.961607 0.924536 583 613704
Table 4: The performance of solutions

6.2.4 Impact of the preset cluster number

Similar to classic -means clustering algorithm, the preset cluster number is important for the construction of LAHs, and thus has important impact on the performance of decision making or classification. Here, the relationships between and structures of LAHs and between and performance are observed. It should be noticed that the real cluster number may be different to the preset cluster number, as the present cluster number decides the range () of distance correlation of a cluster , which is formed with the attributes that have the distance correlations in the range to an attribute , which has the largest distance correlation in the distance correlation matrix. The rest attributes could have larger distance correlations than the members except attribute in cluster . Table 5 shows the solutions for = 2…10. It can be seen that the real cluster numbers are different to the preset cluster number in some cases. The cluster numbers varies from 3 to 6. All SOLAHs have similar performance. The SOLAH constructed with 3 clusters obtained the highest accuracy, but it has more branches. The SOLAHs constructed with 4 clusters have the highest values. The clusters obtained when is the same as that when =8, which is happened when as well. In Table 5, denotes the branch number, denotes the level of the hierarchy.

A (ms)
2 3 0.956 0.946 339 4 221547
3 4 0.937 0.953 184 4 101875
4 4 0.936 0.950 154 4 84093
5 6 0.932 0.949 159 5 91375
6 6 0.932 0.948 154 6 89828
7,8 6 0.932 0.945 108 6 63313
9,10 6 0.932 0.941 130 5 76859
Table 5: The properties of LAHs on SMS14

Similarly, Table 6 lists the performance of SOLAHs constructed with various values from 2 to 10. The AUC values of all SOLAHs are very close. When the preset , the SOLAH was constructed with two clusters. It obtained the highest accuracy, but lowest AUC value and the largest branch number. When the preset =9,10, the same cluster sets were produced, with which, the SOLAH was produced. It has the lowest accuracy, but the smallest branch number. For all preset =3…6, different sets of 5 clusters were produced, with which, four different SOLAHs were produced. They have the same level, and their performance values ( and AUC) are very close. When the preset =5,6,7, the produced SOLAHs have the same performance values ( and AUC). But they have different branch numbers.

A (ms)
2 2 0.957 0.933 540 3 467328
3 5 0.934 0.950 189 5 111016
4 5 0.949 0.941 203 5 114265
5 5 0.936 0.946 187 5 101594
6 5 0.936 0.946 197 5 107766
7 6 0.936 0.946 204 5 115750
8 7 0.929 0.940 163 6 93891
9,10 7 0.891 0.944 157 7 104938
Table 6: The properties of LAHs on SMS20

From the data on both SMS20 and SMS14, it can be seen that a suitable cluster size will have a good trade-off on accuracy performance and time complexity (branch number).

6.3 Validation on some benchmark databases

6.3.1 The 12 benchmark databases

The experiments on 12 benchmark databases from UCI machine learning repository are conducted for validating the proposed approach of constructing LAHs. Table 7 provides the basic properties of the 12 data sets, including, database name, attribute number (), goal state number (), total sample number (), class distribution ().

Datasets
Beast Cancer (BC) 9 2 286 201,85
Wisconsin BC 9 2 569 357,212
Ecoli 7 8 336 143,77,52,35,20,5,2,2
Glass 9 6 214 70,17,76,13,9,29
Heart-C 13 2 303 165,138
Heart-statlog 13 2 270 150,120
Hepatitis 19 2 155 70,85
Wine 13 3 178 59,71,48
Liver Disorders 6 2 346 146,200
Diabetes 8 2 768 500,268
Ionoshpere 33 2 351 231,120
Sonar 60 2 208 97,111
Table 7: Database properties

6.3.2 Comparisons with LDT

Table 8 shows the performance of SOLAHs and corresponding single LDTs on the 12 benchmark data sets. It can be seen that the SOLAH achieved dominated better performance in accuracy and than the single LDT for most databases. For databases, Ecoli and Liver, SOLAHs achieve similar accuracy to LDTs, but their performance in is better than that of LDTs. The SOLAHs for most databases have less branches than the single LDT trained by the same database.

SOLAH LDT
Databases AUC AUC
BreastC. 0.78 0.84 985 9 4 3 0.70 0.64 3201 66
WBC 0.97 0.98 1100 36 3 2 0.95 0.96 1029 45
Ecoli 0.85 0.93 1103 15 4 3 0.85 0.89 3337 12
Glass 0.81 0.91 5124 48 3 2 0.68 0.83 9433 156
Heart-c 0.81 0.81 1452 24 5 4 0.76 0.78 1345 40
Heart-s. 0.79 0.83 972 11 5 4 0.77 0.80 1053 24
Hepatitis 0.92 0.94 4278 47 4 6 0.79 0.82 1677 45
Wine 1.00 0.99 929 6 5 5 0.94 0.98 1309 26
Liver 0.56 0.60 226 3 3 2 0.57 0.56 1585 24
Diabetes 0.76 0.82 1018 33 3 2 0.74 0.79 14865 667
Ionosphere 0.87 0.71 4393 14 5 9 0.87 0.87 12141 6953
Sonar 0.73 0.80 26328 52 6 22 0.67 0.73 9253 4968
Table 8: Performance comparison between constructed LAHs and the single LDTs for different databases, the unit of time (T) is second

As databases Ionosphere and Sonar have 33 and 60 attributes, respectively, the ten-fold crossing validation of a single LDT on the two databases is not acceptable. Therefore, the two-fold crossing validation of single LDTs on the two high dimension databases are performed. For comparison, the two-fold crossing validation of the constructed LAHs on the two databases are carried out as well. The results are provided in Table 10. For database Ionosphere, the LAH has the same accuracy as the LDT, but the area under ROC obtained by the SOLAH is smaller than that by the single LDT. Namely, the half size of data may not be enough to train the SOLAH. However, the running times of SOLAHs for the two databases are 14s and 52s respectively, which are much less than that of LDTs for the two databases. The database Sonar provides a very interesting case: the SOLAH has 26328 branches, while the single LDT has only 9253 branches. Regarding the branch number, the complexity of the SOLAH model is worse than the single LDT. But the running time for SOLAH is 52s, while 4968s for LDT. This is because that the computing time is not only related to branch number, but also related to branch length. The computing complexity can be . It can be seen that the SOLAH for the Sonar database has 22 layers, which means that at least 22 LDTs are embedded in the SOLAH. There are total 60 input attributes, plus 22 intermediate attributes, the average input attributes of an LDT in the SOLAH is 3.68. Hence, the average length of branch is 3,68. We can roughly say that the running time of the SOLAH is proportional to = 96887, but the single LDT has 4968 branches, the length of which is 60, then the running time of the single LDT is proportional to = 298080, which is much larger than the figure for the SOLAH.

In order to observe the performance improvement of the LAH that is trained by 90% of data when ten-fold crossing validation is applied, Table 9 particularly lists the performance of the SOLAHs with ten-fold crossing validation on the two high dimensional databases. The performance in both accuracy and the area under ROC is improved, especially for the database, Ionosphere, the performance in accuracy and the area under ROC are improved very much, compared to the SOLAH with two-fold crossing validation. Moreover, the running times of the SOLAHs for the two databases are 143s and 507s, respectively, which are much shorter than that of the single LDT with two-fold crossing validation in Table 8.

Databases
Ionosphere 0.95 0.98 7717 143 5 9
Sonar 0.78 0.84 33820 507 6 22
Table 9: Performance of SOLAHs on the two high dimensional databases

6.3.3 Comparisons with other machine learning algorithms

The average accuracy and standard deviations of SOLAH for ten runs on the 12 data sets from UCI machine learning repository are further compared with the optimal LAH, obtained by a Genetic Algorithm Wapper in He2014

, as well as three well-known machine learning algorithms, such as C4.5, Naive Bayes (NB) and Neural Networks (NN). The performance of these algorithms WAS evaluated with WEKA

WF00 by Qin and Lawry in Qin2005a , where, WEKA WF00 was used to generate the results of J48 (C4.5 in WEKA) unpruned tree, Naive Bayes and Neural Networks with default parameter settings for ten runs of 50%-50% splitting training and test data.

Table 10 shows the performance of SOLAHs and the existing results in literature for the 12 data sets. SOLAH wins 4 data sets, which is next to OptLAH, which wins 5 data sets. SOLAH obtains comparable performance in accuracy, compared to other algorithms, and even achieves better performance than the LAHs, optimised by GAW, for some tested data sets.

C4.5 N.B. N.N. OptLAH SOLAH
BreastC.
WBC
Ecoli
Glass
Heart-c
Heart-s.
Hepatitis
Ionosphere
Liver
Diabetics
Sonar
Wine
Table 10: Average accuracies (%) and standard deviations obtained by C4.5, NB, NN, OptLAH and SOLAHs

7 Conclusions

In this paper, the contribution of the innovative research are summarised as below:

(1) We proposed heuristic algorithm to construct a linguistic attribute hierarchy for semantic attribute deep learning in decision making or classification. The self-constructed linguistic attribute hierarchy provides a new form of deep learning, in contrast to conventional deep learning.

(2) The proposed algorithm for the self-organisation of an LAH is much more efficient than meta-heuristic algorithm, and the self-organised linguistic attribute hierarchy can obtain the fusion performance better than or comparable to the single linguistic decision tree, fed with the full set of attributes.

(3) The most important is that the heuristical self-organisation of such linguistic attribute hierarchy can effectively solve the ’curse of dimensionality’ in machine learning, which is critical challenge in the implementation of IoT intelligence. Hence, the research results will promote a wider of applications of the linguistic attribute hierarchy in big data analysis and IoT intelligence.

(4) A linguistic attribute hierarchy, embedded with linguistic decision trees, will provide a transparent hierarchical decision making or classification. Hence, it could help us to look insight of the decision making process for different purposes (e.g. the effect of adversary samples in decision making).

(5) Comparing with other machine learning in literature, the self-organised LAH obtains comparable performance on the tested data sets, and even achieved better performance than the optimal LAHs, obtained by GAW for some tested data sets.

We will implement the LAH on an embedded system (e.g. a Raspberry Pi system) for a specific task, and will further improve the algorithms and develop new algorithms to construct high efficient and effective linguistic attribute hierarchy, embedded with other machine learning models for decision making or classification in future.

References

  • (1) Microchip datasheet: dsPIC33EPXXXGM3XX/6XX/7XX. Acceessed 24 May 2020. https://ww1.microchip.com/downloads/en/DeviceDoc/70000689d.pdf.
  • (2) A. L. Blum and R. L. Rivest, Training a 3-Node Neural Network is NP-Complete. Neural Networks, 5, 1992, pp. 117-127.
  • (3) G.U. Raju, J. Zhou, and R. A. Kiner. Hierarchical fuzzy control. Int. J. Control, Vol. 54:55, pp. 1201-1216, 1991.
  • (4) Campello RJGB and Amaral WC (2006) Hierarchical Fuzzy Relational Models: Linguistic Interpretation and Universal Approximation, IEEE Transactions on Fuzzy Systems, 14(3):446-453.
  • (5) Pedrycs W (1993) Fuzzy Control and Fuzzy Systems, 2nd Ed, Research Studies Press, Ltd, Somerset, UK.
  • (6) Lawry J and He H (2008) Multi-Attribute Decision Making Based on Label Semantics, the International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 16(2) supp, pp. 69 - 86.
  • (7)

    He H and Lawry J (2009) Optimal Cascade Hierarchies of Linguistic Decision Trees for Decision Making, in Proc. of IAENG International Conference on Artificial Intelligence and Applications (ICAIA’09), Hong Kong,

  • (8) He H and Lawry J (2009) Optimal Cascade Linguistic Attribute Hierarchies for Information Propagation, IAENG International Journal of Computer Science, 36(2):129-136.
  • (9) [8] H. He and J. Lawry, Linguistic Attribute Hierarchy and Its Optimisation for Classification Problems, Soft Computing, 18(10), pp 1967-1984, Oct. 2014. DOI: 10.1007/s00500-013-1179-3.
  • (10) H. He, Z. Zhu and E. Makinen, Task-oriented Distributed Data Fusion in Autonomous Wireless Sensor Networks, Soft Computing, 19(8), 2015, pp. 2305–2319, DOI 10.1007/s00500-014-1421-7.
  • (11) S. Mitra, K.M. Konwar and S.K. Pal,Fuzzy decision tree, linguistic rules and fuzzy knowledge-based network: Generation and evaluation, IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, 32(4), 2002, pp. 328-339.
  • (12) H. He, Z. Zhu, A. Tiwari and Andrew Mills, A Cascade of Linguistic CMAC Neural Networks for Decision Making, in Proceedings of the International Joint Conference on Neural Networks (IJCNN2015), Killarney, Ireland, 12-17 July, 2015.
  • (13) H. He, T. Watson, C. Maple, J. Mehnen, A. Tiwari, Semantic Attribute Deep Learning with A Hierarchy of Linguistic Decision Trees for Spam Detection, IJCNN2017, Anchorage, Alaska, USA, 14-19 May 2017.
  • (14) Lichman, M. (2013). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science.
  • (15) Y. Miao, M. Gowayyed and F. Metze, EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding, CoRR, abs/1507.08240, 2015, http://arxiv.org/abs/1507.08240, arXiv:1507.08240.
  • (16) Z. Zhang, J. Geiger, J. Pohjalainen, A. E.-D. Mousa, W. Jin, and B. Schuller, Deep Learning for Environmentally Robust Speech Recognition: An Overview of Recent Developments, arXiv:1701.02720v1 [cs.CL], Jan. 2017.
  • (17)

    K. He, X. Zhang, S. Ren and J. Sun, Deep Residual Learning for Image Recognition, The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770-778.

  • (18) R. Girshick, J. Donahue, T. Darrell and J. Malik, Rich feature hierarchies for accurate object detection and semantic segmentation, The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, pp. 580-587.
  • (19) S. Ren, K. He, R. Girshick and J. Sun, Faster R-CNN: Towards Real- Time Object Detection with Region Proposal Networks, Journal IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6), June 2017, pp. 1137-1149.
  • (20) E. Gawehn, J. A. Hiss, G. Schneider, Deep Learning in Drug Discovery, Molecular Informatics, 35(1), 2016, pp. 3-14. WILEY-VCH Verlag, doi:10.1002/minf.201501008.
  • (21) Y. Park and M. Kellis, Deep learning for regulatory genomics, Nature Biotechnology 33, Aug. 2015, 825–826. doi:10.1038/nbt.3313.
  • (22) L. Chen, G. Papandreou, I. Kokkinos, K. Murphy and A. L. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs, CoRR, abs/1606.00915, 2016, http://arxiv.org/abs/1606.00915, arXiv: 1606.00915.
  • (23) [22] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, F. F. Li, Large-scale Video Classification with Convolutional Neural Networks, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA, 23-28 Jun. 2014. DOI: 10.1109/CVPR.2014.223
  • (24) [23] W. Zhang, M. Zhai, Z. Huang, C. Liu, W. Li, and Y. Towards End-to-End Speech Recognition with Deep Multipath Convolutional Neural Networks,In: Yu H., Liu J., Liu L., Ju Z., Liu Y., Zhou D. (eds) Intelligent Robotics and Applications. ICIRA 2019. Lecture Notes in Computer Science, vol 11745. Springer, Cham. DoI:10.1007/978-3-030-27529-7_29.
  • (25) H. Lee, Y. Largman, P. Pham and A. Y. Ng, Unsupervised feature learning for audio classification using convolutional deep belief networks, in Proc. of the 22nd International Conference on Neural Information Processing Systems (NIPS’09), Vancouver, British Columbia, Canada, 07 - 10 Dec. 2009, pp. 1096-1104.
  • (26) [25] Y. LeCun, Y. Bengio and G. Hinton, Deep learning, Nature 521, 2015, pp. 436 - 444.
  • (27) [26] Y. Taigman, M. Yang, M. Ranzato and L. Wolf, Deepface: closing the gap to human-level performance in face verification. In Proc. Conference on Computer Vision and Pattern Recognition, 2014, pp. 1701–1708.
  • (28) [27] P. N. Druzhkov and V. D. Kustikova, A survey of deep learning methods and software tools for image classification and object detection, Pattern Recognition and Image Analysis, 26(1), pp. Jan. 2016, pp. 9–15
  • (29) [28] C. Szegedy, A. Toshev, and D. Erhan, Deep neural networks for object detection, in Proc. of NIPS (Lake Tahoe, 2013), pp. 2553–2561.
  • (30) [29] K. Simonyan, A. Vedaldi, and A. Zisserman, “Deep fisher networks for large-scale image classification,” in Proc. of NIPS (Lake Tahoe, 2013), pp. 163–171.
  • (31) [30] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” in Proc. of NIPS (Lake Tahoe, 2012), pp. 1097–1105.
  • (32) [31] Y. He, K. Kavukcuoglu, Y. Wang, A. Szlam, and Y. Qi, “Unsupervised feature learning by deep sparse coding,” in Proc. of SIAM International Conference on Data Mining (Philadelphia, 2014), pp. 902–910.
  • (33) [32] G. E. Dahl, Deep learning approaches to problems in speech recognition, computational chemistry, and natural language text processing, PhD thesis, University of Toronto, 2015.
  • (34) [33] Abdel-rahman Mohamed, George E. Dahl, and Geoffrey E. Hinton. Acoustic modeling using deep belief networks. Audio, Speech, and Language Processing, IEEE Transactions on, 20(1):14 –22, jan. 2012. ISSN 1558-7916. doi: 10.1109/TASL.2011.2109382.12
  • (35) [34] G. E. Dahl, Tara N. Sainath, and Geoffrey E. Hinton. Improving deep neural networks for LVCSR using rectified linear units and dropout. In ICASSP, IEEE, 2013, pp. 8609–8613.
  • (36)

    [35] D. Yu and L. Deng, Automatic Speech Recognition – A Deep Learning Approach, 2015, Springer London Heidelberg New York Dordrecht. DOI 10.1007/978-1-4471-5779-3.

  • (37) [36] R. Socher, A. Perelygin, J. Y. Wu, J. Chuang, C. D. Manning, A. Y. Ng and C. Potts, Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank, in Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, Washington, USA, October, 2013, pp. 1631-1642.
  • (38) [37] Z. Qin, and J. Lawry, “Decision tree learning with fuzzy labels”, Information Sciences, Vol. 172, pp. 91-129, 2005.
  • (39) [38] J. Lawry, “Appropriateness Measures: An Uncertainty Measure for Vague Concepts”, Synthese, Vol. 161, No. 2, pp. 255-269, March, 2008.
  • (40) [39] Lawry J (2006) Modeling and Reasoning with Vague Concepts, Kacprzyk, J. (Ed.), Springer, New York.
  • (41) [40] R. C. Jeffrey, The Logic of Decision, The University of Chicargo Press, New York. 1990.
  • (42) [41] G. J. Székely and M. L. Rizzo, Brownnian Distance Covariance, The Annals of Applied Statistics, 3(4), 2009, pp. 1236–1265. DOI:10.1214/09-AOAS312.
  • (43) [43] Quinlan JR (1986) Induction of Decision Trees, Machine Learning, 1:81-106. Kluwer Academic Publishers, Boston.
  • (44) [44] T. A. Almeida, J. M. G. Hidalgo, and A. Yamakami. Contributions to the study of sms spam filtering:new collection and results. In DocEng’11, Mountain View, California, USA, 19-22 September 2011.
  • (45) [45] H. He, A. Tiwari, J. Mehnen, T. Watson, C. Maple, Y. Jin, B. Gabrys, Incremental Information Gain Analysis of Input Attribute Impact on RBF-Kernel SVM Spam Detection, WCCI2016, Vancouver, Canada, 24-29 July, 2016.
  • (46) [46] D. Justus, J. Brennan, S. Bonner, A. S, McGough, Predicting the Computational Cost of Deep Learning Models, IEEE International Conference on Big Data, Seattle, WA, USA, 10-13 Dec. 2018. https://arxiv.org/pdf/1811.11880.pdf.28Nov.2018.
  • (47) T. Ching, D. S. Himmelstein, et al. Opportunities and obstacles for deep learning in biology and medicine, Journal of the Royal Society Interface, 15(141), Apr. 2018 DoI:10.1098/rsif.2017.0387
  • (48) X. Yuan, P. He, Q. Zhu and X. Li, Adversarial Examples: Attacks and Defenses for Deep Learning, IEEE Transactions on Neural Networks and Learning Systems, 30(9), Sept. 2019, pp. 2805 - 2824, DoI: 10.1109/TNNLS.2018.2886017.
  • (49) WEKA: The workbench for machine learning, accessed on 24 May 2020, https://www.cs.waikato.ac.nz/ml/weka/

Appendix

7.1 14 attribute-based LAHs for k = 2..10 (Figs. 6-12)

Figure 6: LAH based on SMS14 with = 2
Figure 7: LAH based on SMS14 with = 3
Figure 8: LAH based on SMS14 with = 4
Figure 9: LAH based on SMS14 with = 5
Figure 10: LAH based on SMS14 with = 6
Figure 11: LAH based on SMS14 with = 7 and = 8.
Figure 12: LAH based on SMS14 with = 9 and =10.

7.2 LAHs based on 20 attributes for k=1,..,10 (Figs. 13-20)

Figure 13: LAH for k=2.
Figure 14: LAH for k=3
Figure 15: LAH for k=4.
Figure 16: LAH for k=5.
Figure 17: LAH for k=6.
Figure 18: LAH for k=7.
Figure 19: LAH for k=8.
Figure 20: LAH for =9 and =10