Abstract
A machine learning method needs to adapt to over time changes in the environment. Such changes are known as concept drift. One approach to concept drift handling is by feeding the whole training data set once again into a learning machine for retraining. Another approach is by rebuilding an ensemble classifiers to adapt to a new training data set. In either approach, retraining or rebuilding classifiers are expensive and not practical. In this paper, we propose an enhancement of OnlineSequential Extreme Learning Machine (OSELM) and its variant Constructive Enhancement OSELM (CEOSELM) by adding an adaptive capability for classification and regression problem. The scheme is named as Adaptive OSELM (AOSELM). It is a single classifier scheme that works well to handle real drift, virtual drift, and both drifts occurred at the same time (hybrid drift). The AOSELM also works well for sudden drift as well as recurrent context change type. The scheme is a simple unified method implemented in simple lines of code. We evaluated AOSELM on regression and classification problem by using various public dataset widely used for concept drift verification from SEA and STAGGER; and other public datasets such as MNIST and USPS. Experiments show that our method gives higher kappa value compared to the multiclassifier ELM ensemble. Even though AOSELM in practice does not need hidden nodes increase, we address some issues related to the increasing of the hidden nodes such as error condition and rank values. We propose to take the rank of the pseudo inverse matrix as an indicator parameter to detect ’underfitting’ condition.
Keywords— adaptive, concept drift, extreme learning machine, online sequential.
Introduction
Data stream mining is a data mining technique, in which the trained model is updated whenever new data arrive. However, the trained model must work in dynamic environments, where a vast amount of data is not only continuously generated but also keep changing. This challenging issue is known as concept drift [12], in which the statistical properties of the input attributes and target classes shifted over time. Such shifts can make the trained model becoming less accurate. These methods pursue an accurate, simple, fast and flexible way to retain classification performance when the drift occurs. Ensemble classifier is a wellknown way to retain the classification performance. The combined decision of many single classifiers (mainly using ensemble members diversification) is more accurate than single classifier [9]. However, it has higher complexity when handling multiple (consecutive) concept drifts.
One of the popular machine learning methods is Extreme Learning Machine (ELM) introduced by Huang, et. al. [21][20][18], [17] [19]
. The ELM is a Single Layer Feedforward Neural Network (SLFN) with fast learning speed and good generalization capability.
In this paper, we focused on the learning adaptation method as an enhancement to Online Sequential Extreme Learning Machine (OSELM) [28] and Constructive Enhancement OSELM (CEOSELM) [26]. We named it as Adaptive OSELM (AOSELM). The AOSELM has capability to handle multiple concept drift problems either changes in the number of attributes (virtual drift/VD) or the number of target classes (real drift/RD) or both at the same time (hybrid drift/HD), also for recurrent context (all concepts occur alternately) or sudden drift (new concept substitutes previous concepts) [25]. Our scope of attribute changes discussed in this paper is on the feature space concatenation that widely used in data fusion, kernel fusion, and ensemble learning [8]
and not on the feature selection (irrelevant features removal) methods
[5]. We compared the performance with nonadaptive sequential ELM: OSELM and CEOSELM. We also compared the performance with ELM classifier ensembles as the common adaptive approach for concept drift solution. In the present study, although we focus on the adaptation aspect, we address some possible change detection mechanisms that are suitable for our method.A preliminary version of RD and its early results appeared in conference proceedings [3]. In this paper, we introduced the new scenarios in VD, HD, and consecutive drifts either recurrent or sudden drift scenarios as well as theoretical background explanation. Our main contributions in this research area can be summarized as follows:

We proposed simple adaptive method as enhancement to OSELM and CEOSELM for addressing concept drifts issue. Unlike ensemble systems [41, 17] that need to manage the complex combination of a vast number of classifiers, we pursue a single classifier for simple implementation while retaining comparable performance for handling multiple (consecutive) drifts.

We denote the training data from different concepts (sources or contexts), using the symbol for training data and for target data. We used the subscript font without parenthesis to show the source number.

We denote the drift event using the symbol , where the subscript font shows the drift type. E.g., the Concept 1 has virtual drift event to be replaced by Concept 2 (Sudden drift) : . The Concept 1 has real drift event to be replaced by Concept 1 and Concept 2 recurrently (Recurrent context) in the shuffled composition : .

We introduced a simple unified platform to handle a hybrid drift (HD) when changes in the number of attributes and the number of target classes occurred at the same time.

We elaborated how the AOSELM for transfer learning using hybrid drift strategy. Transfer learning focuses on extracting the knowledge from one or more source task domains and applies the knowledge to a different target task domain
[32]. Concept drift focuses on the timevarying domain with a small number of current data available. In contrast, transfer learn 
We denote the training data from different concepts (sources or contexts), using the symbol for training data and for target data. We used the subscript font without parenthesis to show the source number.

We denote the drift event using the symbol , where the subscript font shows the drift type. E.g., the Concept 1 has virtual drift event to be replaced by Concept 2 (Sudden drift) : . The Concept 1 has real drift event to be replaced by Concept 1 and Concept 2 recurrently (Recurrent context) in the shuffled composition : . ing is not associated with time and requires the entire training and testing data set [38]. The example of transfer learning by using HD strategy is the transition from different data set sources but still related and has the same purpose. In this paper, we discussed the transfer learning on numeric handwritten MNIST [27] to alphanumeric handwritten USPS [33] recognition.

Naturally, the AOSELM handling strategy was based on recurrent context. We devised an AOSELM strategy to handle sudden drift scenario by introducing output marginalization method. This method is also applicable for concept drift in a regression problem.

We studied the effect of increasing the number of hidden nodes, which is treated as one of learning parameters, to improve the accuracy (other learning parameters are input weight, bias, activation function, and regularization factor). We proposed the evaluation parameter to predict the accuracy before the training completed. We applied this assessment parameter actually to prevent ’underfitting’ or nonconvergence condition (the model does not fit the data well enough that makes accuracy performance dropped) when any learning parameter changes such as hidden nodes increased.
This paper is organized as follows. Section 1 explains some issues and challenges in concept drift, the background of ELM, and ELM in sequential learning. Section 2 presents the background theory and algorithm derivation of the proposed method. In section 3, we focus on the empirical experiments to prove the methods and research questions in regression and classification problem. We use artificial and real data set. The artificial data sets are streaming ensemble algorithm (SEA) [35] and STAGGER [23], which are commonly used as benchmark in sequential learning. The real data sets are handwritten recognition data: MNIST for numeric [27] and USPS for alphanumeric classes [33]. We studied the effect of hidden nodes increase as one of important learning parameter in section 3.5. Section 6 discusses research challenges and future directions. The conclusion presents some highlights in Section 7.
1 Related Works
1.1 Notations
We specify the notations used throughout this article for easier understanding:

Matrix is written in uppercase bold (e.g., ).

Vector is written in lowercase bold (e.g., ).

The transpose of a matrix is written as . The pseudoinverse of a matrix is written as .

, will be used as non linear differentiable function (activation function), e.g., sigmoid or tanh function.

The amount of training data is . Each input data contains some attributes. The target has number of classes.An input matrix can be denoted as and the target matrix as .

The hidden layer matrix is . The input weight matrix is . The output weight matrix is . The matrix is the additional block portion of the matrix . The matrix is the auto correlation matrix of . The inverse of matrix is .

can be denoted as . can be denoted as and can be denoted as . denotes the additional nodes number of .

When the number of training data , we employed the online sequential learning method by updating model every time each new training pairs are seen. is the subset of input data at time as the initialization stage. ,,…, are the subset of input data at the next sequential time. Each subset may have different number of quantity. The corresponding label data is presented as . We used the subscript font with parenthesis to show the sequence number.

We denote the training data from different concepts (sources or contexts), using the symbol for training data and for target data. We used the subscript font without parenthesis to show the source number.

We denote the drift event using the symbol , where the subscript font shows the drift type. E.g., the Concept 1 has virtual drift event to be replaced by Concept 2 (Sudden drift) : . The Concept 1 has real drift event to be replaced by Concept 1 and Concept 2 recurrently (Recurrent context) in the shuffled composition : .
1.2 Concept Drift Strategies
In this section, we briefly explained the various concept drift solution strategies. Gama, et. al. [12] explained many concept drift methods have been developed, but the terminologies are not well established. According to Gama, et. al., the basic concept drift based on Bayesian decision theory in the classification problem for class output and incoming data as:
(1) 
Concept drift occurred when has changed; e.g., , where and
are respectively the joint distribution at time
and . Gama, et. al. categorized the concept drift types as following:
Real Drift (RD) refers to changes in . The change in
may be caused by a change in the class boundary (the number of classes) or the class conditional probabilities (likelihood)
. The number of classes expanded and different class of data may come alternately, known as recurrent context. A drift, where a new conditional probabilities replaces the previous conditional probabilities while the number of class remained the same, is known as sudden drift. Other terms are concept shift or conditional change [13]. 
Virtual Drift (VD) refers to the changes in the distribution of the incoming data (e.g. changes). These changes may be due to incomplete or partial feature representation of the current data distribution. The trained model is built with additional data from the same environment without overlapping the true class boundaries. Other terms are feature change [13], temporary drift, or sampling shift.
Kuncheva [25, 24] explained the various configuration patterns of data sources over time as random noise, random trends (gradual changes), random substitutions (abrupt or sudden changes), and systematic trends (recurring context). The random noise will simply be filtered out. A gradual drift occurs when many concepts may reoccur alternately in the gradual stage for a certain period. A consecutive drift takes place when many previously active concepts might keep on changing alternately (recurring context) after some time. The sudden drift (abrupt changes or concept substitutions) is the type that at one time, one concept is suddenly replaced by another concept.
Žliobaitė [41] proposed a taxonomy of concept drift tackling methods as shown in Fig. 1. It describes the methods based on when the model is switched on (the ’when’ axis) and how the learners adapt to training set formation or design and parametrization of the base learner (The ’how’ axis). The ’when’ axis spans drift handling from trigger based to evolving based methods. The ’how’ axis spans drift handling from training set formation to model manipulation (or parametrization) methods.
Žliobaitė [41] explained that most attention on the concept drift tackling methods are drawn to multiclassifier model selection and fusion rules, but little attention on the model construction of base classifier.
Gama, et. al. [12]
proposed a complete online adaptive learning scheme that organized four modules: memory, change detection, learning, and loss estimation (See Fig.
2). These modular components can be integrated, permuted and combined with each other. The key modules are the learning and the change detection modules. Most methods focused on some subset or often mixtures of many types within certain concept drifts.The learning module refers to the methods for the adaptation strategies of the predictive model. The learning module is categorized based on i) How the model is updated when new data points are available (learning mode): retraining or incremental (online) modes. ii) The behavior of predictive models on timeevolving data (model adaptation): a blind (evolving or implicit) based module or an informed (trigger or explicit) based module. iii) The techniques for maintaining active predictive models (model management): a single model or ensemble model. The change detection module refers to drift detection. The change detection identifies change points or small time intervals when changes occur.
Each drift employed different solution strategies. The solution for RD is entirely different from VD. If the systematic changes are likely to reappear, we may want to keep past successful classifiers and simply reuse them. If the changes are gradual, we may use a moving window strategy on the training data. If the changes are abrupt, we can pause the existing static classifiers then retrain the classifier using the new training data. Thus, it is hard to combine simultaneously many strategies at one time to solve many types of concept drift in just a simple platform.
1.3 ELM in Sequential Learning
In this section, we briefly explained the previous related works of ELM in sequential learning and adaptive environments.
ELM is getting popularity thanks to its learning speed, generalization capability, and simplicity. Huang [18]
explained the term ’Extreme’ meant to move beyond conventional artificial neural network learning that required iterative tuning. The ELM moves toward brain like learning in which hidden neurons need not be tuned.
The output function of an SLFN with single hidden layer matrix can be presented as the function of:
(2) 
where is the pseudo inverse of .
can be approximated by left pseudoinverse of H as:
If we have from filled by the number of training data and incremental batch of data filled , the output weights are approximated by:
(4) 
Both and have a different number of training data but have the same number of hidden nodes.
If , then we can rewrite:
(5) 
The OSELM assumes no changes in the number of hidden nodes. However, increasing the number of hidden nodes is required to improve the performance. A CEOSELM [26] has addressed this problem by adding hidden nodes in the sequential learning stage. So .
The submatrix is set to a zero block matrix to simplify the computation in accordance with the fact that the previous data is not related to the new hidden nodes. The additional hidden nodes block matrix for data, has relation to the additional hidden nodes .
Then, we can rewrite with as:
(6) 
If can be solved using block matrix inversion and Schur complement, then:
(7) 
It is important to note that both OSELM and CEOSELM did not address the concept drift issue; e.g., when the number of attributes in or the number of classes in in data set has added. In this paper, we categorized OSELM and CEOSELM as nonadaptive sequential ELM.
To the best of our knowledge, no previous single base ELM approach specifically addresses many concept drifts learning [17]. However, some papers [36, 4] already discussed how the ELM implementation in adaptive environment.
Schaik, et.al. [36]
proposed Online Pseudo Inverse Update Method (OPIUM). OPIUM is based on Greville’s method as the incremental solutions to compute the pseudo inverse of matrix. The pseudo inverse computation can be solved incrementally as linear regression problems and can be adaptive which allows for non stationary data. The derivation of OPIUM is equivalent to the OSELM if the condition
met at each iteration. This condition implies is a linear combination of the previous hidden layer and the simpler derivation of (2) with right pseudo inverse become:(8) 
Schaik, et.al. defined as the cross correlation matrix between and () and as the inverse of the auto correlation (), so . According to Greville’s method, the solution for . And the solution for , or in short writing as .
Schaik, et.al. proposed a simplified version named OPIUM light by computing only the ondiagonal element of . Schaik, et.al. applied the OPIUM light for nonstationary data by using different weight in determining for the most recent pair that appropriate for nonstationary mapping, which are: , and for .
In our opinion, OPIUM only tackled the real drift case with discriminant function boundary shift in the streaming data (e.g. the frequency shift of sine wave). They implemented the weighting as a nonstationary mapping parameter between input and output vectors.
Cao, et. al. [4] proposed twophase classification algorithm: First, weighted ensemble classifier based on ELM (WECELM) algorithm, which can dynamically adjust classifier and the weight of training uncertain data to solve the problem of concept drift. Second, an uncertainty classifier based on ELM (UCELM) algorithm is designed for the classification of unknown data streams, which considers attribute (tuple) value and its uncertainty, thus improving the efficiency and accuracy. When concept drift occurs, WECELM will dynamically adjust the classifiers and the weight of training data, thus a new classifier will be added to the ensemble until it reached a preset maximum then removed the worstperforming classifier. UCELM is designed for the classification of uncertain data streams, which has attributes (tuples) and its uncertainty values. The UCELM evaluated uncertainty value for every newly arrived attribute and decided based on the probability of the new attributes belonging to each class, thus improving the efficiency and accuracy. In our opinion, WECELM is categorized as evolving based method by selecting the bestperforming classifier, and UCELM addressed virtual drift problem by using uncertainty attributes selection.
2 Proposed Method
2.1 Theoritical Background of AOSELM
In sequential learning, some partial training data arrives in time sequential fashion: . Learning is the process of constructing function to map between observation and its nature called (class) [7]. When the number of training data , we need to address the expected value of .
Learning from the data is the process to select a function from a class of by minimizing of the empirical squared error with the error probability of the resulting classifier. According to [7], the empirical squared error minimization is consistent under general conditions.
Theorem 2.1
Assume that is a totally bounded class of functions. If , then the classification rule obtained by minimizing the empirical squared error over is strongly consistent, that is,
(9) 
Based on Law of Large Numbers (LLN) theorem
[15] and Theorem 2.1, in sequential learning with the number of training data , we can make sure the consistency of expected value of learning model is .The Concept drift refers to an online supervised learning model when the relation between the input data and the target variable changes over time [12]. If the learning model from Concept 1 is bounded by hypothesis space and feature space . And the learning model from Concept 2 is bounded by hypothesis space and feature space . We defined the real drift as when the hypothesis space has changed to . We scoped the definition for dimension changes. The virtual drift is when the feature space has changed to . We scoped the definition for dimension changes.
To achieve the consistency of minimized square error in the new hypothesis space or new feature space, the learning model needs a transition map from the former space to the new space. The learning model needs a transition space before it converges to the new learning model
. Our transition space idea was inspired by geometric approach for solving many problems in the fields of pattern recognition and machine learning
[6, 34].For transition space, we propose two approaches: i) Assign the random coordinates in the new concept space. ii) Assign the equivalent projection coordinates in the new design space. The first approach is suitable for VD scenario, in which we assigned the new random coordinates as the new input weight parameters. The second approach is suitable for RD situation, by setting the equivalent projection coordinates in the new space (e.g. The in 1D coordinate has corresponding 2D projection coordinates as ).
Here, we relate the ELM theory to the context of AOSELM concept drift scenarios as follows:

Scenario 1: virtual drift (VD).
Huang, et. al. [17]
explained interpolation theory from ELM point of view as stated by the following description:
Theorem 2.2
Given any small positive value , any activation function which is infinitely differentiable in any interval, and arbitrary distinct samples , there exists such that for any input weight and bias pair randomly generated from any interval of
, according to any continuous probability distribution, then with probability one,
. Furthermore, if , then with probability one, .According to Theorem 2.2 and Learning Principle I of ELM Theory [18], the input weight and bias as hidden nodes parameters are independent of training samples and their learning environment through randomization. Their independence is not only in initial training but also in any sequential training stages. Thus, we can adjust the input weight and bias pair on any sequential stages and still make sure with probability one that .

Scenario 2: real drift (RD).
Huang, et. al. [17] explained universal approximation capability of ELM as described by the following theorem:
Theorem 2.3
Given any nonconstant piecewise continuous function , if span is dense in L2, for any continuous target function and any function sequence randomly generated according to any continuous sampling distribution, holds with probability one if the output weights
are determined by ordinary least square to minimize
.Based on Theorem 2.3 and inspired by the related works [26, 3], we devised the AOSELM real drift capability by modifying the output matrix with zero block matrix concatenation to change the size dimension of the matrix without changing the value. Zero block matrix has meant the previous has no knowledge about the new concept. ELM can approximate any complex decision boundary, as long as the output weights are determined by ordinary least square to keep the minimum.
2.2 AOSELM Algorithms
In this section, we presented the AOSELM pseudo codes ^{1}^{1}1The Matlab source code, data set and demo file implementation are available at https://github.com/abudiman250172/adaptiveOSELM. in the sequential with training input and target to update .
Basically, we have three pseudo codes, namely OSELMSeq (Algorithm 1) as OSELM and CEOSELM pseudo codes; the AOSELMVDSeq (Algorithm 1 as AOSELM pseudo codes tackling virtual drift; and AOSELMRDSeq (Algorithm 2 as AOSELM pseudo codes for addressing real drift. We can combine each pseudo code together to form a hybrid drift Algorithm. We can increase the hidden nodes using CEOSELM in Algorithm 3 after AOSELMVDSeq or AOSELMRDSeq. For initialization, basically we can use any ordinary ELM initialization in offline learning mode.
For sudden drift scenario, we proposed output marginalization method by adding the new output nodes when the new concept presented (See Fig. 3) and marginalized the output result by defining the class of concept S is . We scoped the new concept has the same output nodes quantity with the previous concept. Output marginalization is by shifting the ELM output to the output nodes that belonging to the new concept and ignoring the previous concept output nodes. This strategy is similar with classifier pruning in ELM ensemble. However, in output marginalization, we can reactivate the previous concepts by shifting back to the previous output nodes. If we want to forget the last concept totally, we can quickly delete the previous output nodes without impacting the generalization performance, or we can increase the hidden nodes at the same time with the drift event.
In regression, because we have only one output node, then we can employ sudden drift scenario by amplifying the related output node of the concept with a constant value that makes the maximum output approximated to 1.
The systematic rules make AOSELM more flexibe to handle complex consecutive drifts scenario. The AOSELM only stored the previous output weight and auto correlation . The auto correlation did not keep the training data. This makes AOSELM scalable for big streaming data without impacting the computation performance.
To improve the accuracy, we define the target values , so that class is . According to [22], the target values is equivalent with .
3 Experiments
3.1 Experiments Design in Classification
To verify our method, we designed some experiments with the following purposes:

To investigate the effectiveness of AOSELM on tackling three concept drift scenarios (VD, RD, HD) in two sequential patterns (sudden changes, recurring context). We used various data set starting with synthetic data set (SEA, STAGGER) then with real data set in handwritten recognition (MNIST, USPS). Each data set has different drift characteristics. This experiment is presented in Section 3.2 and 3.4. We also demonstrated the AOSELM capability as drift detection role in section 3.3 using SEA data set.

To investigate the effectiveness of AOSELM on transfer learning to combine different data set sources. This experiment is presented in Section 3.4 using two data set sources (MNIST, and USPS) in handwritten recognition problem.

To investigate the effect of hidden nodes increase in the drift events and how it impacts performance. This experiment is presented in section 3.5.
We used Matlab ™ running on Microsoft Windows ™ Computer with 4 cores 2.5 GHz processor and 8 GB memory.
Our experiments are organized as follows:

Simulation benchmark tests on the datasets that commonly used in concept drift handling of stream data, e.g. SEA [35] and STAGGER [23] (See Table (a)a). Both datasets are binary classification problem. SEA has 3 inputs with random integer values from 0 to 9. STAGGER has three inputs with multiple category values from 1 to 3 (Total inputs are 9). SEA and STAGGER are the examples of concept drift that caused by discriminant function changes while the number of attributes and classes from all concepts are still same. The change type is sudden drift. The expected result is the classifier has good performance for the newest concept [24].

We tested our algorithm with realworld public data sets from MNIST numeric (0 to 9) [27] and the USPS alphanumeric (A to Z, 0 to 9) handwritten dataset [33]. We used original greylevel image attributes [] of MNIST data set and the combination of [] with additional attributes from the 9x9 bins histogram of orientated gradients () of greylevel image features [29]. For USPS, we added more data with Gaussian random and saltpepper noises. Refer to Table (a)a for detail data set information.

We designed the initial input weights and bias based on robust OSELM with regularization scalar (ROS) [16]
and then based on initial random from the normal distribution (NORM). The activation function is sigmoid. The pseudo inverse function is the orthogonal projection using ridge regularization.

Let’s define the following concept as :

is MNIST class (16);

is MNIST class (710);

is MNIST class (16);

is MNIST class (710);

is MNIST class (110);

is USPS class (110, AZ);
We followed the simulated concept drift methods in Dries, et.al [10]. We simulated sudden drift by splitting the composition into two groups, e.g., and and recurring context by shuffled the composition of and . We set the sequential training flow to be the following drift equation:
We followed the simulated concept drift methods in Dries, et.al [10]. We simulated sudden drift by splitting the composition into two groups, e.g., and and recurring context by shuffled the composition of and . We set the sequential training flow to be the following drift equation:

Experiment 1  Virtual drift:
MNIST MNIST

Experiment 2  Real Drift:
For recurring context:
For sudden drift:

Experiment 3  Hybrid Drift:

Experiment 4  MNIST+USPS Transfer Learning:


We measured the performance based on Table (c)c. The testing accuracy and Cohen’s Kappa are to show the quantitative measurement. The predictive accuracy is to demonstrate the trend in a line chart. The sudden drift performance is based on the forgetting capability that compared the testing accuracy of the latest concept against all the previous concepts.

We compared the AOSELM performance with nonadaptive online sequential and offline version of ELM classifier. The performance expectation of sequential version classifier is to approximate the offline version of the classifier (desiderata for online classifiers [24]). We also compared with adaptive ELM ensemble method (See Fig. 4). We designed the hierarchical ensemble using two models of ELM classifier with different roles (See Fig. 4). The first role is a binary classifier that acts as a director based on one against all (OAA) classification. The binary classifier needs all sequential training data to be recalled (full memory). Another role is the data classifier. This ensemble requires total classifiers for concepts, thus not effective for consecutive concept drift case e.g. SEA concepts. The ensemble also applied outdated classifier pruning when the ensemble detects the previous attributes need to be replaced.




3.2 SEA and STAGGER Concepts Result
We addressed the question whether nonadaptive OSELM and CEOSELM with increase could handle the concept drift situation. We compared between AOSELM with no increase (AOSELM1) and with increase (AOSELM2). We used 5fold crossvalidation and compared between NORM and ROS parameter. For SEA, parameters: and increase per drift. For STAGGER, parameters: and hidden nodes increase per drift.
The AOSELM has better accuracy with better recovery time (See Table (a)a,(b)b) than CEOSELM, whereas nonadaptive OSELM fails (See Fig. 5). The AOSELM2 improved the forgetting capability better than AOSELM1. In comparison with Kolter, et.al
result using dynamically weighted majority (DWM) of naive Bayes (DWMNB) for SEA, AOSELM result is near to the DWM result. Comparison with inducing decision trees (DWMITI) for STAGGER
[23], AOSELM outperformed DWM. (See Table (a)a and (b)b).


3.3 Concept Drift Detection
The drift detection works based on loss estimation (See Fig. 2) that compared current prediction accuracy with the previous feedback. Using similar method on [31, 1], we can evaluate the intersection point between accuracy decrease and increase in Fig. 6. If the consecutive loss performance exceeded a certain threshold, then drift warning status triggered. We measured the output performance from the new concept output and compared with the previous output. If met certain criteria, then the new AOSELM is committed. Otherwise, the previous AOSELM is rolled back.
3.4 MNIST and MNIST+USPS Result
We measured the testing accuracy based on holdout test data by experiment trials. The results are as follows:

Experiment 1  Virtual drift.
The AOSELM of has Cohen’s kappa of testing accuracy 95.72 (0.21) % approximated to its nonadaptive ELM and offline ELM of version with the same hidden nodes number . It has better accuracy than single attribute [] or [] only (See Table (b)b). It proves our explanation in the theoretical background on Section 2.1.
Note: We set for [] ELM based on the same ratio between number of input nodes with hidden nodes of [] ELM.

Experiment 2  Real drift
The final result as showed in Table (b)b, the AOSELM has better Cohen’s kappa performance for all concepts than ELM ensemble and little exceed to its nonadaptive and offline ELM. (Table (b)b).
As well as in the split composition, the AOSELM with increase has better performance in forgetting capability than the AOSELM with no increase (See Table (b)b).

Experiment 3  Hybrid drift
The final result in Table (c)c, the AOSELM has better Cohen’s kappa performance for HD than ELM ensemble and approximate to its nonadaptive and offline ELM.

Experiment 4  MNIST+USPS Transfer Learning






3.5 The effect of hidden nodes increase
The initial size of hidden nodes selection is important to have good generalization performance. Some studies [21, 17] suggested for hidden node size to be equal at least to the rank value of training data. However, in a data stream, it is hard to determine a fixed number of hidden nodes following that suggestion. The larger requires more computation resources and processing time, and probably not giving a significant result at the end. Thus, we have a requirement to increase in sequential stage [26].
The experiment result in Table 6 shows the performance improved when certain hidden nodes size increase. We have different conditions: 2000, the rank of initial training data (666), and rank of total training data (713) and multiple conditions of on the drift event: No increase, 500, 1000 and 2000 using ROS parameters. However, the larger has better influence than increase.
Scenario  

VD  2000  95.92 (0.21)  96.37 (0.20)  96.83 (0.18)  96.89 (0.18) 
666  93.10 (0.27)  95.18 (0.23)  96.18 (0.20)  95.60 (0.22)  
713  93.30 (0.26)  95.31 (0.22)  96.28 (0.20)  96.09 (0.20)  
RD  2000  94.71 (0.24)  94.93 (0.23)  95.39 (0.22)  95.42 (0.22) 
630  91.3 (0.30)  91.67 (0.29)  93.61 (0.26)  94.04 (0.25)  
713  91.71 (0.29)  92.70 (0.28)  93.82 (0.25)  94.23 (0.25) 

’Underfitting’ condition.
’Underfitting’ is the condition when the model does not fit the data well enough that makes unconvergence. Based on an empirical experiment with increase in the sequential phase on Table 7, we investigated particular condition when the AOSELM classifier has a bad result. We realized the ELM performance is dependent upon finding general matrix inverse of . Based on orthogonal projection method in CEOSELM, we can employ the rank value of as evaluation parameter to detect ’underfitting’.
The is approximation to matrix . The full rank of is ideally equal with . However, certain condition in the sequential training, e.g. poor training data or poor learning parameter selection may cause the diagonal squared matrix has less diagonalizable [14], thus not full rank anymore.
In the sequential learning, we can compare the before and after hidden nodes increase. The expected result is positive increment. If the rank value becomes lower after hidden nodes increase, then it has a higher probability of ’underfitting’ condition to occur. The is determined by the block size of training data, the number of hidden nodes increment, the scalar selection in ROS parameter, the activation function, input weight, and bias random assignment method. In this experiment, we focused on the block size, the number of hidden nodes and scalar selection. Using as evaluation parameter is more efficient because we do not need to compute .
Scenario Batch Size Before After Pred. Acc. RD 630 500 1000 10 630 1130 0% 630 500 500 10 1130 1122 7% 630 500 100 10 1130 614 5% 630 500 10 10 640 640 0% 630 100 1000 10 630 730 0% 630 100 500 10 730 730 0% 630 100 100 10 730 730 0% 630 100 10 10 640 640 0% 2000 1000 100 5 2000 1868 3% 2000 1000 100 1 2000 1947 16% 2000 1000 100 0.5 2000 1946 17% 2000 1000 100 0.05 2000 2100 0% VD 666 500 500 5 666 1166 0% 666 500 100 5 666 1166 0% 666 100 500 5 666 766 0% 666 100 100 5 666 766 0% 666 50 500 5 666 716 0% 666 50 100 5 666 716 0% Table 7: Predictive accuracy performance in AOSELM for MNIST using different parameters and before and after increase. Each experiment is repeated trials to get the probability of predictive accuracy . 
Sudden drift.
On Table (a)a and (b)b, the hidden nodes increase can improve the forgetting capability on sudden drift (it reduced the accuracy of the outdated concept).
In CEOSELM, when increase in the same time with drift, it makes and the new concept target in split composition, while previous concept . Thus, in the process of finding become simplified because is partially trained by only and not by . Thus, it reduced the generalization capability of to recognize problem.
Data AOSELM Concept Testing Accuracy SEA 0 500 MNIST 0 500 (a) Sudden drift effect caused by split training composition with hidden nodes increase. Data Composition MNIST 0 Split Shuffled 500 Split Shuffled 1000 Split Shuffled (b) MNIST RD simulation: the effect of hidden nodes increase for split and shuffled training composition () . Table 8: Average Testing accuracy in % for RD experiment. For SEA, we started from to . For MNIST, we started from to .
4 AOSELM in Regression
We can use the similar real drift scenario with output marginalization and output amplification to solve concept drift problem in regression. In this experiment, we used AOSELM with single input node and single output node per concept. We defined the following concept as:

is sinc function with 50000 training/5000 testing;

is sinus function with 50000 training/5000 testing;

is gaussian function with 50000 training/5000 testing.
The sequential experiments are following drift equations :

Experiment 1 :

Experiment 2 :
We presented the result on the following figures to compare the performance of each concept at the end of each training experiment. Our objective is to show the AOSELM regression capability to keep the previous regression concept knowledge. We select the constant value that giving the best regression result of each concept. The AOSELM has , , and function. More drifts occurred will weaken the older concepts. Thus, it needs larger amplifier constant value.
5 Simulation in Big Data stream : Intrusion Detection System (IDS) KDD Cup 1999
IDS is a network security technology that scans any network packet traffic to detect any potential exploits then sending the alarm or taking some active action to Intrusion Prevention System. Some machine learning methods have been applied with the hope of improving detection rates and adaptive capability [37].
In this experiment, we used KDD Cup 1999 Competition data set. The full dataset had 4898431 network packets and grouped to be 23 classes (One Normal class and 22 attack names based on a signaturebased detection) [11]. The dataset has a control information (CI) header for delivering the data in numerical and multicategorical values as features. We focused on service names (IP ports) attributes because they are specific differentiators for applications. The CI and the number of attack classes are not stationary. We analyzed the data set for the growing of service names and the number of class attack in the whole dataset on Fig. 11. The challenge in IDS dataset is imbalanced data between the classes. The highest number of data is for ’normal’ class, and the lowest number is for ’spy’ class (only 2 packets). To simplify the experiment, we use oversampling by adding more data based on the random normal distribution of packet signatures and under sampling approaches by dropping some samples randomly.
Based on the growing of service names and the number of classes analysis (See Fig. 11), we designed one drift scenarios based on two concepts (Table (b)b). has ten classes, and 37 service names, and has 23 classes and 70 service names. Total training data for each concept is 920000 packets. No data repetition from the previous event, except at the end of sequential training. The composition between / on HD event is 230000/690000.
The validation data set of is selected from all packets from minority classes and randomly selected original majority classes (10422 packets). We used holdout method with trials. We used AOSELM1 for and AOSELM2 for (Other ELM parameters are same:, NORM, sig). The AOSELM result in this experiment can approximate the nonadaptive OSELM on (See Table (b)b).


6 Challenges and Future Research

We need to investigate the optimum transition space that minimize the gap to the new concept learning model. In certain case, the AOSELM may have the ’underfitting’ condition and require larger training data to achieve the new convergence.

We need to check the consistency of AOSELM for different pseudoinverse methods (E.g., Greville’s method [36]).
We think some ideas for AOSELM future researches:

The need for transfer learning to solve big data problem when the distribution data changes.

The AOSELM integration with another ELM methods, e.g., Weighted OSELM for imbalanced learning [30]
, ELM Autoencoder (ELMAE)
[39], Stacked ELM [40], etc. 
A detail systematic explanation based on rule extraction [2] for AOSELM in handling adaptive environment.
7 Conclusion
The proposed method gives better adaptive capability than non adaptive OSELM and CEOSELM in term of retaining the recognition performance when handling concept drifts. It uses a simple line of code and easy to deploy especially for consecutive drifts, compared with adaptive ensemble methods. While most adaptive classifiers work differently for each virtual, real drift, and hybrid drift scenarios, the AOSELM tackles those drifts through simple block matrix reconstruction and rank evaluation.
AOSELM satisfied the requirement criteria in term of accuracy, simplicity, fast and flexible. However, in certain VD and HD cases, the AOSELM accuracy may not exceed the non adaptive sequential ELM, which include the future training data. In RD cases, the AOSELM has better accuracy. In a real data implementation, the nonadaptive ELM is better and preferred when we know exactly the future behavior of data. However, we can not predict it precisely. We believe using larger training data, the AOSELM performance will approximate the expected value of nonadaptive sequential ELM or offline ELM, which use the future training data. The AOSELM can also add learning adaptation function to the previous offline learning model. It makes AOSELM an excellent choice for the unpredictable situation.
The AOSELM tackles sudden drift change type as well as recurrent context change type. The output marginalization strategy is implemented by simply shifting the output nodes that belonging to the latest concept. The AOSELM does need to increase the hidden nodes to improve the forgetting capability for sudden drift change type. To make sure the convergence to the expected learning model, we proposed the rank value of the pseudo inverse autocorrelation hidden nodes matrix as evaluation parameter to prevent ’underfitting’ condition that makes the accuracy performance dropped.
We can consider the AOSELM as another type of ELM ensemble formation using shared and interconnected hidden nodes between ensemble members. We can implement the AOSELM in similar fashion compared to the ELM ensemble for adaptive learning scheme, but with better performance, simpler and more resource efficient. However, the AOSELM does have some drawbacks. Any hidden node changes could impact all notions.
References
 1. C. Alippi, G. Boracchi, and M. Roveri. Justintime classifiers for recurrent concepts. Neural Networks and Learning Systems, IEEE Transactions on, 24(4):620–634, April 2013.

2.
N. H. Barakat and A. P. Bradley.
Rule extraction from support vector machines: A review.
Neurocomputing, 74(13):178–190, 2010.  3. A. Budiman, M. Fanany, and C. Basaruddin. Constructive, robust and adaptive oselm in human action recognition. In Industrial Automation, Information and Communications Technology (IAICT), 2014 International Conference on, pages 39–45, Aug 2014.
 4. K. Cao, G. Wang, D. Han, J. Ning, and X. Zhang. Classification of uncertain data streams based on extreme learning machine. Cognitive Computation, pages 1–11, 2014.
 5. G. Chandrashekar and F. Sahin. A survey on feature selection methods. Computers & Electrical Engineering, 40(1):16 – 28, 2014. 40thyear commemorative issue.
 6. L. C. A. Corsten. Statistical methods: The geometric approach (david j. saville and graham r. wood). SIAM Review, 34(3):506–508, 1992.
 7. L. Devroye, L. Györfi, and G. Lugosi. A Probabilistic Theory of Pattern Recognition. Springer, 1996.
 8. T. G. Dietterich. Ensemble methods in machine learning. In Proceedings of the First International Workshop on Multiple Classifier Systems, MCS ’00, pages 1–15, London, UK, UK, 2000. SpringerVerlag.
 9. P. Dongre and L. Malik. A review on real time data stream classification and adapting to various concept drift scenarios. In Advance Computing Conference (IACC), 2014 IEEE International, pages 533–537, Feb 2014.
 10. A. Dries and U. Rückert. Adaptive concept drift detection. Statistical Analysis and Data Mining, 2(56):311–327, Dec. 2009.
 11. J. Gama. Knowledge discovery from ubiquitous streams  datasets for concept drift.
 12. J. a. Gama, I. Žliobaitė, A. Bifet, M. Pechenizkiy, and A. Bouchachia. A survey on concept drift adaptation. ACM Comput. Surv., 46(4):44:1–44:37, Mar. 2014.

13.
J. Gao, W. Fan, J. Han, and P. S. Yu.
A general framework for mining conceptdrifting data streams with skewed distributions.
In In Proc. SDM’07, 2007.  14. G. H. Golub and C. F. Van Loan. Matrix Computations (3rd Ed.). Johns Hopkins University Press, Baltimore, MD, USA, 1996.
 15. C. M. Grinstead and J. L. Snell. Introduction to Probability. AMS, 2003.
 16. M.T. T. Hoang, H. T. Huynh, N. H. Vo, and Y. Won. A robust online sequential extreme learning machine. In Proceedings of the 4th International Symposium on Neural Networks: Advances in Neural Networks, pages 1077–1086, Berlin, Heidelberg, 2007. SpringerVerlag.
 17. G. Huang, G.B. Huang, S. Song, and K. You. Trends in extreme learning machines: A review. Neural Networks, 61(0):32 – 48, 2015.
 18. G.B. Huang. An insight into extreme learning machines: Random neurons, random features and kernels. Cognitive Computation, 6(3), 2014.
 19. G.B. Huang. What are extreme learning machines? filling the gap between frank rosenblatt’s dream and john von neumann’s puzzle. Cognitive Computation, 7:263–278, June 2015.
 20. G.B. Huang, H. Zhou, X. Ding, and R. Zhang. Extreme learning machine for regression and multiclass classification. IEEE Transactions on Systems, Man, and Cybernetics, Part B, 42(2):513–529, 2012.
 21. G.B. Huang, Q. Y. Zhu, and C. K. Siew. Extreme learning machine: theory and applications. Neurocomputing, 70(13):489–501, 2006.
 22. A. Iosifidis. Extreme learning machine based supervised subspace learning. Neurocomputing, 167:158 – 164, 2015.
 23. J. Z. Kolter and M. A. Maloof. Dynamic weighted majority: An ensemble method for drifting concepts. J. Mach. Learn. Res., 8:2755–2790, Dec. 2007.
 24. L. Kuncheva. Classifier ensembles for changing environments. In Multiple Classifier Systems, volume 3077 of Lecture Notes in Computer Science, pages 1–15. Springer Berlin Heidelberg, 2004.
 25. L. I. Kuncheva. Classifier ensembles for detecting concept change in streaming data: Overview and perspectives. In 2nd Workshop SUEMA 2008 (ECAI 2008), pages 5–10, 2008.
 26. Y. Lan, Y. C. Soh, and G.B. Huang. A constructive enhancement for online sequential extreme learning machine. In Neural Networks, 2009. IJCNN 2009. International Joint Conference on, pages 1708–1713, June 2009.
 27. Y. LeCun and C. Cortes. MNIST handwritten digit database, 2010.
 28. N.Y. Liang, G.B. Huang, P. Saratchandran, and N. Sundararajan. A fast and accurate online sequential learning algorithm for feedforward networks. Neural Networks, IEEE Transactions on, 17(6):1411–1423, Nov 2006.
 29. O. Ludwig, D. Delgado, V. Goncalves, and U. Nunes. Trainable classifierfusion schemes: An application to pedestrian detection. In Intelligent Transportation Systems, 2009. ITSC ’09. 12th International IEEE Conference on, pages 1–6, Oct 2009.
 30. B. Mirza, Z. Lin, and K.A. Toh. Weighted online sequential extreme learning machine for class imbalance learning. Neural Processing Letters, 38(3):465–486, 2013.
 31. K. Nishida and K. Yamauchi. Adaptive classifiersensemble system for tracking concept drift. In Machine Learning and Cybernetics, 2007 International Conference on, volume 6, pages 3607–3612, Aug 2007.
 32. S. J. Pan and Q. Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10):1345–1359, 2010.
 33. S. Roweis. Data for matlab hackers handwritten digits.
 34. R. Strack. Geometric Approach to Support Vector Machines Learning for Large Datasets. PhD thesis, Richmond, VA, USA, 2013. AAI3563551.
 35. W. N. Street and Y. Kim. A streaming ensemble algorithm (sea) for largescale classification. pages 377–382. ACM, 2001.
 36. A. van Schaik and J. Tapson. Online and adaptive pseudoinverse solutions for {ELM} weights. Neurocomputing, 149, Part A:233 – 238, 2015.
 37. S. K. Wagh, V. K. Pachghare, and S. R. Kolhe. Article: Survey on intrusion detection system using machine learning techniques. International Journal of Computer Applications, 78(16):30–37, September 2013.
 38. Q. Yang. Transfer learning beyond text classification. In Proceedings of the 1st Asian Conference on Machine Learning: Advances in Machine Learning, ACML ’09, pages 10–22, Berlin, Heidelberg, 2009. SpringerVerlag.
 39. N. Zhang, S. Ding, and Z. Shi. Denoising laplacian multilayer extreme learning machine. Neurocomputing, 171:1066 – 1074, 2016.
 40. H. Zhou, G.B. Huang, Z. Lin, H. Wang, and Y. C. Soh. Stacked extreme learning machines. Cybernetics, IEEE Transactions on, 45(9):2013–2025, 2015.
 41. I. Zliobaite. Learning under Concept Drift: an Overview. Computing Research Repository, abs/1010.4, 2010.
Comments
There are no comments yet.