With the advancement in technologies of online data mining, scalable and low cost data management, and distributed and cloud computing, data-based modeling for streaming data has gained tremendous popularity in industrial asset management. The use cases range from statistical process control to monitor machine operation condition for manufacturing systems [1, 2], early failure detection and diagnosis as prognostics and health management , detecting and tracking objects and abnormalities in video streams , and monitoring and forecasting application demand to dynamically improve resource allocation utility , just to name a few. In many of these scenarios, the relationship between the modeling target and the available observations are often characterized using various machine learning models.
Conventional machine learning models assume that the training data well represent the relationship in all data drawn from the process of interest. In other words, data stationarity is required over the course of model learning and application. However, this is commonly violated in real practice, especially for applications involving streaming analysis. In many industrial use cases, data recorded from manufacture and operation processes demonstrate an inherited nonstationary nature. For example, a sensor’s measurement can drift due to the sensor’s fault or aging, changes in operation conditions or control command, and machine degradation as a result of wearing. Even without seeing any drift in available data, there is no guarantee that data to become available in the future follow similar distribution as the past in a stream setting. For all these observed, unseen, or uncertain changes in data, or in the process, we call them concept drift as in the literature.
On account of these practical limitations, the modeling technologies can either be used under constraints, or have to be adapted to handle nonstationarity. The first point of view follows the thought process of dynamic systems modeling . It assesses the robustness of a system model by carrying out sensitivity analysis and uncertainty decomposition, which results in a set of feasible working conditions (states) where models are valid and stable. Once the actual system states do not meet those constraints, the model does not provide any analytic conclusions. Seeing this disadvantage, most recent researches take the latter perspective, and continuously cope with the drifting concept. The problem of concept drift learning can be stated as: sequentially given (labeled) measured data that may be nonstationary over time, how to construct a predictive model that 1) predicts in a timely manner, and 2) tracks and responds to meaningful changes in data, and adapt to valid system behavior.
The term concept drift roughly denotes the source to generate observations and targetfor , changes over time . The causes can be categorized as: change in data distribution , so that a fitted model that achieves optimal averaged prediction performance for one period of time (the training phase) is no longer optimal for the new test data; this is also called virtual drift, since the response for a same remains the same. The other type of drift refers to the change in the conditional distribution , describing the relationship between and . This is also called real drift, since response may change despite the same observation. Changes purely caused by seasonal effect and operational condition (e.g., temperature, humidity) are examples of virtual drift; while machine degradation belongs to real drift.
Depending on how model adapts to newer concepts, existing works can be generally classified as trigger-based and evolving methods. Thetrigger-based approaches explicitly perform concept change detection on input streams, the derived feature space, or model performance metric; various window based, or sequential hypothesis tests have been developed, aiming at an optimal detection rate and sample efficiency [7, 8, 9]. Once a drift is detected, the models are retrained or incrementally updated with valid training data from recent experience or through resampling [10, 11, 12, 13]. In contrast, the evolving approaches continuously update the model without an explicit goal to detect change. Many online learning models are evolving over time. The most studied methods with overall better performance utilize an ensemble of base models, different implementations are distinguished by the type of base models, training set composition methods, ensemble update rules, and the way model output is fused, [14, 15, 16, 17]. It is commonly viewed that the trigger-based approaches work well for abrupt changes, while evolving approaches can pick up gradual changes with less delay, more discussions can be found in recent survey papers on this topic , ,.
In addition to the challenges sharing across concept drift learning literature, the recent rapidly rising pull from the Industrial Internet of Things (IIOT) applications also set further constraints on computational resources: some analysis are performed in embedded systems with limited computing power and memory (edge analytics), and this sometimes makes typical ensemble learning technologies unsuitable.
Notations: Throughout this paper, we use regular lower case letters (e.g., , ) to denote single scalar observation or parameter, bold lower letters (e.g., ) to denote vectors or realization of random vectors, and upper case bold letters to denote matrices or random vectors (e.g., , ). The subscripts means time stamp (e.g., , ).
In this paper, we propose a light-weight, easy-to-implement alternating learner framework that continuously adapt to new concept, and provide high quality predictions. In particular, we employ two models: a long-memory learner (L) that is trained on a long time window (LW) with samples relevant to the current concept, and a short-memory (S) learner that is trained on a short window (SW) that only contains most recent samples. The intuition is: when the concept is stationary, L performs no worse than S, since the training data of S is a subset of L, and data distribution is homogeneous; however, on occurrence of concept drift, S adapts faster since the percentage of newer samples in SW is typically much higher than LW. This gives us the rule of alternating: As a new batch of measurements and corresponding labels arrive, S and L are individually tested on the new samples. If performance of L is acceptable, then frontier of LW moves forward, and L is updated using information in the fresh samples. If L does not meet the performance target but S does, this is an indication of concept change. In this case, we truncate the tail of LW to be the same as SW, so all information prior to SW is retired. On the other hand, the fixed-sized SW window always slides forward with most updated information. As a result, we alternatively choose prediction from L and S. And S acts like a clutch to determine the boundary of each concept (LW), and serves as a (vague) change detector as well as providing a baseline after change.
designed a new learner comparison criterion that better suits the regression case,
modified the learner alternating policy to prevent frequent reset under noisy environment, and
included data quality check to mitigate overfitting effect due to small sample size followed by learner reset.
In , extensive comparative studies have been done on the paired learner (PL) approach against most popular trigger-based and ensemble algorithms in classification setting, and PL has outperformed or shown comparable performance with much less computational costs. So in this paper, we focus on comparison with the PL algorithm and two other baseline models. Through a set of empirical studies, we show that the alternating learner approach successfully responses to both abrupt and gradual changes with different pattens, and provides the overall best prediction performance across comparison, and also achieves common industrial standard.
The remainder of the paper is organized as follows: in Section II, we describe the alternating learner algorithm in detail, and propose an implementation using the online extreme learning machine as the base model; we then perform a case study on real power plant performance modeling and present the results in Section III; we close the paper with conclusion in Section IV.
Ii-a Alternating Learners Framework
We consider a typical streaming analysis scenario: a set of labeled data (henceforth referred to as the initial dataset) are available in the beginning of learning process, after that, new measurements with labels become available one by one or in small batches (not necessary to have equal batch sizes). In industry practice, the initial dataset is usually from design of experiments (DoE), testbed runs or historical operation records. Sometimes they already contain a rich variety of concepts, but more often this is not sufficient due to limited duration and operation modes covered by the initial data.
The alternating learners (AL) framework consists of two learning models, L for long-memory learner and S for short-memory learner, an alternating policy is employed to choose the final prediction output either from L or S. Like many online sequential learning algorithms, the AL framework has two phases: initial learning phase and continuous learning phase.
In the initial learning phase, predictive models are built from the initial dataset. Without further assumption, this is a conventional (static) model training task. We initialize L trained by all initial dataset, and for S, at most samples from the latter portion (SW) are used, where is a user defined size for the short memory window. Note that, only samples in SW will be kept in memory. We choose online learning algorithms to incrementally update L afterwards (except those reset instances), so there is no need to save an increasing volume of training data. The base model for L discussed in this paper is the online sequential extreme-learning machine (OSELM), which we will introduce in subsection II-B.
In the continuous learning phase, learning is driven by update, compare, and select the two models using the new available data, and concepts. The process is illustrated in Figure 1. Model S is always refit to the short window SW, serving as a baseline local prediction. Model L continuously integrates new information by including new samples into its long-memory, , where is the time since when new concept has not been observed. In case of concept change, outdated data must be discarded from training set by resetting , and this is done by comparing predictive performance of L and S, on the new data.
Upon arrival of a new batch of samples , where and , the two learners both make prediction on the labels (response values). Let and be the prediction and Err and Err be the corresponding performance of and . The performance measure can be any error metric of interest, assuming the smaller the better, common choices can be the Mean Squared Error (MSE) or the Mean Absolute Percent Error (MAPE). The comparison outcome can be one of the following four cases:
Err is acceptable, Err Err: there is no sign of concept drift, old data () is sufficient to train L to achieve the desired accuracy, performance of S is likely to be restricted by small training set;
Err is acceptable, Err Err: although the local model shows a better performance on the new batch, since L still meets the prediction objective, this does not count towards a sign of concept drift. Moreover, in this case, we let L augment the training knowledge and pick up variability in system behavior;
Err is unacceptable, Err Err: predictions of both S and L are off. This may related to concept change, but a better substitution for L is not available;
Err is unacceptable, Err Err: local model outperforms long memory model in a meaningful way, this may indicate the distribution of pairs in the short window and the long window differs significantly, and model is closer to the upcoming concept.
From all four cases, only the last one both indicates a change and suggests a meaningful alternative for L. In other cases either performance of L is satisfactory (Case 1 and 2), or a better alternative is not available (Case 3). So we use the occurrence of Case 4 to design a “forgetting rule” for L.
In particular, we use a queue, Q, with a maximum length to record the status of performance comparison: register to Q whenever Case 4 happens, otherwise register . The concept change is asserted when the rate of seeing is high in the Q, one criterion is the percentage of exceeds a user defined threshold , which related to forgetting sensitivity. When a new concept is asserted, we shrink window LW to match SW by setting , and also reset L by S, so that the performance of L after reset is not arbitrarily bad (warm restart). The flowchart of the algorithm is provide in Figure 2.
Overall, L represents a long and relative stable concept, and it aims at improving prediction accuracy by expanding training set. And S serves as a change detector, and also the alternative model to be exerted from an ensemble of two models. When concept is stable, L typically outperform S, and in the ensemble notion, we choose to output the prediction of L solely. When concept drift happen gently, L slowly catches up by online update. If change is rapid or the magnitude of the change is large, online update fails to follow since a large portion of training data are still associated to the old concept, then learners switch and reset mechanism determines the beginning of new concept.
The user-specified reset threshold , the maximum length of the queue , and the acceptable performance threshold are domain dependent and can be pre-learned. In general, is not too small, small value results in frequent reset, which leads to overall small training set size, and harms the overall prediction performance. We provide sensitivity study on and in Section III.
To prevent the alternating mechanism from overly reaction to noise, we also introduce a leastWait parameter , so that reset is only allowed after the length of exceeds . In practice, is determined by reset threshold and sensitivity evaluation. For example, when , selecting essentially calls for a learner reset after seeing Case 4 one out of two checks after a previous reset. Frequent reset is undesired, especially when the learner accuracy fluctuation is caused by noise, since the information used to train the learners is almost as short as SW. In our sensitivity experiments, we find for mid-range (between and ), performance does not change much with respect to as long as it is larger than .
We assume equal batch size for the ease of description, this framework also works for varying batch size, as long as it is compatible with the online updating algorithm. The OSELM algorithm we use for this study is able to handle varying batch size.
The actual models to implement L and S are not restrictive: any model that can be updated in an online fashion (efficiently updated and does not store an increasing amount of data) will suffice.
Ii-B Base model selection: OS-ELM
We adopt the Extreme Learning Machine (ELM) as the base learners for L and S. The ELM is a special type of feed-forward neural networks introduced by Huang, et al.
. Unlike in traditional feed-forward neural networks where training the network involves finding all connection weights and bias, in ELM, connections between input and hidden neurons arerandomly generated and fixed. This implies for any given inputs, the hidden neuron outputs are uniquely determined; this mapping and the values are referred to as ELM feature mapping and ELM features. Thus, training an ELM reduces to finding connection weights from the ELM features to output neurons only. Because of such special design of the network, ELM training becomes very fast. Numerous empirical studies and recently some analytical studies have shown that ELM has better generalization performance than other machine learning algorithms including SVMs and is efficient and effective for both classification and regression [20, 21].
Consider a set of training samples, . Assume the number of hidden neurons is . The output function of ELM for generalized SLFNs is
where is the output of hidden neuron with respect to the input , and are the neuron’s weights and bias respectively; is a nonlinear piecewise continuous function satisfying ELM universal approximation capability theorems 
, for example, the sigmoid function;is the output weight matrix between hidden neuron and the output nodes. The is the random feature map transforming a -dimensional input to a -dimensional random ELM feature.
For batch ELM where all training samples are available at the same time, the output weight vector
can be estimated as the least-squares solution of, that is,
where is the Moore-Penrose generalized inverse of the hidden layer output matrix (see  for details), which can be calculated through orthogonal projection method: when has full column rank.
The online sequential ELM (OSELM), proposed by Liang, et al. , is a variant of classical ELM, it addresses the need of incrementally learning the output weights from chunks of data available in sequence. As described in details in, OS-ELM involves two learning phases, initial training and sequential learning.
Initial training: choose a small chunk of initial training samples, , where , from the given training samples; and calculate the initial output weight matrix, , using the batch ELM formula (2). And initialize
Sequential learning: for the training sample, where , iterate between the following two steps.
Calculate the partial hidden layer output matrix for the new sample:
and set the target .
Calculate the output weight matrix:
• Note that the number of hidden node and the random feature mapping is fixed after the initial training, only the ELM feature matrix is re-calculated for new data chunks, and the output weight matrix is updated accordingly.
The OSELM algorithm enjoys the benefits of fast training and incrementally updating, which fulfills our needs to maintain the long-memory learner L. For given (fixed) input-hidden layer weights , the iteratively calculated output connection weight matrix by (3) and (4) is the same as the batch-mode solution when all training samples are fed at once. The structure of the SLFN, or the number of hidden neurons K, is determined by performing a -fold cross validation in initial learning phase. This structure parameter, , once determined, remains the same for the online updating process.
The short-memory learner, S, on the other hand, is repeatedly trained by batch mode ELM using all samples in SW, and can be served as the starting point of OSELM after switching. Note that, except the samples in SW, old data are not saved, so L is not completely retrained.
When solving for the output weights from SLFN, if the number of sample points is less than number of hidden neurons, Equation (2) become under-determined, and the fitted network is easily overfit. So in implementation, we maintain two versions of L: learns complicated concepts trained and updated by OSELM;
learns simple hypothesis, and we choose a linear regression model based on the same input features. We add anoverfit test for , checking if the training samples in LW is more than , we then switch the output of L to point to either or . The detailed implementation steps are provided in Algorithm 1.
Here we make a note that, the comparison aims to conclude relative performance of the concept-drift handling frameworks, either triggering policies or ensemble rules, rather than the selection of base learner. Our base learner selection only considers the suitability of online update and efficient training/application.
Iii Experimental Results
We perform empirical performance assessment for the alternating learner algorithm in the regression setting. To demonstrate on an industrial use case that is typically suffered by the issue of concept drift, we use simulated data representing a power gas turbine operation with synthesized deterioration and maintenance work. The synthesized drift information we used in data generation are treated as ground truth. Comparison against three baselines are provided.
Algorithms to compare with: The conventional static machine learning model is implemented with the basic regularized ELM trained on the initial batch and tested on the remaining time series, this model misses the target right after concept change as expected. The OSELM is an example of typical online learning methods, under nonstationary environment, online learning gradually picks up new relationship but tends to have long lag, since the sensitivity and stability trade-off is controlled globally. The third algorithm we implemented is the Paired Learner algorithm proposed in , the original algorithm is designed for classification, we re-implemented two versions for regression with slightly different queue registration criteria. Only results with better performance is reported in this paper.
In multi-variate time series regression for power plant applications, we choose the Mean Absolute Percent Error (MAPE) of the model prediction as the evaluation metric, i.e., for thebatch, , where is the batch size. Note that, we assume the true response values () of every new batch is available after and predict.
Iii-a Case study: gas turbine performance modeling for power plant
In this case study, we consider to model the performance of a gas turbine in a power plant. The data are time series sensor readings from a power plant. A collection of signals related to turbine operating status are simultaneously recorded: compressor inlet temperature, compressor inlet humidity, ambient pressure, inlet pressure drop, exhaust pressure drop, inlet guide vane angle, fuel temperature, compressor flow, and controller calculated firing temperature.
The generated power depends on the machine condition, and is comprehensively captured by a hidden variable compressor efficiency . The recorded sensor data and predefined compressor efficiency profile are fed into the GE internal power simulation tool, Gas Turbine Performance (GTP) to get the instantaneous gross electric power and net heat rate. The GTP simulator has been independently developed by GE Power, it has high fidelity, and requires large computational resources to configure and run.
In real plant, the efficiency drops as regular wear, dust cumulation, coating material oxidation and some non-ideal usage condition. On an opposite side, maintenance activities such as cleaning and renovation improves machine status, and significantly increases after these events. In order to investigate the algorithm performance under different types of concept drift. We generate different change patterns of and simulate turbine response respectively. Our learning target is to predict generated power using measured signal only.
To obtain statistical description of the algorithm performance, we generate datasets, each set is a sequence of samples with inputs and outputs introduced above. We generate two change patterns of :
Abrupt change: this pattern contains two sudden jumps on the compressor efficiency . is set to be in the beginning, and linearly drops to in sample range, then suddenly jumps to . Then in sample range, slowly drops to , and again jumps back to . After that, stays around for sample range, then begins to decrease, and finally ends at in the end of simulation. The duration of each section, , and are randomly generated positive integers satisfying .
Gradual change: this pattern contains one relatively long linear change of . In this case, begins at value and stays near constant for a duration of samples. Then linearly decreases to in sample range, after that it stays near . Similarly, the random integers satisfies .
• For both cases, independent and identically distributed Gaussian noise is added to every simulated efficiency profile and lasts for the entire time steps. The dataset contains cases with abrupt change and with gradual change.
Iii-B Performance summary
We select the initial dataset with samples, and perform a repeated -fold cross validation on each time series to select the best hidden node number for its SLFN.
The three user defined parameters are , maximum queue length to compare learner performance; , the learner switching threshold, and , the least queue length allows a reset. As commented in Section II-A, and are closely related, once the threshold is given, we can choose properly, so we first determine the and jointly by a preliminary sensitivity experiments: on a subset of randomly selected dataset from all the ( abrupt cases and gradual cases). The prediction MAPE distribution on the sampled datasets are displayed in Figure 3, the Box-and-Whisker plots indicate the average, median, and the th and
th quantiles and extreme value ranges (end of vertical lines). It is seen that all combination show good MAPE score fromto , with an overall increasing trend with respect to and . We choose , and . The reason not selecting or is that, in order to avoid “single instance reset” described in Section II-A, selecting smaller leads to , which is about of the queue length, and we decide to trade this toward response speed.
Examples of the algorithm prediction results are shown in Figure 4 (gradual) and Figure 5 (abrupt). Assume data comes one sample at a time, so batch size , batches are available during the initial learning. We take industrial standard and set the performance acceptance limit . The figures show results when selecting the maximum queue capacity as batches, and reset trigger to be . On the top figures, real power output value is plotted in red, and the green stars are the alternating learner’s prediction. The bottom plot depicts MAPE of prediction on each prediction batch (sample). Blue vertical line indicates the beginning of continuous learning phase, and the black lines indicate the time instance to reset long-memory learner. It is seen that, reset usually occurs when the MAPE score shows an increasing trend. It happens less frequently for gradual changes, and can be caused by either slow concept drift, or local fluctuations (Figure 4). The algorithm also responses quickly after abrupt changes, and the prediction performance returns to small MAPE range () after the reset (Figure 5).
We run the AL algorithm on all of the simulated dataset and compare the predicted power with the power output from the GTP simulator and calculate the MAPE measure for each series. Then the same data are modeled by the three other methods (batch-ELM, OSELM, and PL 111Only the version with the Q registration criterion that leads to a better performance is reported in comparison.). The key statistics for the resulting distribution of the prediction MAPE metric over the set is summarized in Table I. Figure 6 and Figure 7 visually depict the performance comparison of different algorithms under abrupt changes and gradual changes. It is observed that, the framework with dual learners to detect change achieves significantly better prediction accuracy: the average MAPE for PL and AL are both below
and meet industrial standard with standard error of about, while the average MAPE for the batch-ELM is about and for the OSELM is about .
The poor performance of the batch-ELM can be understood as the concept change exceeds the model generalization capability, in other words, the model predicts with the initial concept for the entire data stream. The OSELM adjusts the output weights using (3), but the contribution of each new concept data is small comparing with the historical learned information, so it needs more iterations to settle the model. The methods with two learners apply a reset rule so that more aggressively adapt to the up-to-date concept.
The modification of AL upon PL makes it works better on regression setting, we provide a set of rescaled Box plots in Figure 8 to better illustrate the improvement.
We proposed an efficient online machine learning framework to model nonstationary data streams with concept drift. By training and maintaining two learners with long and short memory, and compare their prediction performance on new coming batches, the two-model ensemble effectively identifies data or relationship changes. By alternating ensemble prediction output between the two learners, the model continuously adapts to new concept. We performed a numerical algorithm evaluation using simulated power plant operation data. After extensive tests on various change pattens, the proposed algorithm outperforms other baseline and benchmark methods and consistently meets the industrial standard for power performance prediction.
-  D. C. Montgomery, Introduction to statistical quality control. John Wiley & Sons, 2007.
-  J. F. MacGregor and T. Kourti, “Statistical process control of multivariate processes,” Control Engineering Practice, vol. 3, no. 3, pp. 403–414, 1995.
-  B. S. Parker and L. Khan, “Detecting and tracking concept class drift and emergence in non-stationary fast data streams.” in AAAI, 2015, pp. 2908–2913.
-  H. Ghanbari, C. Barna, M. Litoiu, M. Woodside, T. Zheng, J. Wong, and G. Iszlai, “Tracking adaptive performance models using dynamic clustering of user classes,” in ACM SIGSOFT Software Engineering Notes, vol. 36, no. 5. ACM, 2011, pp. 179–188.
-  L. Ljung, System identification. Wiley Online Library, 1999.
-  I. Zliobaite, “Learning under concept drift: an overview,” Overview”, Technical report, Vilnius University, 2009 techniques, related areas, applications Subjects: Artificial Intelligence, Tech. Rep., 2009. [Online]. Available: http://arxiv.org/abs/1010.4784v1
-  C. Alippi and M. Roveri, “Just-in-time adaptive classifiers—part i: Detecting nonstationary changes,” IEEE Transactions on Neural Networks, vol. 19, no. 7, pp. 1145–1153, 2008.
-  H. Wang and Z. Abraham, “Concept drift detection for streaming data,” in 2015 International Joint Conference on Neural Networks (IJCNN). IEEE, 2015, pp. 1–9.
R. Klinkenberg and T. Joachims, “Detecting concept drift with support vector machines.” inICML, 2000, pp. 487–494.
-  M. Harel, S. Mannor, R. El-Yaniv, and K. Crammer, “Concept drift detection through resampling.” in ICML, 2014, pp. 1009–1017.
-  S. H. Bach and M. A. Maloof, “Paired learners for concept drift,” in 2008 Eighth IEEE International Conference on Data Mining, Dec 2008, pp. 23–32.
-  Y. Yang, X. Wu, and X. Zhu, “Mining in anticipation for concept change: Proactive-reactive prediction in data streams,” Data mining and knowledge discovery, vol. 13, no. 3, pp. 261–289, 2006.
-  W. Fan, “Systematic data selection to mine concept-drifting data streams,” in Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ser. KDD ’04. New York, NY, USA: ACM, 2004, pp. 128–137. [Online]. Available: http://doi.acm.org/10.1145/1014052.1014069
-  S. G. Soares and R. Araújo, “A dynamic and on-line ensemble regression for changing environments,” Expert Systems with Applications, vol. 42, no. 6, pp. 2935–2948, 2015.
I. Katakis, G. Tsoumakas, and I. Vlahavas, “An ensemble of classifiers for
coping with recurring contexts in data streams,” in
Proceedings of the 2008 Conference on ECAI 2008: 18th European Conference on Artificial Intelligence. Amsterdam, The Netherlands, The Netherlands: IOS Press, 2008, pp. 763–764. [Online]. Available: http://dl.acm.org/citation.cfm?id=1567281.1567458
-  J. Z. Kolter and M. A. Maloof, “Dynamic weighted majority: An ensemble method for drifting concepts,” J. Mach. Learn. Res., vol. 8, pp. 2755–2790, Dec. 2007. [Online]. Available: http://dl.acm.org/citation.cfm?id=1314498.1390333
-  W. N. Street and Y. Kim, “A streaming ensemble algorithm (sea) for large-scale classification,” in Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ser. KDD ’01. New York, NY, USA: ACM, 2001, pp. 377–382. [Online]. Available: http://doi.acm.org/10.1145/502512.502568
-  G. Ditzler, M. Roveri, C. Alippi, and R. Polikar, “Learning in nonstationary environments: a survey,” IEEE Computational Intelligence Magazine, vol. 10, no. 4, pp. 12–25, 2015.
-  J. a. Gama, I. Žliobaitė, A. Bifet, M. Pechenizkiy, and A. Bouchachia, “A survey on concept drift adaptation,” ACM Comput. Surv., vol. 46, no. 4, pp. 44:1–44:37, Mar. 2014. [Online]. Available: http://doi.acm.org/10.1145/2523813
-  G.-B. Huang, Q.-Y. Zhu, and C.-K. Siew, “Extreme learning machine: theory and applications,” Neurocomputing, vol. 70, no. 1, pp. 489–501, 2006.
-  G.-B. Huang, “An insight into extreme learning machines: random neurons, random features and kernels,” Cognitive Computation, vol. 6, no. 3, pp. 376–390, 2014.
-  N.-Y. Liang, G.-B. Huang, P. Saratchandran, and N. Sundararajan, “A fast and accurate online sequential learning algorithm for feedforward networks,” IEEE Transactions on Neural networks, vol. 17, no. 6, pp. 1411–1423, 2006.