The analysis of functional data is becoming more and more important in many areas of application such as medicine, economics, or geology (cf. Ullah and Finch (2013), Wang et al. (2016)), where this type of data occurs naturally. In industry, functional data are often a by-product of continuous monitoring of production processes, yielding great potential for data mining tasks. A common type of functional data are time series, as time series can often be considered as discretized functions over time.
Many researchers publish software implementations of their algorithms, therefore simplifying the access to already established methods. Even though such a readily available, broad range of methods to choose from is desirable in general, it also makes it harder for non-expert users to decide which method to apply to a problem at hand and to figure out how to optimize their performance. As a result, there is an increasing demand for automated model selection and parameter tuning.
Furthermore, the functionality of available pipeline steps ranges from simple data structures for functional data, to feature extraction methods and packages offering direct modeling procedures for regression and classification. Users are again faced with a multiplicity of software implementations to choose from and, in many instances, combining several implementations may be required. This can be difficult and time-consuming, since the various implementations utilize a multiplicity of different workflows which the user needs to become familiar with and synchronize in order to correctly carry out the desired analysis.
There is a wide variety of packages for functional data analysis in R R Core Team (2014) available that provide functionality for analyzing functional data. Examples range from the fda (Ramsay et al., 2018) package which includes object types for functional data and allows for smoothing and simple regression, to, e.g., boosted additive regression models for functional data in FDboost (Brockhaus and Ruegamer, 2018). For an extensive overview, see the CRAN task view (Scheipl, 2018).
Many of those packages are designed to provide algorithmic solutions for one specific problem, and each of them requires the user to become familiar with its user interface. Some of the packages, however, such as fda.usc (Febrero-Bande and Oviedo de la Fuente, 2012) or refund (Goldsmith et al., 2018) are not designed for only one specific analysis task, but combine several approaches. Nevertheless, these packages do not offer unified frameworks or consistent user interfaces for their various methods, and most of the packages can still only be applied separately.
A crucial advantage of providing several algorithms in one package with a unified and principled user interface is that it becomes much easier to compare the provided methods with the intention to find the best solution for a problem at hand. But to determine the best alternative, one still has to be able to compare the methods at their best performance on the considered data, which requires hyperparameter search and, more preferably, efficient tuning methods.
While the different underlying packages are often difficult and sometimes even impossible to extend to new methods, custom implementations and extensions can be easily included in the accompanying software.
We want to stress that the focus of this paper does not lie in proposing new algorithms for functional data analysis. Its added value lies in a large comparison of algorithms while providing a unified and easily accessible interface for combining statistical methods for functional data with the broad range of functions provided by mlr, most importantly benchmarking and tuning. Additionally, the often overlooked possibility of extracting non-functional features from functional data is integrated, which enables the user to apply classical machine learning algorithms such as support vector machines (Cortes and Vapnik, 1995) to functional data problems.
Can functional data problems be solved with classical machine learning methods ignoring the functional structure of the data as well as with more elaborate methods designed for this type of data? Bagnall et al. (2017a) measure the performance of some non-functional-data-specific algorithms such as the rotation forest (Rodriguez et al., 2006), but this does not yield a complete picture.
Guidance on the wide range of available algorithms is often hard to obtain. We aim to make some recommendations in order to simplify the choice of learning algorithm.
Do statistical methods explicitly tailored to the analysis of functional data (e.g. FDboost, Brockhaus and Ruegamer, 2018) perform well on classical time series tasks? No benchmark results for these methods, which provide interpretable results, are currently available.
Many methods that represent functional data in a non-functional domain have been proposed and are also often applied in practice. Examples for this include either hand crafted features (cf. Stachl and Bühner, 2015), summary statistics (Hyndman et al., 2018), or generally applicable methods such as wavelet decomposition (Aldrich, 2013).
Hyperparameter optimization is a very important step in many machine learning applications. In our benchmark, we aim to quantify the impact of hyperparameter optimization for a set of given algorithms on several data sets.
As contributions of this paper, we aim to answer the questions posed above. Additionally, we provide a toolbox for the analysis of functional data. It implements several methods for feature extraction and directly modeling functional data, including a thorough benchmark of those algorithms. This toolbox also allows for full or partial replication of the conducted benchmark comparison.
2 Related Work
In the remainder of the paper, we focus on comparing algorithms from the functional data analysis and the machine learning domain. Functional data analysis traditionally values interpretable results and valid statistical inference over prediction quality. Therefore functional data algorithms are often not compared with respect to their predictive performance in literature. We aim to close this gap. On the other hand, machine learning algorithms often do not yield interpretable results. While we consider both aspects to be important, we want to focus on predictive performance in this paper.
2.1 Feature extraction and classical machine learning methods
In this work, we differentiate between machine learning algorithms that can directly be applied to functional data, and algorithms intended for scalar features, which we call classical machine learning methods.
A popular approach when dealing with functional data is to reduce the problem to a non-functional task by extracting relevant non-functional features (Ullah and Finch, 2013). Applying classical machine learning methods after extracting meaningful features can then lead to competitive results (cf. Goldsmith and Scheipl, 2014, e.g.)
or at least provide baselines, which are in general not covered by functional data frameworks. In our framework, such functionality is easily available by combining feature extraction, e.g., based on extracting heuristic properties(cf. tsfeatures; Hyndman et al., 2018) or wavelet coefficients (Mallat, 1989; Aldrich, 2013) and analyzing these derived scalar features with classical machine learning tools provided by mlr.
Based on some existing functionality of the listed packages, we adapt different feature extraction methods. Along with different algorithms already proposed in literature, we propose two new custom methods, DTWKernel and MultiResFeatures:
tsfeatures (Hyndman et al., 2018) extracts scalar features, such as auto-correlation functions, entropy and other heuristics from a time series.
wavelets (Aldrich, 2013) applies a discrete wavelet transform to time series or functional data, e.g., with Haar or Daubechies wavelets. The extracted features are wavelet coefficients at several resolution levels.
projects the data on their principal component vectors. Only a subset of the principal component scores representing a given proportion of signal variance is retained.
DTWKernel computes the dynamic time warping distances of functional or time series data to (a set of) reference data. We implement dynamic time warping (DTW) based feature extraction. This method computes the dynamic time warping distance of each observed function to a (user-specified) set of reference curves. The distances of each observation to the reference curves is then extracted as a vector-valued feature. The reference curves can either be supplied by the user, e.g., they could be several typical functions for the respective classes, or they can be obtained from the training data. In order to compute dynamic time warping distances, we use a fast dynamic time warping (Rakthanmanon et al., 2012) implementation from package rucrdtw (Boersch-Supan, 2016).
MultiResFeatures extracts features, such as the mean at different levels of resolution (zoom-in steps). Inspired by the image pyramid and wavelet methods, we implement a feature extraction method, multi-resolution feature extraction where we extract features like mean and variance computed over specified windows of varying widths. Starting from the full sequence, the sequence is repeatedly divided into smaller pieces, where at each resolution level, a scalar value is extracted. All extracted features are concatenated to form the final feature vector.
2.2 Methods for functional data
Without feature extraction, direct functional data modeling (both classification and regression) methods incorporated in our package span two families: The first family of semi-parametric approaches includes FGAM (Goldsmith et al., 2018), FDboost (Brockhaus and Ruegamer, 2018), and the functional generalized linear model (FGLM; Ramsay, 2006)
, which are all structured additive models. Those methods use (tensor product) spline basis functions or functional principal components (FPCs)(Srivastava and Klassen, 2016)
to represent effects fitted in a generalized additive model. While FGAM and FGLM use the iterated weighted least square (IWLS) method to generate maximum likelihood estimates, FDboost uses component-wise gradient boosting(Hothorn et al., 2010) to optimize the parameters. Additionally, the estimated parameters can be penalized. A general formula for this family of methods is , where
represents a functional of the conditional response distribution (e.g., an expectation or a quantile),is a vector of (functional) covariates and are partial additive effects of subsets of in basis function representation, cf. Greven and Scheipl (2017) for a general introduction.
The second family of methods are non-parametric methods as introduced in Ferraty and Vieu (2006), e.g., based on (semi-)metrics which quantify local or global differences or distances across curves. For example, the distance between two instances could be defined by the distance of two curves . Kernel functions are used to average over the training instances and weigh their respective contributions based on the value of their distance semi-metric to the predicted instance. Functional -nearest neighbors algorithms can also be defined based on such semi-metrics. Implementations can be found in packages fda.usc (Febrero-Bande and Oviedo de la Fuente, 2012) and classiFunc (Maierhofer and Pfisterer, 2018).
2.3 Toolboxes for functional data analysis
The package fda (Ramsay et al., 2018) contains several object types for functional data and allows for smoothing and regression for functional data. Analogously, the R-package fda.usc (Febrero-Bande and Oviedo de la Fuente, 2012) contains several classification algorithms that can be used with functional data. In Python, scikit-fda (Grupo de Aprendizaje Automatico - Universidad Autonoma de Madrid, 2019) offers both representation of and (pre-)processing methods for functional data, but only a very small set of machine learning methods for classification or regression problems is implemented at the time of writing.
As a byproduct of the Time-Series Classification Bake-off Bagnall et al. (2017a), a wide variety of algorithms were implemented and made available. But this implementation emphasizes the benchmark over providing a data analysis toolbox for users, and is therefore not easily usable for inexperienced users.
The recently published benchmark analysis Time-Series Classification Bake-off by Bagnall et al. (2017a) provides an overview of the performance of state-of-the-art algorithms for time series classification. They re-implement (in Java) and compare 18 algorithms designed especially for time series classification on 85 benchmark time series data sets from Bagnall et al. (2017b). In their analysis, they also include results from several standard machine learning algorithms. They note that the rotation forest (Rodriguez et al., 2006) and random forest (Breiman, 2001) are competitive with their time series classification baseline (1-nearest neighbor with dynamic time warping distance; Tormene et al., 2008). Their results show that ensemble methods such as collection of transformation ensembles (COTE; Bagnall et al., 2015) perform best, but for the price of considerable runtime.
Deep learning methods applied to time series classification tasks have also shown competitive prediction power. For example, Fawaz et al. (2019) provide a comprehensive review of state-of-the-art methods. The authors compared both generative models and discriminative models, including and echo state networks, whereas only discriminative end-to-end approaches were incorporated in the benchmark study.
The benchmark study conducted in this work does not aim to replicate or compete with earlier studies like (Bagnall et al., 2017a), but instead tries to extend their results.
3 Functional Data
In contrast to non-functional data analysis, where the measurement of a single observation is a vector of scalar components whose entries represent values of the separate multidimensional features, functional data analysis treats and analyses the features themselves as functions over their domain. By learning to represent the underlying function, the carried out analysis is not just restricted to the measured discrete values but it is possible to sample from (and analyze) the entire domain space.
In this work, we focus on pairs of features and corresponding labels for supervised learning. In contrast to non-functional data analysis, where the measurement of a single observation is a vector of scalar components, functional features are function-valued over their domain. The features can thus also be a function, i.e., , . In practice, functional data comes in the form of observed values , where each corresponds to a discrete point on the continuum. Those observed values stem from an underlying function evaluated over a set of points. A frequent type of functional data is time series data, i.e., measurements of a process measured at discrete time-points.
For example, in some electrical engineering applications, signals are obtained over time at a certain sampling rate, but other domains are possible as well. Spectroscopic data, for example, are functional data recorded over certain parts of the electromagnetic spectrum. One such example is depicted in Figure 1. It shows spectroscopy data of fossil fuels (Fuchs et al., 2015) where the measured signal represents reflected energies in the ultraviolet-visible (UV-VIS) and the near infrared spectrum (NIR). In the plot, different colors correspond to different instances. This is a typical example of a scalar-on-function regression problem, where the inputs are a collection of spectroscopic curves for a fuel, and the prediction target is the heating value of the fossil fuel.
In Figure 2, we display two functional classification scenarios. The goal in those scenarios is to distinguish the class type of the curve, which can also be understood as a function-on-scalar problem. Figure 1(a) shows the vertical position of an actor’s hand while either drawing a toy-gun and aiming at a target, or just imitating the motion with the blank hand. This position is measured over time. The two different types of classes of the curves can be distinguished by the color scheme.
Figure 1(b) shows a data set built for distinguishing images of beetles from images of flies based on their outlines. While following the outline, the distance to the center of the object is measured which is then used for classification purposes. The latter data sets are available from Bagnall et al. (2017b).
4 Functional Data Analysis with mlrFDA
Along with the benchmark, we implement the software mlrFDA, which extends the popular machine learning framework mlr for the analysis of functional data. As the implemented functionality is an extension of the mlr package, all of the functionality available in mlr transfers to the newly added methods for functional data analysis. We include a brief overview of the implemented functionality in A.1. A more in-detail overview and tutorial on mlr can be found in the mlr tutorial (Schiffner et al., 2016).
mlr provides a unified framework for machine learning methods in R, currently supporting tasks from
main problem types: (multilabel-)classification, regression, cluster analysis, and survival analysis. For each problem type, many algorithms (calledlearners
) are integrated. This yields an extensive set of modeling options with a unified, simple interface. Moreover, advanced techniques such as hyperparameter tuning, preprocessing and feature selection are also part of the package. An additional focus lies on extensibility, allowing the user to integrate their own algorithms, performance measures and preprocessing methods. AsmlrMBO seamlessly integrates into the new software, many different tuning procedures can be readily adapted by the user. Tuning of hyperparameters is usually not integrated in software packages for functional data analysis and thus would require the user to write additional, non-trivial code that handles (nested) resampling, evaluation and optimization methods.
mlrFDA contains several functional data algorithms from several R packages, e.g., fda.usc, refund or FDboost. The algorithms’ functionality, however, remains unchanged, only their user interface is standardized for use with mlr. For detailed insights into the respective algorithms, full documentation is available in the respective packages.
Since our toolbox is built on mlr’s extensible class system, our framework is easily extensible to other methods that have not yet been integrated, and the user can include his or her own methods which do not necessarily need to be available as a packaged implementation. Additionally, mlrFDA inherits mlr’s functionality for performance evaluation and benchmarking, along with extensive and advanced (hyperparameter) tuning. This makes our platform very attractive for evaluating which algorithm fits best to a problem at hand, and even allows for large benchmark studies.
5 Benchmark Experiment
In order to enable a comparison of the different approaches, an extensive benchmark study is conducted. This paper does not aim to replicate or reproduce results obtained by Bagnall et al. (2017a) or Fawaz et al. (2019). Instead we focus on providing a benchmark complementary to previous benchmarks. This is done because the experiments require large amounts of computational resources, and
the added value of an exact replication of the experiments (with open source code) is comparatively small. Nonetheless, we aim for results that can be compared, and thus extend the results obtained byBagnall et al. (2017a) by staying close to their setup. The experiments were carried out on a high performance computing cluster, supported by the Leibniz Rechenzentrum Munich. Individual runs were allowed up to GB of RAM and hours run-time for each evaluation. We want to stress that this benchmark compares implementations, which does not always necessarily correspond to the performance of the corresponding theoretical algorithm. Additionally, methods for functional data analysis are traditionally more focused on valid statistical inference and interpretable results, which does not necessarily coincide with high predictive performance.
5.1 Benchmark Setup
|Data sets||51 Data sets, see table 7|
|Machine Learning:||- glmnet (glmnet)|
|- rpart (rpart)|
|- ksvm (kernlab)|
|- ranger (ranger)|
|- xgboost (xgboost)|
|- classif.glm (fda.usc)|
|- classif.np (fda.usc)|
|- FDboost (FDboost)|
|- fgam (refund)|
|- knn with dtw (classiFunc)|
|Feature Extraction + ML||- feature extraction: see table 6|
|- in combination with ML algorithms marked with a .|
|Measures||mean misclassification error, training time|
|Resampling||20-fold stratified sub-sampling;|
|class balances and train/test set size as in Bagnall et al. (2017a).|
|Tuning||100 iterations of Bayesian optimization (3-fold inner CV).|
|Corresponding hyperparameter-ranges can be obtained|
|from tables 3 and 5.|
A benchmark experiment is defined by four important characteristics: The data sets algorithms are tested on, the algorithms to be evaluated, the measures used for evaluating predictive performance, and a resampling strategy used for generating train and test splits of the data. A comprehensive overview of the conducted benchmark setup can be obtained from Table 1.
These characteristics are briefly described subsequently before providing and discussing the results. We use a subset of 51 data sets from the popular UCR archive (Bagnall et al., 2017b) in order to enable a comparison of results in Bagnall et al. (2017a) with the additional methods described in this paper. The data sets stem from various application types such as ECG measurements, sensor data, or image outlines, therefore having varying training set sizes or measurement lengths. For more detailed information about the data sets, see Bagnall et al. (2017b).
We selected data using the following criteria: In order to reduce the computational resources we did not run data sets that have multiple versions, exclude data sets with less then 3 examples in each class remove data sets with more than instances or time series longer than
measurements. As some of the classifiers only work with multi-class targets via1-vs-all classification, we additionally excluded data sets with more then 40 classes. In essence, we benchmark small and medium sized data sets with a moderate amount of different classes.
We add new algorithms and feature extraction methods which can be combined with arbitrary machine learning methods for scalar features (c.f. Table 1). Additionally we test classical machine learning methods, in order to obtain a broader perspective on expected performance if the functional nature of the data is ignored. As we benchmark default settings as well as tuned algorithms, in total different algorithms are evaluated across all data sets. When combining feature extraction and machine learning methods, we fuse the learning algorithm and the preprocessing, thus treating them as a pipeline where data is internally transformed before applying the learner. This allows us to jointly tune the hyperparameters of learning algorithm and preprocessing method. The respective defaults and parameter ranges can be obtained from Table 3 (feature extractors) and Table 5 (learning algorithms). More detailed description of the hyperparameters can be obtained from the respective packages documentation.
In order to generate train/test splits, and thus obtain an unbiased estimate of the algorithm’s performance, we use stratified sub-sampling. We use 20 different train/test splits for each data set in order to reduce variance and report the average. For tuned models, we use use nested cross-validation(Bischl et al., 2012) to ensure unbiased estimates, where the outer loop is again subsampling with 20 splits, and the inner resampling for tuning is a 3-fold (stratified) cross-validation. All compared 80 algorithms are presented exactly the same index sets for the 20 train-test outer subsampling splits.
Mean misclassification error (MMCE) is chosen as a measure of predictive performance in order to stay consistent with Bagnall et al. (2017a)
. Other measures, such as area under the curve (AUC) require predicted probabilities and do not trivially extend to multi-class settings.
While Bagnall et al. (2017a) tune all algorithms across a carefully handcrafted grid, we use Bayesian optimization (Snoek et al., 2012). In order to stay comparable, we analogously fix the amount of tuning iterations to 100.
We use mlrMBO (Bischl et al., 2017) in order to perform Bayesian optimization of the hyperparameters of the respective algorithm. Additionally, in order to scale the method to a larger amount of data sets and machines, the R-package batchtools (Bischl et al. (2015), Lang et al. (2017)) is used. This enables running benchmark experiments on high-performance clusters. For the benchmark experiment, a job is defined as re-sampling of a single algorithm (or tuning thereof) on a single version of a data set. This allows for parallelization to an arbitrary number of CPU’s, while at the same time guaranteeing reproducibility. The code for the benchmark is available from https://github.com/compstat-lmu/2019_fda_benchmark for reproducibility.
This Section tries to answer the questions posed in section 1. We evaluate various machine learning algorithms in combination with feature extraction, classical time series classification approaches, the effect of tuning hyperparameters for several methods, and try to give recommendations with respect to which algorithm(s) to choose for new classification problems.
Algorithms evaluated in this benchmark have been divided into three groups: Algorithms specifically tailored to functional data, classical machine learning algorithms without feature extraction and classical machine learning algorithms in combination with feature extraction.
5.2.1 Algorithms for functional data
Performances of algorithms specifically tailored to functional data analysis can be obtained from Figure 3. The k-nearest neighbors algorithm from package classiFunc (Maierhofer and Pfisterer, 2018) in combination with dynamic time warping (Rakthanmanon et al., 2012) distance seems to perform best across data sets. It is also considered a strong baseline in Bagnall et al. (2017a).
5.2.2 Machine Learning algorithms with feature extraction
Performances of different machine learning algorithms in combination with feature extraction with and without tuning can be obtained from Figure 4.
We conclude that feature extraction using splines (bsignal) and wavelets as well as extracting dynamic time warping distances works well when combined with conventional machine learning algorithms, even at their default hyper-parameters. Among the learners, random forests, especially in combination with bsignal show quite advantageous performance. In addition, we find an obvious improvement from hyper-parameter tuning for the Fourier feature extraction. In terms of learners, random forest and gradient boosted tree learners (xgboost) perform better than support vector machines.
5.2.3 Machine Learning algorithms without feature extraction
Additionally, we aim to provide some insight with regards to the performance of machine learning algorithms that ignore the functional nature of our data. Figure 5 provides an overview over the performance of different machine learning algorithms that are often used together with traditional tabular data. Performances in this figure are obtained from algorithms directly applied to the functional data without any additional feature extraction. The widely used gradient boosting (xgboost) and random forest (ranger) implementations seem to work reasonably well for functional data even without additional feature extraction.
5.2.4 The effect of tuning hyperparameters
From our experiments, we conclude, that tuning hyperparameters of machine learning algorithms in general has a non-negligible effect on the performance. Using Bayesian optimization in order to tune algorithm hyperparameters on average yielded an absolute increase in accuracy of 5.4% across data sets and learners.
Figure 6 displays the aggregated time over all data sets, taking into account the time required for hyperparameter tuning. All experiments have been run on equivalent hardware on high-performance computing infrastructure. Due to fluctuations in server load, this does not allow for an exact comparison with respect to computation time, but we hope to achieve comparable results as we repeatedly evaluate on sub-samples. Note that we restrict the tuning to algorithms where tuning traditionally leads to higher performances.333Additionally, we find significantly improved performance for tuned FDboost in Figure 3
5.2.5 Top 10 Algorithms and recommendations
Table 2 showcases the top algorithms from the benchmark in terms of average rank in predictive accuracy across data sets. With this list, we aim to provide some initial understanding of the performance of different algorithms and feature extraction methods. Note that this list by no means reflects performance on future data sets, but might serve as an indicator, of which algorithms one might want to try first given computational constraints.
|Algorithm Setting||Accuracy %||Average Rank|
We observe that wavelet extraction in combination with either ranger or xgboost seems to be very strong. They obtain an average rank of 12.90 and 14.45 (out of 80) respectively. Dynamic time warping distances for k-nearest neighbors indeed seems to be a strong baseline, even without tuning. Another strong feature extraction method seems to be the extraction of B-spline features. Using the algorithms above allows us to obtain an accuracy within of the maximum on of the data sets.
If the only criterion for model selection is predictive performance, (tuned) machine learning models in combination with feature extraction is a competitive baseline. This class of methods achieves within 95% of the optimal performance on out of data sets, while they include the best performing classifier in cases.
5.2.6 Comparison to classical time series classification
Even though the main purpose of this paper is not a direct comparison with the results from Bagnall et al. (2017a), we can use our results to show that applying functional data approaches and classical machine learning approaches together with feature extraction can still improve classification accuracy compared to current state-of-the-art time series classification methods.
In the experiments we conducted, the methods described in this paper improved accuracy on 9 out of the 51 data sets which is displayed in Figure 7. The 9 data sets and the corresponding best learner are displayed in Table 4. For each data set, only the best reached accuracy for both sets of algorithms is displayed.
Additionally, we evaluate how our learners rank in comparison to the individual bake-off algorithms from Bagnall et al. (2017a). The algorithm which performs best on a data set obtains the rank 1. The mean rank of the individual learners over all data sets (we take the intersection of the data sets from our benchmark and the ones from Bagnall et al. (2017a)). The average sorted ranks for the top 50% algorithms are displayed in Figure 8. We observe that the ensemble methods get the top ranks, which is no surprise, as for instance the COTE algorithm Bagnall et al. (2015) internally combines several classifiers from 4 different time series domains.
However, compared to the classical time series algorithms from Bagnall et al. (2017a) with the ensemble methods removed, our functional data algorithms obtain an overall good rank in accuracy performance, interleaved with the algorithms from Bagnall et al. (2017a). Note that the benchmarks are not exactly comparable due to minor differences in the benchmark setup, and we instead only include their reported results.
6 Summary and Outlook
In this work, we provide a benchmark along with a software implementation that integrates the functionality of a diverse set of R-packages into a single user interface and API. Both contributions come with a multiplicity of benefits:
The user is not required to learn and deal with the vast complexity of the different interfaces the underlying packages expose.
All of the existing functionality (e.g., preprocessing, resampling, performance measures, tuning, parallelization) of the mlr ecosystem can now be used in conjunction with already existing algorithms for functional data.
We expose functionality that allows us to work with functional data using traditional machine learning methods via feature extraction methods.
Integration of additional preprocessing methods or models is (fairly) trivial and automatically benefits from the full mlr ecosystem.
In order to obtain a broader overview of the performance of the integrated methods, we perform a large benchmark study. This allows users to get an initial overview of potential performances of the different algorithms. Specifically,
We open up new perspectives for time series classification tasks by incorporating methods from functional data analysis, as well as feature transformations combined with conventional machine learning models.
Based on the large scale benchmark, we conclude that many learners have competitive performance (Figure 7) and additionally offer the interpretability of many functional data analysis methods. Our toolbox serves as a strong complement and alternative to other time series classification software.
The presented benchmark study uses state-of-the-art Bayesian optimization for hyperparameter optimization, which results in significant improvements over models that are not tuned. This kind of hyperparameter tuning is easy to do with mlrFDA. Tuning, albeit heavily influencing performance is often not investigated. Our benchmark closes this gap in existing literature.
We find that extracting vector valued features and feeding them to a conventional machine learning model can often form competitive learners.
The pareto-optimal set in terms of performance on each data set contains different algorithmfeature-extraction combinations. Our toolbox offers the same API for all methods and allows to automatically search over this space, and thus allows users to obtain optimal models without knowing all underlying methods.
Concerning the questions we proposed at the beginning of the paper, we draw the following conclusions:
Tuning only a subset of the presented learners and feature extractions, i.e., the methods listed in Table 2, is sufficient to achieve good performances on almost all data sets in our benchmark.
A simple random forest without any preprocessing can also be a reasonable baseline for time series data. It achieves an average rank of (top 4) in our benchmark.
Most algorithms for functional data (e.g., FDboost) do not perform well in our benchmark study. As those algorithms are fully interpretable and offer statistically valid coefficients, they can still be useful in some applications, and should thus not be ruled out.
Feature extraction techniques, such as b-spline representations (bsignal) and wavelet extraction work well in conjunction with machine learning techniques for vector valued features such as xgboost and random forest.
Tuning leads to an average reduction in absolute MMCE of (ranger), (xgboost), (ksvm) (across feature extraction techniques) and (FDboost). This holds for all feature extraction techniques, where improvements range from multires to fourier.
In future work we will continue to expand the available toolbox along with a benchmark of new methods, and provide the R community a wider range of methods that can be used for the analysis of functional data. This includes not only integrating many already available packages, and as a result to enable preprocessing operations such as smoothing (e.g., fda (Ramsay et al., 2018)) and alignment (e.g., fdasrvf (Tucker, 2016) or tidyfun (Scheipl and Goldsmith, 2019)
), but also to explore and integrate advanced imputation methods for functional data. Further work will also extend the current implementation to support data that is measured on unequal or irregular grids. Additionally, we aim to implement some of the current state-of-the art machine learning models from the time series classification bake-off(Bagnall et al., 2017a), such as the Collective of Transformation-Based Ensembles (COTE) (Bagnall et al., 2015). This enables researchers to use and compare with current state-of-the-art methods.
- Wavelets: a package of functions for computing wavelet filters, wavelet transforms and multiresolution analyses. Note: R package version 0.3-0 External Links: Cited by: §A.2, item 4, §2.1, §2.1.
- Time-series classification with cote: the collective of transformation-based ensembles. IEEE Transactions on Knowledge and Data Engineering 27 (9), pp. 2522–2535. External Links: Cited by: §2.4, §6.
- The great time series classification bake off: a review and experimental evaluation of recent algorithmic advances. Data Mining and Knowledge Discovery 31 (3), pp. 606–660. Cited by: item 1, §1, §2.3, §2.4, §2.4, Figure 7, Figure 8, §5.1, §5.1, §5.1, §5.2.1, §5.2.6, §5.2.6, §5.2.6, Table 1, §5, §6.
- Time-series classification with cote: the collective of transformation-based ensembles. IEEE Transactions on Knowledge and Data Engineering 27 (9), pp. 2522–2535. Cited by: §5.2.6.
- The UEA & UCR time series classification repository. Note: www.timeseriesclassification.com Cited by: §2.4, §3, §5.1.
Resampling methods for meta-model validation with recommendations for evolutionary computation. Evolutionary Computation 20 (2), pp. 249–275. Note: PMID: 22339368 External Links: Cited by: §5.1.
- BatchJobs and BatchExperiments: abstraction mechanisms for using R in batch environments. Journal of Statistical Software 64 (11), pp. 1–25. External Links: Cited by: §5.1.
- mlrMBO: A Modular Framework for Model-Based Optimization of Expensive Black-Box Functions. External Links: Cited by: §5.1.
- rucrdtw: fast time series subsequence search in r. The Journal of Open Source Software 1, pp. 1–2. External Links: Cited by: §A.2, §2.1.
- Random forests. Mach. Learn. 45 (1), pp. 5–32. External Links: Cited by: §2.4.
- The fast fourier transform. IEEE Spectrum 4 (12), pp. 63–70. External Links: Cited by: §2.1.
- FDboost: boosting functional regression models. Cited by: item 3, §1, §2.1, §2.2.
- Support-vector networks. Mach. Learn. 20 (3), pp. 273–297. External Links: Cited by: §1.
- Deep learning for time series classification: a review. Data Mining and Knowledge Discovery, pp. 1–47. Cited by: §1, §2.4, §5.
- Statistical computing in functional data analysis: the r package fda.usc. Journal of Statistical Software 51 (4), pp. 1–28. External Links: Cited by: §1, §2.2, §2.3.
- Nonparametric functional data analysis: theory and practice. Springer Science & Business Media. Cited by: §2.2.
- Penalized scalar-on-functions regression with interaction term. Computational Statistics & Data Analysis 81, pp. 38–51. External Links: Cited by: Figure 1, §3.
- Refund: regression with functional data. Note: R package version 0.1-17 External Links: Cited by: §1, §2.2.
- Estimator selection and combination in scalar-on-function regression. Computational Statistics & Data Analysis 70, pp. 362–372. Cited by: §2.1.
- A general framework for functional regression modelling. Statistical Modelling 17 (1-2), pp. 1–35. Cited by: §2.2.
- Scikit-fda: functional data analysis in python. External Links: Cited by: §2.3.
- Classification of time series by shapelet transformation. Data Min. Knowl. Discov. 28 (4), pp. 851–881. External Links: Cited by: Figure 2.
- Model-based boosting 2.0. Journal of Machine Learning Research 11 (Aug), pp. 2109–2113. Cited by: §2.2.
- Tsfeatures: time series feature extraction. Note: R package version 0.1 External Links: Cited by: item 4, §2.1, §2.1.
- Driver sleepiness detection system based on eye movements variables. Advances in Mechanical Engineering 5 (), pp. 648431. External Links: Cited by: §A.2.
- Introduction to functional data analysis. CRC Press. Cited by: §3.
- Batchtools: tools for r to work on batch systems. The Journal of Open Source Software 2 (10). External Links: Cited by: §5.1.
- ClassiFunc: classification of functional data. Note: R package version 0.1.1 External Links: Cited by: §2.2, §5.2.1.
- A theory for multiresolution signal decomposition: the wavelet representation. IEEE Trans. Pattern Anal. Mach. Intell. 11, pp. 674–693. Cited by: §2.1.
- EEG feature extraction for classifying emotions using fcm and fkm. In Proceedings of the 7th WSEAS International Conference on Applied Computer and Applied Computational Science, ACACOS’08, Stevens Point, Wisconsin, USA, pp. 299–304. External Links: Cited by: §A.2.
- R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. External Links: Cited by: §1.
- Searching and mining trillions of time series subsequences under dynamic time warping. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 262–270. External Links: Cited by: §A.2, §2.1, §5.2.1.
- Fda: functional data analysis. Note: R package version 2.4.8 External Links: Cited by: §1, §2.3, §6.
- Functional data analysis. Wiley Online Library. Cited by: §2.2, §3.
- Three myths about dynamic time warping data mining.. In SDM, H. Kargupta, J. Srivastava, C. Kamath, and A. Goodman (Eds.), pp. 506–510. External Links: Cited by: Figure 2.
- Rotation forest: a new classifier ensemble method. IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (10), pp. 1619–1630. External Links: Cited by: item 1, §2.4.
- Tidyfun. GitHub. Note: https://github.com/fabian-s/tidyfun Cited by: §6.
- CRAN task view - functional data analysis. External Links: Cited by: §1.
- Mlr tutorial. arXiv preprint arXiv:1609.06146. Cited by: §4.
- Practical bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems 25, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (Eds.), pp. 2951–2959. External Links: Cited by: §5.1.
- On the use of the wavelet decomposition for time series prediction. Neurocomputing 48 (1), pp. 267 – 277. External Links: Cited by: §A.2.
- Functional and shape data analysis. Springer. Cited by: §2.2.
- Show me how you drive and i’ll tell you who you are. recognizing gender using automotive driving parameters. Procedia Manufacturing 3, pp. 5587 – 5594. Note: 6th International Conference on Applied Human Factors and Ergonomics (AHFE 2015) and the Affiliated Conferences, AHFE 2015 External Links: Cited by: item 4.
- Matching incomplete time series with dynamic time warping: an algorithm and an application to post-stroke rehabilitation. Artificial Intelligence in Medicine 45 (1), pp. 11–34. External Links: Cited by: §2.4.
- Fdasrvf: elastic functional data analysis. Note: R package version 1.6.0 External Links: Cited by: §6.
- Applications of functional data analysis: a systematic review. BMC Medical Research Methodology 13 (1), pp. 43. External Links: Cited by: §1, §2.1.
- Functional data analysis. Annual Review of Statistics and Its Application 3 (1), pp. 257–295. External Links: Cited by: §1.
Appendix A API overview
For the interested reader, we introduce a brief overview of the API and functionality.
a.1 Representing functional data in mlrFDA
A sketch of the data structure we use to represent functional data can be found in the right part of Figure 9. We assume a data set consists of data for observational units, organized in rows of features, where one row contains all observed features for one observational unit, i.e., each row typically contains several functional and/or scalar covariates. For a classical, non-functional data set, the features are single columns (as depicted in the left part of Figure 9). A functional data set, on the other hand, consists of single-column scalar features as well as functional features of different length for each functional covariate, each represented by multiple adjacent and connected columns. Each of these columns contains the evaluations of the functional feature at a certain argument value for all observational units (right part of Figure 9).
As an example which will be used throughout the remainder of this paper, we use the fuelSubset data set from package FDboost, see also Figure 1. It contains a numeric target variable heatan, the fuel’s heating value, a scalar feature h20, the fuel’s water content, and two functional features NIR and UVVIS, measured at 231 and 129 wavelengths, respectively. To start with a clean sheet, we create a data.frame containing all features as separate columns.
R¿ library(mlr) R¿ library(FDboost) R¿ df = data.frame(fuelSubset[c(”heatan”, ”h2o”, ”UVVIS”, ”NIR”)])
The first step when setting up an experiment in any analysis is to make the data accessible for the specific algorithms that will be applied. In mlr, the data itself, and additional information, such as which column corresponds to the target variable is stored as a Task, requiring the input data to be of type data.frame.
The list of column positions of the functional features is then passed as argument fd.features to makeFunctionalData(), which returns an object of type data.frame in which the columns corresponding to each functional feature are combined into matrix columns.
444As an alternative, a list of the column names containing the functional features is also valid as argument to fd.features, which is especially useful if columns are already labeled.
R¿ fd.features = list(”UVVIS” = 3:136, ”NIR” = 137:367) R¿ fdf = makeFunctionalData(df, fd.features = fd.features) R¿ str(fdf) ’data.frame’: 129 obs. of 4 variables: $ heatan: num 26.8 27.5 23.8 18.2 17.5 … $ h2o : num 2.3 3 2 1.85 2.39 … $ UVVIS : num [1:129, 1:134] 0.145 -1.584 -0.814 -1.311 -1.373 … ..- attr(*, ”dimnames”)=List of 2 .. ..$ : NULL .. ..$ : chr ”UVVIS.1” ”UVVIS.2” ”UVVIS.3” ”UVVIS.4” … $ NIR : num [1:129, 1:231] 0.2818 0.2916 -0.0042 -0.034 -0.1804 … ..- attr(*, ”dimnames”)=List of 2 .. ..$ : NULL .. ..$ : chr ”NIR.1” ”NIR.2” ”NIR.3” ”NIR.4” …
We additionally specify the name "fuelsubset" and the target variable "heatan". The structure of the functional Task object is rather similar to the non-functional Task, with the additional information functionals, which states how many functional features are present in the underlying data.
R¿ tsk1 = makeRegrTask(”fuelsubset”, data = fdf, target = ”heatan”) R¿ print(tsk1) Supervised task: fuelsubset Type: regr Target: heatan Observations: 129 Features: numerics factors ordered functionals 1 0 0 2 Missings: FALSE Has weights: FALSE Has blocking: FALSE Has coordinates: FALSE
After defining the task, a learner is created by calling makeLearner. This contains the algorithm that will be fitted on the data in order to obtain a model. Currently, mlrFDA supports both functional regression and functional classification. A list of supported learners can be found in Table 1
a.2 Machine Learning and Feature Extraction
Classical machine learning algorithms do not take into account the characteristics of functional data and treat the input data as vector valued features. Without additional preprocessing, this typically yields poor performance on, as the models cannot exploit the lower intrinsic dimensionality of the functional covariates nor the fact that they represent observations over a continuum.
In mlrFDA, classical algorithms can be applied to functional data, however, a warning message will be displayed. In our example, we train a partitioning tree on the functional data, while ignoring the functional structure.
R¿ rpart.lrn = makeLearner(”regr.rpart”) R¿ m = train(learner = rpart.lrn, task = tsk1) Functional features have been converted to numerics
For conventional learning algorithms to work well on functional data, informative scalar features need to be extracted from the functional features.
Feature extraction is applied in practice for a multiplicity of reasons, as it often not only reduces the dimensionality of the resulting problem, but also allows researchers to make use of domain knowledge, for example by hand-crafting features from measurements of continuous processes. Examples for this include deriving features that allow for sleepiness detection (Jin et al., 2013), or by extracting features from electro-cardiogram data in order to detect emotions (Murugappan et al., 2008). The resulting features often have a much lower dimensionality, which often improves fitted models. Other preprocessing methods for functional or time series data include extracting general purpose features such as wavelet coefficients (Aldrich, 2013; Soltani, 2002), principal component scores or Fourier coefficients. The resulting scalar features can then be used with different machine learning methods such as -nearest neighbors.
In the following section, we showcase the feature extraction procedure using general purpose features as an example. We want to emphasize that it is also easily possible to write custom feature extraction methods using the makeFeatureExtractionMethod function.
|Discrete Wavelet Transform||extractFDAWavelets()||wavelets|
|Fast Fourier Transform||extractFDAFourier()||stats|
|Principal Component Analysis||extractFDAPCA()||stats|
|Multi-Resolution Feature Extraction||extractFDAMultiResFeatures()||-|
|Time Series Features||extractFDATsfeatures()||tsfeatures|
|Dynamic Time-Warping Kernel||extractFDADTWKernel()||rucrdtw|
In our example, we extract the Fourier coefficients from the functional feature UVVIS, and principal component scores from the second functional feature NIR in order to transform the original task with functional data into a conventional task.
R¿ feat.methods = list(”UVVIS” = extractFDAFourier(), ”NIR” = extractFDAPCA()) R¿ extracted = extractFDAFeatures(tsk, feat.methods = feat.methods) R¿ extracted
desc Extraction of features from functional data: Target: heatan Functional Features: 2; Extracted features: 2
As an alternative, the feature extraction can be applied in a wrapper method
makeExtractFDAFeatsWrapper(). In general, a wrapper combines a learner method with another method, thereby creating a new learner that can be handled like any other learner. In our case, a classical machine learning method is combined with the data preprocessing step of feature transformation from functional to non-functional data.
R¿ wrapped.lrn = makeExtractFDAFeatsWrapper(”regr.rpart”, feat.methods = feat.methods)
This is suitable for honest cross-validation of data-adaptive feature extraction methods like principal components. We can now cross-validate the learner created above using mlr’s resample function with 10-fold cross-validation.
R¿ res = resample(learner = wrapped.lrn, task = tsk1, resampling = cv10)
In the same way, we can train and predict on data, or benchmark multiple learners across multiple data sets. Additionally, we can apply a tuneWrapper to our learner in order to automatically tune hyperparameters of the learner and the preprocessing method during cross-validation.
Appendix B Data sets used in the Benchmark
Table 7 contains all data sets used in the benchmark along with additional data properties.
Appendix C Failed and missing experiments
Experiments for some algorithm / data set combinations failed due to implementation details or algorithm properties. In order to increase transparency, failed algorithms are listed here, and if available reasons for failure are provided.
At the time of the benchmark, the implementation in the tsfeatures package was not stable enough to be included in the benchmark.
Data sets: BeetleFly, BirdChicken, Coffee, Computers, DistalPhalanxOutlineCorrect, Earthquakes, ECG200, ECGFiveDays, ElectricDeviceOn, GunPoint, Ham, Herring, ItalyPowerDemand, Lightning2, MoteStrain, ShapeletSim, SonyAIBORobotSurface1, Strawberry, ToeSegmentation1, TwoLeadECG, Wafer, Wine, Yoga
Reason: Too few instances in some classes, such that p ¿ n.
classif.fdausc.kernel and .np Data sets: ElectricDeviceOn, ShapeletSim
Data sets: DistalPhalanxTW, EpilepsyX