Feature selection is to select a parsimonious feature subset to improve model interpretability and efficiency without degrading model accuracy (Guyon and Elisseeff, 2003). Traditionally, online feature selection deals with the observations sequentially added while the total dimensionality is fixed (Wang et al., 2014; Hoi et al., 2014). However, in many real world applications, it is either impossible to acquire an entire feature set or impractical to wait for the complete set before feature selection starts. For instance, in Twitter, trending topics keep changing over time, and thus the dimensionality of data is changed dynamically. When a new top topic appears, it may come with a set of new keywords, which usually serve as key features to identify new hot topics. In bioinformatics, it is expensive for feature selection to acquire an entire set of features for each data observation due to the high cost in conducting wet lab experiments (Wang et al., 2014). Many big data applications call for online streaming feature selection to consume sequentially added dimensions over time.
In general, assuming is the feature set containing all features available till time , and is the class attribute, then a training data set is defined by , which is a sequence of features that is presented over time. As we process one dimension at a time, the research problem is that at any time , how to online maintain a minimum size of feature subset of maximizing its predictive performance for classification. If is a new coming feature at time , is the selected feature set till time and
denotes the posterior probability ofconditioned on a subset , the problem of online streaming feature selection is formulated as Eq.(1).
To solve Eq.(1), currently the state-of-the-art algorithms include Grafting (Perkins and Theiler, 2003), Alpha-investing (Zhou et al., 2006) , OSFS (Wu et al., 2010), Fast-OSFS (Wu et al., 2013), and SAOLA (Yu et al., 2014). All of those algorithms only deal with one dimension at a time upon its arrival.
Group information sometimes is embedded in a feature space. For instance, in image analysis, features are generated in groups which represent color, texture and other visual information. If is the set of all feature groups available till time , then at at time , is denoted by , which is a sequence of feature groups that is added sequentially. To consume grouped features sequentially added over time, online selection of dynamic groups is formulated as Eq.(2).
In Eq.(2), is a new coming group at time , and is the set of selected groups till time . Eq.(2) attempts to yield a set of groups at time that is as parsimonious as possible at the levels of both intra-groups (constraint (a)) and inter-groups (constraint (b)) simultaneously for maximizing its predictive performance for classification. To online utilize grouped features, the group-SAOLA algorithm was proposed (Yu et al., 2015).
As an emerging research direction, online streaming feature selection provides a new, complementary algorithmic methodology to enrich online feature selection, especially addresses high dimensionality in big data analytics. But to the best of our knowledge, there is no comprehensive open-source packages existing for this problem. To facilitate research efforts on this novel direction, we develop the open-source library called LOFS (Library of Online streaming Feature Selection).
The main contribution of LOFS lies on three aspects. (1) It is the first comprehensive open-source library for implementing algorithms of online streaming feature selection. (2) It provides the state-of-the-art algorithms of online streaming feature selection mainly developed by our research group, including the algorithms to learn features added individually over time and the methods to mine grouped features added sequentially. (3) It is written in MATLAB, easy to use, and completely open source. We hope it will facilitate the development of new online algorithms for tackling the grand challenges of high dimensionality in big data analytics, and encourage researchers to extend LOFS and share their algorithms through the LOFS framework.
2 Architecture of LOFS
The LOFS architecture is based on a separation of three modules, that is, CM (Correlation Measure), Learning, and SC (Statistical Comparison), as shown in Figure 1. The learning module consists of two submodules, LFI (Learning Features added Individually) and LGF (Learning Grouped Features added sequentially).
In the CM module, the library provides four measures to calculate correlations between features, Chi-square test, test, the Fisher’s Z test, and mutual information, where Chi-square test, test, and mutual information for dealing with discrete data while the Fisher’s Z test for handling continuous data.
With the measures above, the LFI module includes Alpha-investing (Zhou et al., 2006), OSFS (Wu et al., 2010), Fast-OSFS (Wu et al., 2013), and SAOLA (Yu et al., 2014) to learn features added individually over time, while the LGF module provides the group-SAOLA algorithm (Yu et al., 2015) to online mine grouped features added sequentially.
Based on the learning module, the SC module provides a series of performance evaluation metrics (i.e., prediction accuracy, AUC, kappa statistic, and compactness, etc ). To conduct statistical comparisons of algorithms over multiple data sets, the SC model further provides the Friedman test and the Nemenyi test(Demšar, 2006; Li, 2014).
The three modules in the LOFS architecture are designed independently, and all codes follow the MATALB standards. This guarantees that the LOFS library is simple, easy to implement, and easily extendable. One can easily add a new algorithm to the LOFS library and share it through the LOFS framework.
3 Usage of LOFS
The LOFS library comes with detailed documentation. The documentation is available from https://github.com/kuiy/LOFS. This documentation describes the setup and usage of LOFS. All the functions and related data structures are explained in detail. In Figure 2, we give an example to illustrate how to implement the functions of the Fast-OSFS algorithm in LOFS, such as loading data, setting parameters, running algorithms, and evaluating performance.
In addition to the documentation, the extensive performance comparisons between online streaming feature selection and traditional online feature selection, and the empirical comparisons between online streaming feature selection and the traditional feature selection, such as GDM (Zhai et al., 2012), FCBF (Yu and Liu, 2004), and SPSF-LAR (Zhao et al., 2013), can be found in Yu et al. (Yu et al., 2015).
4 Conclusion and Future Work
This paper presents LOSF, an easy-to-use open-source package for online streaming feature selection to facilitate research efforts in machine learning and data mining. It is the first open-source package for online streaming feature selection. Through the LOFS framework, we hope that it will facilitate researchers to develop new online learning algorithms for big data analytics and share their algorithms. Future versions of LOFS will have the ability to load and save the ARFF (Weka Attribute-Relation File Format) files and LIBSVM data files111https://www.csie.ntu.edu.tw/cjlin/libsvmtools/datasets/. LOFS will keep trying its best to integrate new developed algorithms in future.
Statistical comparisons of classifiers over multiple data sets.The Journal of Machine Learning Research, 7:1–30, 2006.
- Guyon and Elisseeff (2003) Isabelle Guyon and André Elisseeff. An introduction to variable and feature selection. The Journal of Machine Learning Research, 3:1157–1182, 2003.
- Hoi et al. (2014) Steven CH Hoi, Jialei Wang, and Peilin Zhao. Libol: A library for online learning algorithms. The Journal of Machine Learning Research, 15(1):495–499, 2014.
- Li (2014) Yifeng Li. Sparse machine learning models in bioinformatics, PhD thesis, School of Computer Science, University of Windsor, Canada. 2014.
- Perkins and Theiler (2003) Simon Perkins and James Theiler. Online feature selection using grafting. In ICML-2003, pages 592–599, 2003.
- Wang et al. (2014) Jialei Wang, Peilin Zhao, Steven CH Hoi, and Rong Jin. Online feature selection and its applications. Knowledge and Data Engineering, IEEE Transactions on, 26(3):698–710, 2014.
- Wu et al. (2010) Xindong Wu, Kui Yu, Hao Wang, and Wei Ding. Online streaming feature selection. In ICML-2010, pages 1159–1166, 2010.
- Wu et al. (2013) Xindong Wu, Kui Yu, Wei Ding, Hao Wang, and Xingquan Zhu. Online feature selection with streaming features. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 35(5):1178–1192, 2013.
- Yu et al. (2014) Kui Yu, Xindong Wu, Wei Ding, and Jian Pei. Towards scalable and accurate online feature selection for big data. In IEEE ICDM-2014, pages 660–669. IEEE, 2014.
- Yu et al. (2015) Kui Yu, Xindong Wu, Wei Ding, and Jian Pei. Scalable and accurate online feature selection for big data. arXiv:1511.09263v1 [cs.LG], 2015.
- Yu and Liu (2004) Lei Yu and Huan Liu. Efficient feature selection via analysis of relevance and redundancy. The Journal of Machine Learning Research, 5:1205–1224, 2004.
- Zhai et al. (2012) Yiteng Zhai, Mingkui Tan, Ivor Tsang, and Yew Soon Ong. Discovering support and affiliated features from very high dimensions. In ICML-2012, pages 1455–14629, 2012.
- Zhao et al. (2013) Zheng Zhao, Lei Wang, Huan Liu, and Jieping Ye. On similarity preserving feature selection. Knowledge and Data Engineering, IEEE Transactions on, 25(3):619–632, 2013.
- Zhou et al. (2006) Jing Zhou, Dean P Foster, Robert A Stine, and Lyle H Ungar. Streamwise feature selection. The Journal of Machine Learning Research, 7:1861–1885, 2006.