Combined statistical procedures, also known as ensemble or aggregation methods, are very popular in Machine Learning competitions – the celebrated Netflix Challenge (as are repeatedly Kaggle competitions) was won by an ensemble technique (see for example bell2007lessons, for a discussion). In a nutshell, ensemble methods combine different predictors (each trained on the same dataset) to improve performance. scikit-learn offers implementations of some ensemble methods but it does not ship with a specific analysis toolbox for aggregation. pycobra attempts to fill this gap by providing to the scikit-learn environment a toolbox to analyse and visualise the different preliminary predictors (also called machines hereafter), in addition to providing pythonic implementations of several ensemble algorithms (the COBRA algorithm introduced by biau2016cobra; the Exponential Weighted Aggregate (EWA) introduced by vovk; a COBRA-flavored majority vote inspired by mojirsheibani). All these algorithms are supported by oracle bounds whose main message is that their performance outperforms the performance of the best preliminary predictor, up to a remainder term which decays to zero.
COBRA (standing for COmBined Regression Alternative) is a nonlinear ensemble method designed for regression problems. To predict the response for a new data point, COBRA operates in two steps. First, it retains data points for which the prediction made by preliminary machines is close (in the Euclidean sense) to the prediction made for the new point. The final prediction is then formed by averaging responses over the retained points’ indices. COBRA outputs a predictor which outperforms any combination of the preliminary predictors (as shown by see Theorem 2.1 in biau2016cobra). We describe the COBRA algorithm as implemented in pycobra in Algorithm 1
. Exponential weights have been used for a long time in statistical learning theory(vovk). The EWA algorithm implemented in Pycobra is inspired by the description of dalalyan. EWA amounts to forming an exponentially weighted average of preliminary predictors, where the weights include a measure of each predictor’s performance. COBRA has been inspired by the work of mojirsheibani in classification, and therefore pycobra includes a classification version of COBRA called ClassifierCobra. The consensus step is similar to what is done in regression but the final prediction is delivered by a majority vote among the labels of the retained data points.
pycobra allows the user to gauge the performance of the preliminary predictors used in the aggregation, with built-in methods to easily plot boxplots and QQ-plots. A salient feature of pycobra is using Voronoi tessellations for generic visualisation. By implementing ensemble algorithms which were not represented in the Python machine learning community, and providing a variety of tools to visualise and analyse their behavior and performance, we present to the machine learning open source community a toolbox designed for ensemble learning and visualisation.
2 The pycobra library
Our toolbox is written in Python and uses NumPy (walt2011numpy) and scikit-learn (pedregosa2011scikit) for computation and linear algebra operations. Matplotlib (hunter2007matplotlib) is used for visualisation purposes, and SciPy (jones2014scipy) is used to help create Voronoi tessellations. Tutorials and examples are created using Jupyter IPython notebooks (PER-GRA:2007). Currently pycobra supports any machine learning algorithm with a predict method, casting our procedure into a very flexible and generic framework. By default, pycobra relies on scikit-learnEWA and COBRA implementations. As for ClassifierCobra, scikit-learnpycobra itself and all software it relies on is open source, with no dependence on proprietary software. Algorithm 1
presents the pseudo-code of the COBRA implementation. All the pycobra estimators arescikit-learn compatible and can be used as part of the existing scikit-learn ecosystem, such as GridSearchCV and Pipeline.
While hyperparameter initialisation is systematically done usingscikit-learn’s GridSearchCV, pycobra’s diagnostics class allows us to compare between different combinations of the constituent predictors and data-splitting, among other basic parameters.
The visualisation class allows the user to compare all the machines used to create aggregates, as well as visualise the results, for all pycobra estimators. pycobra ships with a notebook on visualisation to illustrate this.
QQ-plots and boxplots.
Once all the basic machines are set up, the user can easily compare their performance with boxplots and QQ-plots. All the plotting details are handled by both the diagnostics and visualisation classes. An example is given in Figure 1.
Visualisation through Voronoi tessellations.
Voronoi tessellations can be used to visualise the selection of optimal machines for COBRA, as well as visualising outputs from clustering algorithms, as illustrated by the tutorial notebook. An example is reproduced Figure 2.
3 Project Focus
We intend pycobra to be an ongoing collaborative project. To that matter, it is referenced on Python Package Index (PyPi) and Machine Learning Open Source Software. Moreover, pycobra is under active development and available on GitHub to enforce collaborative programming, issue tracking, code integration, and idea discussions.
Documentation and Jupyter Notebooks.
A consistent API documentation is provided, along with an additional installation guide and examples. The documentation website is here. The notebooks directory contains Jupyter notebooks which serve as both a documentation tool and a tutorial resource. These notebooks cover use-cases of Pycobra, from solving regression and classification problems to using Voronoi tesselations.
Ease of use and quality assurance.
Ensemble learning with pycobra is as simple as loading trained scikit-learn machines – or any machine which has a predict method. Visualising involves little to no parameters, and after loading machines it is straightforward to analyse their performance on a particular dataset. In order to ensure code quality, a set of unit tests is provided for all classes in pycobra, and continuous integration via Travis CI ensures all commits are tested. All the code is PEP8 compliant, keeping the code easy to read and contribute to.
4 Conclusion and Future Work
The future of pycobra would be to grow its user base by adding new ways to add predictors, as well as further implement ensemble learning techniques and visualisation tools. Statistical aggregation and ensemble learning are an important part of the machine learning literature and are widely used by practitioners, yet it seems under-represented in the machine learning open source community. By creating pycobra and releasing it to the community, we intend to enrich the existing ecosystem with ensemble algorithms, a generic toolbox for ensemble learning on-the-go, along with analysis and visualisation tools.