There is a great deal of interest in forecasting energy-related time series, specifically regarding the generation of renewable energy sources, load forecasting, electricity price forecasting or gas consumption forecasting [3, 1, 2]. For transparency purposes two European platforms offer open access to such data: ENTSO-E and ENTSOG. The data from these sources are rich, some data quality issues arise, however, that need some investigation for the platform to be useful. In this paper we utilize the free data sources to show that releasing data to the public is beneficial and should be promoted. By building a framework for the automatic prediction of energy-related time series published on an open portal we wish to add our part of encouragement.
Furthermore we incorporate the idea promoted in the Global Energy Forecasting Competition ; a new and unique approach to forecasting hourly renewable energy production, electric load and price was introduced: the forecasting of a probability distribution of the target variable in contrast to forecasting a single value. Probabilistic forecasts provide more detailed insight for stakeholders offering immediate business value. Our solution achieved good results in terms of pinball loss in the competition, and we assess if the framework is suitable for reliable probabilistic forecasts in a rolling fashion for a number of energy-related time series.
The document is organized as follows. After some basic information on the used data source in Section II, we present in Section III our comprehensive report on data quality of the platform. A brief methodological overview can be found in Section IV and the extensive tour of the internal framework structure in Section V. We perform an in-depth analysis of real-world data, then present and discuss detailed experimental results (Section VI). Conclusions and further ideas are provided in the final Section VII.
Ii Data source
ENTSO-E, the European Network of Transmission System Operators, represents electricity transmission system operators (TSOs) from countries across Europe. ENTSO-E was established and given legal mandates by the EU’s Third Legislative Package for the Internal Energy Market in 2009, which aims at further liberalizing the gas and electricity markets in the EU. The main objectives of ENTSO-E centre on the integration of renewable energy sources such as wind and solar power into the power system, and the completion of the internal energy market, which is central to meeting the European Union’s energy policy objectives of affordability, sustainability and security of supply.
The ENTSO-E Transparency Platform was launched in early 2015. It is aimed at providing free, continuous access to pan-European electricity market data for all users, across categories such as load, generation, transmission, balancing, outages and congestion. Our work focuses on the country load data published on this new portal.
Iii Data quality report
We performed the data validation on the available load data on the ENTSO-E Transparency Platform. The time span of the validation started on January 2015 and went on until October 2015, spanning 3 seasons of the year. The portal is used to access historical load data in a timely manner. Every day we perform a screen scrape of the data source for the last day of available data. Observation of the data showed that there are major types of data reporting frequencies in the publicly available load data: hourly, half-hourly and quarter hourly data reporting.
Data consistency was measured by
simple heuristics; check if the country provided data was either ’N/A’ or ’0’ at the time of the download. We summarize the occurrence of the above events for both of the provided data columns: Day-ahead Total Load Forecast, Actual Total Load. To get a bigger picture and a general score of data quality for each country, we observe the frequency of errors relative to the theoretical maximum reportable values for the validation time interval (TableI).
We selected a number of countries for investigation (boldfaced) based on their data quality and market size; Belgium, Czech Republic, Finland, France, Germany, Hungary, Italy, Norway, Poland and Slovakia. There are several countries that do not have any data on the platform at all in the validation time span, we omit these from further analysis. These countries are the following: Bosnia and Herzegovina, Cyprus, Belarus, Iceland, Moldavia, Malta, Russia, Turkey and Ukraine.
|Country code||Country name||Data quality||Frequency|
In general we can say that in the data quality assessment period the majority 21 countries out of 32 have less than bad data frequency. After cross-referencing the missing values from our historical real-time database, with the state of the database of the Transparency Platform, we see that the majority of these values are backfilled and the errors are corrected. However if one tries to rely on the timely updates of the Transparency Platform one could get into trouble with missing load values and forecasts as well. Surprisingly the better reporting countries are from the Central-Eastern Europe, most notably Croatia, Poland, Slovakia and Hungary.
All of the above countries have missing values in both the forecasted load and the actual reported load, however only a handful have zeros (Croatia, Portugal, Lithuania, Montenegro, Sweden, Serbia, Macedonia and Ireland). It should be self-evident not to report zeros as forecasts or load values, we strongly advise a check to be made at the time of the report. Table II summarizes the two metrics as percentages for each country.
|Country||Target is NA||Target is 0||Forecast is NA||Forecast is 0|
In general it is beneficial that the errors are taken care of in time, but this issue makes it harder to act upon the reported information on the portal. Also it would be beneficial to have, at least for the research community, a simple API for the data that is otherwise only accessible through screen scraping.
The aim of this paper is not only to assess the utility of the new ENTSO-E platform, but to motivate publishing open access time series data in energy by building accurate models using the state-of-the-art in machine learning.
Point forecasts for time series are often provided by reliable statistical methods like ARMA and ARIMA. Our approach to forecasting is based on a relatively new machine learning method; Gradient Boosted Regression Trees (GBRTs) were successful in a number of competitions in the past [7, 2].
Iv-a Gradient Boosting Regression Trees
Gradient boosting is an ensemble method responsible for combining weak learners for higher model accuracy, as suggested by Friedman in 2000 . The predictor generated in gradient boosting is a linear combination of weak learners, here tree models are utilized for this purpose.
A sequence of models is built iteratively, starting from a constant initial prediction (in a least squares regression task, the mean of the target variable), and with each tree built on the residuals of the previous tree. To avoid overfitting, there is a shrinkage parameter (also called learning rate), which controls how much each tree contributes to the final model; this effectively slows learning down, yielding better performance at the expense of requiring more iterations.
The final prediction is calculated as follows:
Where is the number of individual trees built, is the initial prediction, is the prediction at stage , is the prediction of an individual tree learner and is the shrinkage parameter to reduce the effect of an individual tree in the sequence.
The general idea is to compute a sequence of simple trees, where each successive tree is built for the prediction residuals of the preceding tree. Each new base-learner is chosen to be maximally correlated with the negative gradient of the loss function, associated with the whole ensemble. This way the subsequent stages will focus harder on fitting these examples and the resulting predictor is a linear combination of weak learners.
Utilizing boosting has many beneficial properties; various risk functions are applicable, intrinsic variable selection is carried out, also resolves multicollinearity issues, and works well with large number of features without overfitting.
Model performance is measured using the Mean Absolute Percentage Error (MAPE) for point forecasts, a well-established error measure throughout the energy industry domain. Probabilistic error rates were assessed using the Pinball loss function 
, a well-known tool in both statistics and machine learning and quantile time series forecasting in particular.
For a quantile forecast with as the target quantile, the pinball loss score is defined as:
where is the observation used for verification, and a = 1, 2, …, 99.
The pinball loss function returns a value that can be interpreted as the accuracy of a quantile forecasting model. Being a loss function the lower rates are better.
V Framework structure
V-a Module overview
The framework consists of modules responsible for scraping the data, storing it in a NoSQL database, building GBRT models and using the models to issue timely forecasts (see Figure 1). The main output of the framework is electric load value predictions for the next hours for the selected countries. The prediction function launches subprocesses with prediction horizon set between and hours, it is also capable of parallel execution. Each subprocess consists of three main parts; data preparation, model building (if model rebuild is requested) and model application.
The end result of the prediction function is a dataset of predicted hourly load values ahead of the test time for the given country. This means that if the forecasting system is in continuous operation, there will be forecasts for each hour, with horizons decreasing from to .
V-B Data preparation
First, Vertical Load and total load values and the calendar of the target country are loaded from the NoSQL database. Total load means a load equal to generation and any imports deducting any exports and power used for energy storage. While Vertical Load equals the total amount of power flowing out of the transmission network to the distribution networks, to directly connected final customers or to the consuming part of generation. There is an issue that total load values are not available prior to , but in previous years substituting total load with Vertical Load data improves performance. This is a curious behaviour as Vertical Load and Total Load are inherently different concepts. Needless to say, when enough Total Load data has been acquired this tweak no longer will be necessary. For some countries, Total Load values are available in quarter-hourly or half-hourly granularity, so the data preparation module aggregates them to uniform hourly values.
Next, feature vectors are created out of the load value observations. Calculating the feature vectors involves both calendar variables and lagged values. Do note, although desired at this point no weather data is available to be used. Currently two lagged values are used; the load value one week before the predicted time slot and the last known load value (adjusted by the forecast horizon). If a lagged load value happens to be missing in the training data that row is discarded entirely. It is crucial that no test row has missing values, so if a lagged load is missing the data preparation function imputes it using values from previous periods. If no lagged values are found for the past 6 weeks, the module switches to the more robust setting where no lagged load variables are used.
V-C Model building
This part is executed only if the model rebuild is requested, during normal operations model rebuild is scheduled for every midnight. There are two distinct models built: basic and advanced. Basic models are constructed on calendar-based features only, and are used as fall back method. Advanced models use lagged load features in addition meaning their performance is much better. The underlying algorithm used for both models is Gradient Boosted Regression Trees. For each country the basic version builds 50 trees with a depth of 5, while the up-scaled version has twice as many trees with larger depth. All generated models are saved in the NoSQL database along with their metadata.
V-D Model application
In this step predictions for each test row are generated by applying the most recent model in the database that matches the country, horizon and model type (basic or advanced).
Vi-a Model evaluation & result comparison
To justify the usage of a new approach in time series forecasting it is often benchmarked with an established methodology like ARMA on a set of time series in the respective domain . In this particular application, if provided in the open access data, the country issued estimates of energy consumption are used to assess the relative accuracy of the GBRT methodology. These estimates are issued by the respective country as part of their transparency regulations. The framework provides 24 estimates for every given time point with a forecast horizon spanning between 1 and 24 hours. Day ahead forecasts are typical in energy industry so if otherwise not stated all evaluation was performed on the 24 hours horizon.
Performance is measured using the Mean Absolute Percentage Error (MAPE) and model forecasts are compared to Actual Load values directly, and error measures are compared to country benchmark model forecasts as seen on the ENTSO-E Transparency Platform. Figure 2 shows how different time series stack up in the sample period of late March.
Table III summarizes measured MAPE over the test period for each country. The GBRT framework generally performs well, but accuracy varies from country to country, the best being Czech Republic, Poland, Slovakia and to some extent Hungary. The model coped very well with the different seasons; all 3 have very similar mean error rates. Do note, that there was no Actual Load data published for Italy from August and onwards.
Performance measurements are put into context when looking at the country provided estimates used as benchmark here (see Table IV). Observed MAPE rates are typically lower here, with the sole exception of Czech Republic. Cases where our GBRT framework offers comparable results include Czech Republic, Hungary and to a lesser extent Germany and Slovakia. Scandinavian and Benelux countries operate with extremely low error rates. At this point, very little is known of the models and data used by any of the countries. Though country estimates could potentially (and should) incorporate additional information besides historical and calendar-related data such as weather forecasts, longer spanning historical data, customer data, market-related data. Again, providing at least partial open-access to this data could greatly boost development in this field.
Looking at the error statistics in Table V
one can see that all error distributions are naturally right skewed suggesting a distribution close to log-normal, meaning the GBRT powered framework more frequently commits smaller errors than higher ones. The framework performance is rather robust over countries, major error percentiles are well aligned with the distribution tail having varying length. Figure3 shows sample distributions for Germany comparing framework and benchmark errors. The GBRT errors demonstrate a much longer tail indicating that some patterns or information might be missing from this representation. On the other hand errors close to zero are also more frequent in the model output.
The framework provides 24 estimates for every given time point, shorter horizons often produce better results as underlined by Table VI. Forecast errors between 1 and 24 hours length typically draw an arc, with higher error rates in the middle and the lowest on the edges. But the distribution is far from symmetrical; a 1-length horizon typically yields approximately 12% lower error rates than its 24 hour counterpart.
Furthermore we investigate the forecasting of a probability distribution of the target variable in contrast to forecasting a single value. Probabilistic forecasts provide more detailed insight for stakeholders offering immediate business value. For this purpose we utilize the quantile regression capabilities of the GBRT model and estimate target variable distribution over the major deciles. Probabilistic forecasts were experimented with Hungary only for the month of September, training the model on the period of January 2012 - August 2015. Figure4 shows a sample of the estimated time series distribution along with the Actual Load. The quantile intervals here are relatively narrow on most days, with slight bias issues in some cases.
Figure 5 shows the probabilistic Pinball Loss distribution over the quantiles. The distribution is almost symmetrical with higher rates observed close to the median. The average pinball loss measured for this setting is . Unfortunately with only point forecasts published on the portal there is no direct way to compare the loss to country estimates. We look forward to probabilistic forecasts being released in the near future as they provide more detailed insight thus generating business value.
Vii Future work & Conclusions
The framework presented in this paper is functional and fully operational, yet there are still plenty of ideas to further extend it. As mentioned in the paper, at the moment no external data sources are used. Previous experience shows that incorporating country specific weather forecasts can greatly boost prediction accuracy and provide richer data sources encapsulated in the framework. ENTSOG integration, the gas counterpart of ENTSO-E, is also a reasonable next step as gas usage is even more weather dependent than electricity. To further justify the usage of GBRT in time series forecasting there is a need to benchmark it to an established methodology like ARMA on a relatively large set of time series in the energy domain in a rolling fashion. Country specific load data is particularly suitable for this purpose.
Benchmarking is not the sole purpose of this paper, as it deals with the data analysis process as a whole from ETL to data preparation (ENTSO-E integration, data cleansing and data auditing), through modeling and result presentation (see Figure 6) with an API providing parametric forecasts for a number of countries in Europe. It will hopefully contribute to the agenda of releasing more open access data to the public that can be put to good use in the future.
-  E. Mangalova, E. Agafonov, Wind power forecasting using the k-nearest neighbors algorithm, International Journal of Forecasting 30 (2) (2014) 402–406.
-  G. Barta, G. Borbely, G. Nagy, S. Kazi et al., GEFCOM 2014 - Probabilistic Electricity Price Forecasting Intelligent Decision Technologies, 39 (2015) 67–76.
-  S. Aggarwal, L. Saini, Solar energy prediction using linear and non-linear regularization models: A study on ams (american meteorological society) 2013–14 solar energy prediction contest, Energy 78 (C) (2014) 247–256.
-  T. Hong, P. Pinson et al. Probabilistic energy forecasting: State-of-the-art 2015, International Journal of Forecasting.
-  Central collection and publication of electricity generation, transportation and consumption data and information for the pan-European market. [online] 2015, https://transparency.entsoe.eu (Accessed: 20 November 2015).
-  J. H. Friedman, Stochastic gradient boosting, Comput. Stat. Data Anal. 38 (4) (2002) 367–378.
-  G. Nagy, Gefcom2012: Wind power forecasting using gradient boosting machines, in: Power and Energy Society General Meeting, Power and Energy Society, 2012, p. GM2555.
-  J. H. Friedman, ”Greedy function approximation: a gradient boosting machine,” Annals of Statistics, pp. 1189-1232, 2001.
-  R. Koenker and G. Bassett Jr., “Regression quantiles,” Econometrica, vol. 46, pp. 33–50, 1978
-  Biau, Gérard, and Benoît Patra. ”Sequential quantile prediction of time series.” Information Theory, IEEE Transactions on 57.3 (2011): 1664-1674.
-  ENTSO-E Metadata Repository Glossary. [online] 2015, https://emr.entsoe.eu/glossary/bin/view/Main/ (Accessed: 21 November 2015).