Process analytics is an area of process mining [van2016data] which encompasses Predictive Process Monitoring (PPM) aimed at making predictions for individual process instances or overall process models. At the instance level, various novel PPM techniques have been recently devised, tackling problems such as next activity, remaining cycle time, or other outcome predictions [DBLP:conf/bpm/Francescomarino18]
. These techniques make use of neural networks[DBLP:conf/caise/TaxVRD17], stochastic Petri nets [DBLP:conf/icsoc/Rogge-SoltiW13], and general classification techniques [DBLP:journals/tkdd/TeinemaaDRM19].
At the model level, there is a notable void. Many analytical tasks require not only an understanding of the current as-is, but also the anticipated will-be process model. A key challenge in this context is the consideration of evolution as processes are known to be subject to drift [maaradji2017detecting, DBLP:conf/bpm/PollPRRR18, yeshchenko2019comprehensive, yeshchenko2021visual]. A forecast can then inform the process analyst how the will-be process model might differ from the current as-is if no measures are taken, e.g., against emerging bottlenecks.
This paper presents the first technique to forecast whole process models. To this end, we develop a technique that builds on a representation of event data as multiple time series. Each of these time series captures the evolution of a behavioural aspect of the process model in the form of directly-follows relations (DFs), such that corresponding forecasting techniques can be applied for directly-follows graphs (DFGs). Our implementation on six real-life event logs demonstrates that forecasted models with medium-sized alphabets (10-30 activities) obtain below 15% mean average percentage error in terms of conformance. Furthermore, we introduce the Process Change Exploration (PCE) system which allows to visualise past and present models from event logs and compare them with forecasted models.
2 Related work and motivation
Within the field of process mining, research on and use of predictive modelling techniques has attracted plenty of attention in the last five years. PPM techniques are usually developed with a specific purpose in mind, ranging from next activity prediction [evermann2017predicting, DBLP:conf/caise/TaxVRD17], over remaining time prediction [verenich2019survey], to outcome prediction [kratsch2020machine]. For a systematic literature review of the field, we refer to [neu2021systematic]. Beyond the PPM field, this work is related to previous research on stage-based process mining [nguyen2016business], in which a technique is presented to decompose an event log into stages, and work on the detection of time granularity in event logs [pourbafrani2020].
The shift from fine-granular PPM techniques, including next activity, remaining time, and outcome prediction, to model-based prediction, allows to obtain new insights into the global development of the process. Consider the example in Figure 1 where the road fine traffic management event log is partitioned into 100 intervals in which an equal number of DF relations occur.
The DFs in the first 50 intervals are used to predict the next 25 intervals. The DFGs show how process model forecasting and change exploration can provide multiple unique insights at a glance:
Compared to the initial 50 intervals the proportion of fines sent decreases in the later intervals;
The proportion of penalties remains comparable between the first 50 and next 25 time intervals;
The number of occurrences and arc weights between Create Fine and Send Fine are forecasted with reasonable error (15%);
The arc weights of the ending activities are predicted with reasonable error (15%).
These results provide insight both in terms of the past and present model, see items (1)-(2), and the quality of forecasts between the actual and forecasted model, (3)-(4). Being able to construct such forecasts allows stakeholders to make estimates regarding how the overall fine system will evolve and allows to answer questions such as “How many more fines will be received?”, “Will the backlog of fines be reduced?”, “Will all fines be paid”, and ‘Will the ratio of unpaid fines stay the same?’ This motivating example shows that, where process mining focuses on learning the as-is model to reason about trajectories of future cases and suggest potential repairs and improvements, process model forecasting allows to grasp the future stage of the overall process model in terms of a will-be model.
A suitable means to evaluate the forecasts quantitatively is entropic relevance [DBLP:conf/icpm/PolyvyanyyMG20]. This measure captures the quality of the discovered and forecasted DFGs with respect to the event logs they represent. Entropic relevance penalises the discrepancies in the relative frequencies of traces recorded in the log and described by the DFG as it stands for the average number of bits used to encode a log trace using the DFG, with small values being preferable to large ones. If the entropic relevance of the forecasted DFG and the actual future DFG with respect to the test log is the same, then both DFGs represent the future behaviour similarly well. The entropic relevance of the historical DFG derived from the training log with respect to the testing log is 6.66 as indicated in Figure 1, suggesting that the future behaviour shifts and the historical DFG still represents the behaviour in the log better than both the actual and forecasted DFGs which sit at an entropic relevance of 11.11.
Measurement values are not enough to fully reveal the change of behaviour to the analyst. To this end, we complement the model-level prediction technique with a visualisation system to enable analysts to understand the forthcoming changes to the processes. Various process analysis tasks benefit from process forecasting [DBLP:conf/bpm/PollPRRR18]; most notably process forecasting helps understanding the incremental changes and adaptations that happen to the process model and to project them into the future. In terms of visualisation principles, we follow the “Visual Information-Seeking Mantra”: overview first, zoom and filter, then details-on-demand [DBLP:conf/vl/Shneiderman96]. Thus, we expect the design of our visualisation system to assist in the following tasks:
The visualisation system should assist the user in identifying the changes that happen in the process model of the future in respect to the past;
The user should be able to follow the visual information-seeking principles, including overview first, filtering, zooming, and details-on-demand.
Forecasting entire process models provides a new perspective on predictive process monitoring. The forecast horizon is substantially longer as compared to what existing next-activity prediction models can achieve. Moreover, where next activity and related PPM techniques have a strong case-level focus, a forecast at the model level provides a more comprehensive picture of the future development of the process.
3 Process model forecasting
This section outlines how time series of directly-follows relationships are extracted from event logs as well as how they are used to obtain process model forecasts with a range of widely-used forecasting techniques. Finally, the visualisation of such forecasts is introduced.
3.1 From event log to directly-follows time series
An event log contains the recording of traces which are sequences of events produced by an information system during its execution. A trace is a finite sequence over the alphabet of activities which serves as the set of event types. Directly-follows relations between activities in an event log can be expressed as counting functions over activity pairs so counts the number of times activity is immediately followed by activity in the event log . Directly-follows relations can be calculated over all traces or a subset of subtraces of the log. Finally, a Directly-Follows Graph (DFG) of the process then is the weighted directed graph with the activities as nodes and DF relations as weighted edges, i.e., .
In order to obtain forecasts regarding the evolution of the DFG we construct DFGs for subsets of the log. Many aggregations and bucketing techniques exist for next-step, performance, and goal-oriented outcome prediction [DBLP:conf/caise/TaxVRD17, DBLP:journals/tkdd/TeinemaaDRM19, nguyen2016business], e.g., predictions at a point in the process rely on prefixes of a certain length, or particular state aggregations [DBLP:journals/sosym/AalstRVDKG10]. In the forecasting approach proposed here, we integrate concepts from time-series analysis. Hence, the evolution of the DFGs is monitored over intervals of the log where multiple aggregations are possible:
Equitemporal aggregation: each sublog of interval contains a part of the event log of some fixed time duration. This can lead to sparsely populated sublogs when the events’ occurrences are not uniformly spread over time; however, it is easy to apply on new traces.
Equisized aggregation: each sublog of interval contains a part of the event log where an equal amount of DF pairs occurred which leads to well-populated sublogs when enough events are available.
exemplify the aggregations. These aggregations are useful for the following reasons. First, an equisized aggregation, in general, has a higher likelihood of the underlying DFs approaching a white noise time series which is required for a wide range of time series forecasting techniques[hyndman2018forecasting]. Second, both offer different thresholds at which forecasting can be applied. In the case of the equisized aggregation, it is easier to quickly construct a desired number of intervals by simply dividing an event log into the equisized intervals. However, most time series forecasting techniques rely on the time intervals being of equal duration which is embodied into the equitemporal aggregation [kil1997optimum]. Time series for the DFs can be obtained for all activity pairs where by applying the aforementioned aggregations to obtain the sublogs for the intervals.
3.2 From DF time series to process model forecasts
The goal of process model forecasting is to obtain a forecast for future DFGs by combining the forecasts of all the DF time series. To this purpose, we propose to use time series techniques to forecast the DFG at time given time series up until for which various algorithms can be used. In time series modelling, the main objective is to obtain a forecast for a horizon based on previous values in the series [hyndman2018forecasting]. For example, the naive forecast simply uses the last value of the time series as its forecast . An alternative naive forecast uses the average value of the time series as its forecast .
A trade-off exists between approaching DFGs as a multivariate collection of DF time series, or treating each DF separately. Traditional time series techniques use univariate data in contrast with multivariate approaches such as Vector AutoRegression (VAR) models, and machine learning-based methods such as neural networks or random forest regressors. Despite their simple setup, it is debated whether machine learning methods necessarily outperform traditional statistical approaches. The study in[makridakis2018statistical] found that this is not the case on a large number of datasets and the authors note that machine learning algorithms require significantly more computational power. This result was later reaffirmed, although it is noted that hybrid solutions are effective [makridakis2020m4]. For longer horizons, traditional time series approaches still outperform machine learning-based models. Given the potentially high number of DF pairs in a DFG, the proposed approach uses a time series algorithm for each DF series separately. VAR models would require a high number of intervals (at least as many as there are DFs times the lag coefficient) to estimate all parameters of all the time series despite their potentially strong performance [thomakos2004naive]. Machine learning models could potentially leverage interrelations between the different DFs but again would require training set way larger than typically available for process mining to account for dimensionality issues due to the potentially high number of DFs. Therefore, in this paper, traditional time series approaches are chosen and applied to the univariate DF time series, with at least one observation per sublog/time interval present.
Autoregressive, moving averages, ARIMA, and varying variance models make up the main families of traditional time series forecasting techniques[hyndman2018forecasting]. In addition, a wide array of other forecasting techniques exist, ranging from simple models such as naive forecasts over to more advanced approaches such as exponential smoothing and auto-regressive models. Many also exist in a seasonal variant due to their application in contexts such as sales forecasting.
The Simple Exponential Smoothing (SES) model uses a weighted average of past values whose importance exponentially decays as they are further into the past, where the Holt’s models introduce a trend in the forecast, meaning the forecast is not ‘flat’. Exponential smoothing models often perform very well despite their simple setup [makridakis2018statistical]. AutoRegressive Integrating Moving Average (ARIMA) models are based on auto-correlations within time series. They combine auto-regressions with a moving average over error terms. It is established by a combination of an AutoRegressive (AR) model of order to use the past values in the time series and to apply a regression over them and a Moving Average (MA) model of order to create a moving average of the past forecast errors. Given the necessity of using a white noise series for AR and MA models, data is often differenced to obtain such series [hyndman2018forecasting]. ARIMA models then combine both AR and MA models where the integration occurs after modelling, as these models are fitted over differenced time series. ARIMA models are considered to be one of the strongest time series modelling techniques [hyndman2018forecasting]. An extension to ARIMA, which is widely used in econometrics, are the (Generalized) AutoRegressive Conditional Heteroskedasticity ((G)ARCH) models [francq2019garch]. These models relax the assumption that the variance of the error term has to be constant over time, and rather model this variance as a function of the previous error term. For AR-models, this leads to the use of ARCH-models, while for ARMA models GARCH-models are used as follows. An ARCH(q) model captures the change in variance by allowing it to gradually increase over time or to allow for short bursts of increased variance. A GARCH(p,q) model combines both the past values of observations and the past values of variance. (G)ARCH models often outperform ARIMA models in contexts such as the forecast of financial indicators, in which the variance often changes over time [francq2019garch].
In general, we can regard linear SES models as a subset of ARIMA models, where (G)ARCH models are specializations of ARIMA models that can be regarded as increasingly complex and better capable of modelling particular intricacies in the time series. However, the success of different models for forecasting purposes does not depend on their complexity, and the most suitable technique is mainly determined by performance on training and test sets.
3.3 Process change exploration
In Sections 3.1 and 3.2 we described the approach for forecasting process models. To that end, gaining actual insights from such forecasted values remains a difficult task for the analyst. This section sets off to present the design of a novel visualisation system to aid analysts in the exploration of the event logs and their corresponding (forecasted) discovered process models.
Following the user tasks LABEL:req:adaptation and LABEL:req:interactive from Section 2, we designed a Process Change Exploration (PCE) system to support the interpretation of the process model forecasts. PCE is an interactive visualisation system that consists of three connected views.
Adaptation Directly-Follows Graph (aDFG) view. This is the main view of the visualisation that will show the model of the process. In order to accomplish task LABEL:req:adaptation, we modify the DFG syntax. To display the process model adaptation from time range , to , we display the union of the process models of these regions, annotating the nodes and edges with the numbers of both ranges. We colour the aDFG as follows: we use colour saturation to show the nodes with higher values. We colour edges with a diverging saturation (red-black-green) schema. This colouring applies red colour to edges that are dominant in the range, and green if edges are dominant in the range, otherwise the edge colour is close to black. For coloring edges, we reused the idea of the three colour schema from [DBLP:conf/grapp/KriglsteinR12].
Timeline view with brushed regions. This view represents the area chart graph that shows how the number of activity executions changes with time. The colour of the the area chart is split into two parts, one for the actual data and the other to show the time range of forecasted values. Analysts can brush one region in order to zoom in, creating one region of interest that is displayed on the DFG. Analysts can also brush two regions of the area chart to select two time ranges, updating the DFG to the aDFG representation. The brushed regions are coloured accordingly to the schema for colouring aDFG transitions. The earlier brushed region is coloured in red, while the second one is coloured green.
Activity and path sliders. We adopt two sliders to simplify the DFG [leemans2019directly] and the aDFG for detailed exploration of the models.
Based on the described views, we conjecture that the analyst can accomplish tasks LABEL:req:adaptation and LABEL:req:interactive with ease.
4 Implementation and evaluation
In this section, an experimental evaluation over six real-life event logs is reported. The aim of the evaluation is to measure to what extent the forecasted DFG process models are capable of correctly reproducing actual future DFGs in terms of allowing for the same process model behaviour. To this end, we benchmark the actual against the forecasted entropic relevance, as discussed in Section 2. This is done for various parts of the log, i.e. forecasts for the middle time spans of the event logs up to the later parts of the event log to capture the robustness of the forecasting techniques in terms of the amount of data required to obtain good results for both the equisized and equitemporal aggregation.
4.1 Re-sampling and test setup
To obtain training data, time series are constructed by specifying the number of intervals (i.e., time steps in the DF time series) using either equitemporal or equisized aggregation, as described in Section 3.1. Time series algorithms are parametric and sensitive to sample size requirements [hanke2001business]. Depending on the number of parameters a model uses, a minimum size of at least 50 steps is not uncommon. However, typically, model performance should be monitored at a varying number of steps. In the experimental evaluation, the event logs are divided into 100 time intervals with a varying share of training and test intervals. A constant and long horizon is used meaning all test sets contain 25 intervals, but the training sets are varied from to intervals; the forecasts progressively target the forecast of intervals 25-50 (the second quarter of intervals) over to 75-100 (the last quarter of intervals). This allows us to inspect the difference in results when only a few data points are used, or data points in the middle or towards the end of the available event data are used.
Resampling is applied based on 10-fold cross-validation constructed following a rolling window approach for all horizon values where a recursive strategy is used to iteratively obtain with [weigend2018time]. Ten training sets are hence constructed for each training set length and range from and the test sets from with the fold index [bergmeir2012use]. While direct strategies with a separate model for every value of can be used as well and avoid the accumulation of error, they do not take into account statistical dependencies for subsequent forecasts.
Six often-used, publicly available event logs are used: the BPI challenge of 2012 log, 2017, and 2018, the Sepsis cases event, an Italian help desk, and the Road Traffic Fine Management Process log (RTFMP) event log. Each of these logs has a diverse set of characteristics in terms of case and activity volume and average trace length, as shown in Table 3.
|Event log||# cases||# activities||Average trace length|
There are a few considerations concerning the DF time series in these event logs. Firstly, DFs of activity pairs containing endpoint activities (i.e. at the start/end of a trace) often only contain meaningful numbers at very particular parts of the series and are hard to process by longitudinal algorithms which require a more extended pattern to extract a meaningful pattern for foresting. Secondly, the equitemporal aggregation can suffer from event logs in which events do not occur frequently throughout the complete log’s timespan. For instance, the Sepsis log’s number of event occurrences tails off towards the end which can be alleviated by pre-processing (not done here to remain consistent over the event logs). Finally, suppose the level of occurrences of the DF pairs is low and close to zero. In that case, the series might be too unsuitable for analysis using white noise series analysis techniques that assume stationarity. Ideally, every time series should be evaluated using a stationarity test such as the Dickey-Fuller unit root test [leybourne1995testing], and an appropriate lag order established for differencing to ensure a white noise process is used for training. Furthermore, for each algorithm, especially for ARIMA-based models, (partial) auto-correlation has to be established to obtain the ideal and parameters. However, for the sake of simplicity and to avoid solutions where each activity pair has to have different parameters, various values are used for , , and and applied to all DF pairs where only the best-performing are reported below for comparison with the other time series techniques. The results contain the best-performing representative of each forecasting family.
All pre-processing was done in Python with a combination of pm4py111https://pm4py.fit.fraunhofer.de and the statsmodels package [seabold2010statsmodels]. The code is publicly available.222https://github.com/JohannesDeSmedt/pmf
To get a grasp of the forecasting performance in combination with the actual use of DFGs (which are rarely used in their non-aggregated form [van2019practitioner]) we present the mean absolute percentage error (MAPE) between the entropic relevance of the actual and forecasted DFGs at both full size, at 50%, and 75% reduction which is node-based (i.e. only the Q2/Q3 percentile of nodes in terms of frequency is retained). Hence, we obtain a measure of accuracy in terms of the discrepancy of the actual and forecasted model behaviour. Using different levels of aggregation also balances recall and precision, as aggregated DFGs are less precise but possibly less overfitting. The results can be found in Tables 4 to 6. NAs are reported when the algorithms did not converge, no data was available (e.g. Sepsis for the 75-100 equitemporal intervals), or extremely high values were forecasted.
When no reduction is applied, Table 4
shows that for the BPI12 and BPI17 logs, a below 10% error can be achieved, primarily for equisized aggregation. For the Italian help desk log, results are in the 10-37% bracket, while for the other logs, results are often well above a 100% deviation (with the entropic relevance of the actual DFGs being lower, hence better, than the entropic relevance of the forecasted DFGs). However, for the RTFMP and BPI18 log, results are better when more training points are used (e.g., 50 or 75 to obtain forecasts for the 50-75 and 75-100 intervals). There is no significant difference between equisized and equitemporal aggregation except for the occasional outliers. Overall, the percentage error is lower in Table5 when a reduction of 50% is applied with sub-10% results for the BPI12, Sepsis, and BPI17 logs. The results for the RTFMP log are occasionally better but mostly worse, similar to BPI18. Finally, the results in Table 6 show a further reduction of errors for the BPI12, Sepsis, BPI17, and Italian logs and a drastic decrease to close to 0% for RTFMP. The results for the BPI18 log remain bad at over 100% error rates.
These results are commensurate with the findings in [DBLP:conf/icpm/PolyvyanyyMG20], which contains entropic relevance results for the BPI12, Sepsis, and RTFMP logs, indicating that entropic relevance of larger DFGs is lower (better) for RTFMP/Sespsis, and the entropic relevance goes up strongly for small models of RTFMP meaning the drastically improved error rates reported here are for models performing worse in terms of recall and precision. The entropic relevance for the BPI12 log is stable for the full spectrum of DFG sizes as per [DBLP:conf/icpm/PolyvyanyyMG20], which is reflected in the consistently good error rates presented here. This means that the low error rates reported are produced by the reduced DFGs, which still score strongly in terms of recall and precision. Matching all results to the event log characteristics, we notice that the event logs with longer traces with medium-sized alphabets (20) such as BPI12 and BPI17 consistently report good results. The BPI18 log’s high number of activities seems to inflate error rates quickly, which is further aggravated when DFGs are reduced. Given that DFGs are based on activity pairs, this result is not surprising. For Sepsis and the Italian event logs, good error rates are obtained once DFGs are reduced, indicating that forecasting the low-frequent edges and activities might lead to high error rates when the alphabet is smaller and traces are shorter, which is potentially also caused by the lack of precision as witnessed with the RTFMP log.
Overall, there exist many scenarios in which process model forecasting is delivering solid results. For the BPI12, BPI17, Italian, and Sepsis event logs, sub-10% error rates can be achieved both for equisized and equitemporal aggregation combined with model reductions which readers of DFGs typically apply. In some cases, even a naive forecast is enough to obtain a low error rate. However, the AR and ARIMA models report the best error rates in most cases. Nevertheless, results are often close except when fewer training points are used. Then, results are often varying widely. In future work, the robustness of the forecast algorithms will be further investigated, e.g., via scrutinising the confidence intervals of the forecasted DF outcomes.
4.3 Visualising Process Model Forecasts
In Section 4.2, we evaluated forecasting results, ensuring the conformance and interpretability of the predicted process models. To that end, gaining insights from such predicted data remains a difficult task for the analyst. This section sets off to present a novel visualisation system to aid analysts in exploring the event logs. The process of designing and implementing the system started by designing several prototypes that undergone rounds of discussions to mature into the implemented visualisation system.
In this paper, we presented the first genuine approach to forecast a process model as a whole. To this end, we developed a technique based on time series analysis of DF relations to forecast entire DFGs from historical event data. In this way, we are able to make promising forecasts regarding the future development of the process, including whether process drifts or major changes might occur in particular parts of the process. The presented forecasting approach is supported by the Process Change Exploration system, which allows analysts to compare various parts of the past, present, and forecasted future behaviour of the process. Our empirical evaluation demonstrates that, most notably for reduced process models with medium-sized alphabets, we can obtain below 15% MAPE in terms of conformance to the true models.
In future research, we plan to evaluate the use of machine learning techniques for process model forecasting. More specifically, we aim at using recurrent neural networks or their extension in long short-term memory networks (LSTMs) and transformer-based architectures, as well as hybrid methods or ensemble forecasts with the traditional time series approaches presented here. Furthermore, we want to explore opportunities for enriching our forecasted process models with confidence intervals by calculating the entropic relevance at different confidence levels and reporting the confidence intervals in the PCE system. Finally, we will conduct design studies with process analysts to evaluate the usability of different visualisation techniques.