Rigorous Assessment of Model Inference Accuracy using Language Cardinality

by   Donato Clun, et al.

Models such as finite state automata are widely used to abstract the behavior of software systems by capturing the sequences of events observable during their execution. Nevertheless, models rarely exist in practice and, when they do, get easily outdated; moreover, manually building and maintaining models is costly and error-prone. As a result, a variety of model inference methods that automatically construct models from execution traces have been proposed to address these issues. However, performing a systematic and reliable accuracy assessment of inferred models remains an open problem. Even when a reference model is given, most existing model accuracy assessment methods may return misleading and biased results. This is mainly due to their reliance on statistical estimators over a finite number of randomly generated traces, introducing avoidable uncertainty about the estimation and being sensitive to the parameters of the random trace generative process. This paper addresses this problem by developing a systematic approach based on analytic combinatorics that minimizes bias and uncertainty in model accuracy assessment by replacing statistical estimation with deterministic accuracy measures. We experimentally demonstrate the consistency and applicability of our approach by assessing the accuracy of models inferred by state-of-the-art inference tools against reference models from established specification mining benchmarks.


page 1

page 2

page 3

page 4


Heuristic-based Mining of Service Behavioral Models from Interaction Traces

Software behavioral models have proven useful for emulating and testing ...

Learning Concise Models from Long Execution Traces

Abstract models of system-level behaviour have applications in design ex...

Adversarial Specification Mining

There have been numerous studies on mining temporal specifications from ...

Mining Android App Usages for Generating Actionable GUI-based Execution Scenarios

GUI-based models extracted from Android app execution traces, events, or...

Active Learning of Abstract System Models from Traces using Model Checking [Extended]

We present a new active model-learning approach to generating abstractio...

An Empirical Study on Practicality of Specification Mining Algorithms on a Real-world Application

Dynamic model inference techniques have been the center of many research...

Please sign up or login with your details

Forgot password? Click here to reset