Learning to Encode and Classify Test Executions

by   Foivos Tsimpourlas, et al.

The challenge of automatically determining the correctness of test executions is referred to as the test oracle problem and is one of the key remaining issues for automated testing. The goal in this paper is to solve the test oracle problem in a way that is general, scalable and accurate. To achieve this, we use supervised learning over test execution traces. We label a small fraction of the execution traces with their verdict of pass or fail. We use the labelled traces to train a neural network (NN) model to learn to distinguish runtime patterns for passing versus failing executions for a given program. Our approach for building this NN model involves the following steps, 1. Instrument the program to record execution traces as sequences of method invocations and global state, 2. Label a small fraction of the execution traces with their verdicts, 3. Designing a NN component that embeds information in execution traces to fixed length vectors, 4. Design a NN model that uses the trace information for classification, 5. Evaluate the inferred classification model on unseen execution traces from the program. We evaluate our approach using case studies from different application domains: 1. Module from Ethereum Blockchain, 2. Module from PyTorch deep learning framework, 3. Microsoft SEAL encryption library components, 4. Sed stream editor, 5. Value pointer library and 6. Nine network protocols from Linux packet identifier, L7-Filter. We found the classification models for all subject programs resulted in high precision, recall and specificity, over 95 while only training with an average 9 show that the proposed neural network model is highly effective as a test oracle and is able to learn runtime patterns to distinguish passing and failing test executions for systems and tests from different application domains.



page 11


Learning Test Traces

Modern software projects include automated tests written to check the pr...

Learning Concise Models from Long Execution Traces

Abstract models of system-level behaviour have applications in design ex...

Predicting Software Tests Traces

Modern software projects include automated tests written to check the pr...

Learning from Shader Program Traces

Deep networks for image processing typically learn from RGB pixels. This...

Adversarial Specification Mining

There have been numerous studies on mining temporal specifications from ...

TraceTracker: Hardware/Software Co-Evaluation for Large-Scale I/O Workload Reconstruction

Block traces are widely used for system studies, model verifications, an...

Trex: Learning Execution Semantics from Micro-Traces for Binary Similarity

Detecting semantically similar functions – a crucial analysis capability...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.