Mining Message Flows from System-on-Chip Execution Traces

05/22/2020
by   MD Rubel Ahmed, et al.
Intel
University of South Florida
0

Comprehensive and well-defined specifications are necessary to perform rigorous and thorough validation of system-on-chip (SoC) designs. Message flows specify how components of an SoC design communicate and coordinate with each other to realize various system functions. Message flow specifications are essential for efficient system-level validation and debug for SoC designs. However, in practice such specifications are usually not available, often ambiguous, incomplete, or even contain errors. This paper addresses that problem by proposing a specification mining framework, FlowMiner, that automatically extracts message flows from SoC execution traces, which, unlike software traces, show a high degree of concurrency. A set of inference rules and optimization techniques are presented to improve mining performance and reduce mining complexity. Evaluation of this framework in several experiments shows promising results.

READ FULL TEXT VIEW PDF

Authors

page 1

page 2

page 3

page 4

04/29/2020

Mining Message Flows using Recurrent Neural Networks for System-on-Chip Designs

Comprehensive specifications are essential for various activities across...
03/09/2022

Deep Bidirectional Transformers for SoC Flow Specification Mining

High-quality system-level message flow specifications can lead to compre...
02/13/2021

Model Synthesis for Communication Traces of System-on-Chip Designs

Concise and abstract models of system-level behaviors are invaluable in ...
07/17/2019

A Communication-Centric Observability Selection for Post-Silicon System-on-Chip Integration Debug

Reconstruction of how components communicate with each other during syst...
03/29/2021

Adversarial Specification Mining

There have been numerous studies on mining temporal specifications from ...
05/06/2020

A Post-Silicon Trace Analysis Approach for System-on-Chip Protocol Debug

Reconstructing system-level behavior from silicon traces is a critical p...
03/01/2021

validation method to improve behavioral flows on uml requirements analysis model by cross-checking with state transition model

We propose a method to evaluate and improve the validity of required spe...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

System-on-chip (SoC) designs integrate a large number of functional blocks that communicate and coordinate with each other to implement various sophisticated functions according to system-level protocols. Experiences have shown that the implementation of those system-level protocols is the major source of various design errors. Therefore, communication-centric validation and debug methods have attracted a lot of attentions recently.

Specifications on system-level communications are essential for communication-centric validation and debug methods. In practice, such specifications are usually not available, often ambiguous, incomplete, or even contain errors. Very often, such specifications can also become outdated and disconnected from the design implementation as the design progresses.

Due to the nature of SoC designs, the SoC executions are highly concurrent where multiple system transactions are executed simultaneously. Therefore, the existing software trace mining methods are inadequate to handle SoC execution traces. In this paper, we present a specification mining framework, FlowMiner, to address the above problem. The overview of FlowMiner is shown in Fig. 1.

It takes as input a set of execution traces over messages observed in various communication links in a SoC design, and produces a set of a set of sequential patterns that satisfy certain interestingness measures and other requirements. Input traces first go through a pre-processing step where a couple of trace slicing techniques are applied for better mining performance. Next, the traces go through the actual mining process which consists of a basic mining procedure and several techniques for discovering longer patterns. In the final step, mined patterns are processed to remove redundancy, and a visual representation of these patterns is produced for user examination.

Figure 1: The overview of FlowMiner for specification mining from SoC execution traces. (1) Collect SoC execution traces (2) Process traces for mining, (3–4) to extract sequential patterns from SoC execution traces, (5) Post-processing of mined patterns.

The contributions of this paper are

  • The FlowMiner, a new specification mining framework that can automatically infer sequential patterns about the system level communication behavior from highly concurrent SoC execution traces

  • A set of inference rules that allow longer sequential patterns to be extracted without the need of the expensive mining step, thus reducing complexity greatly,

  • Comprehensive evaluation to demonstrate its effectiveness in comparison with existing tools.

The organization of this paper is as follows. Section II surveys related literature in specification mining. Section III provides required background for related concepts and formalizes the problem. Section IV describes the proposed specification mining algorithm. Section V presents the experimental results, section VI tells about observed limitations and we conclude in section VII.

2 Related Work

Specification has been a growing point of interest for mining assertions from various artifacts [1, 2, 3, 4, 5, 6, 7, 8]. In [2], a dynamic model based specification mining approach has been proposed that builds Finite State Automatons (FSA) from the execution logs. Learned FSAs can capture both temporal and data dependencies which could be used to verify API or ADT protocols. Another model based approach Synoptic [9] mines invariants from log of sequential execution traces where concurrency is recorded in partial order. It generates a FSM that satisfies the temporal invariants mined from the logs. Such model based approaches target software traces and do not perform well for concurrent hardware traces[10]. A framework has been proposed in [11] to infer temporal properties of a program as pre-chart and post-chart rules. The almost invariants mined by this tool represents total order of messages, so it may miss many partial order or concurrent scenarios which is common in SoC executions. The approaches presented in [4, 5, 6, 7, 8] mine assertions from either gate-level representations [4], or RTL models [3, 5, 6, 7, 8]. Work [12]

describes an assertion mining approach in the form of sequential patterns from the simulation traces of system level designs using data mining and supervised learning algorithms. It mines assertions relying on the system level assertions mined by another tool called

Goldmine [13] that requires the system itself to flush out spurious assertions. This flushing may not be feasible in many cases. Aside from this, these miners only focus on assertion generation, not complex execution patterns deduction, thus not much interesting for building flow specification scenarios.

Some recent approaches are trying to solve this specification mining problem taking advantage of deep learning algorithms. As the execution traces have the temporal component involved, a recurrent neural network (RNN) based approach is presented in

[14]

that utilizes Long Short-Term Memory (LSTM) models. But LSTM models handles one event at time and serialises them, when concurrency involved, serialization between two more events may not be possible, so this approach is limited to sequentially executed program traces.

The work BaySpec [10] uses a dynamic mining approach to extracts formal specifications in the form of LTL formulas from trained Baysian networks. It does not relay on some user defined template, thus may produce many useful unknown patterns.

But silicon trace specification extraction can not be done using this method, because BaySpec requires functional segmentation of traces that is starting and terminating events should be known before hand, which is very hard to know for many cases thus compromises the generalization of this tool. Perracotta [15] is another execution log analysis tool that mines temporal properties of in the form of API rules. Though it does not intend to infer model of the system, it shows a chaining technique that can be used to find sequential patterns. Such patterns can be useful for inferring message flows for SoCs. But this work is also limited by invariant mining, so it fails to capture flows with complex execution scenarios.

In spite of the fact that these mining methods can bring out impressive results, they are not suitable for highly complex SoC designs, which generally have multiple tasks executing in parallel in an interleaved manner, thus produce a complex set of execution traces. We propose a generalized case of specification mining where an execution trace is composed of sets of events. Events in the same set represent concurrency, thus no ordering relation is inferred among them. FlowMiner searches for strong temporal relation among the events of different sets and constructs flow specifications in the form of sequential patterns.

3 Background

3.1 Message Flow Specification

Figure 2: A CPU downstream write flow.

System-level protocols are often represented as message flow diagrams in SoC architecture documents, therefore the word “flow” and “protocol” are used interchangeably in our context. We also refer system flows as message flows in this paper.

A flow specification for an SoC design system is a collection of message flows. A message flow is represented as a directed acyclic graph with an example shown in Fig. 2. This flow describes how a CPU downstream write operation can be fulfilled. Formally, a flow consists a set of messages and a set of connections . Each message is a triple where the denotes the originating component of the message while dest denotes the receiving component of the message. Field cmd denotes the operation to be performed at dest. In the example shown in Fig. 2, message (CPU_x, Cache_x, wr_req) is the write request from CPU_x to Cache_x. Each message flow is associated with an unique start message, and may terminate with one or multiple different messages. A message flow may contain multiple branches describing different paths a system can execute such flow. For example, the flow shown in Fig. 2 has three branches covering the cases where the cache snoop operation is hit or miss. The flow specification for an SoC is a set of flows, denoted as .

During the execution of an SoC implements a set of flows , instances of individual flows may be generated, and executed. An instance of a flow is the execution of a path of that flow where the instances of messages on that path may include runtime information, e.g.memory addresses, which are typical. Therefore, a flow can be viewed as the type of template for its instances.

Definition 3.1

Given an SoC design that implements a set of flows , an execution of a flow instance is a sequence of mesages .

For example, the flow in Fig. 2 has three possible executions: {}, {}, and {}, where are instances of messages shown in Fig. 2.

To simplify the presentation, hereafter flows (messages) and flow instances (message instances) are used interchangeably if their meanings are clear in the context.

During execution of an SoC design, instances of flows it implements are executed concurrently in an interleaved manner. Typically, multiple instances of different flows are executed concurrently that are captured in the trace. Consequently, an SoC execution trace is a sequence of sets of message instances.

Definition 3.2

Suppose that an SoC design implements a flow specification . An SoC execution trace is

where is a set of messages observed at time , and is an message instance of an flow instance for every .

For example, a CPU_0 read flow and a CPU_1 write flow might be active at time . The idea of message sets helps to represent the outcomes of these concurrently executing flows. It is also important to note that the ordering of the messages in the same set of a trace is unknown. Therefore, given two messages and and a trace , we define if , , and .

3.2 Sequential Patterns

In this paper, we are interested in mining flows like the CPU downstream write flow shown in Fig. 2. Since a flow usually consists of one or multiple executions paths, FlowMiner aims to mine sequential patterns to characterize those execution paths, thus the entire flows. As these flows are implemented by the on-chip communication fabrics, the mined sequential patterns are invariants across different execution traces.

As a convenience, for a sequence , we use to refer to the message in , i.e. . We also use to denote the sub-sequence where , , and . We also use and to denote and , respectively. Given two sequences , and , denotes the concatenation of and , resulting in a longer sequence . Finally, we use to represent the sequence .

Definition 3.3

A sequential pattern is a sequence of messages such that

  • consists of at least two different messages,

  • All messages in are unique, and

  • meets the threshold of certain interestingness measure.

There are many different interestingness measures defined in the previous work. See [16] for a comprehensive list. This work uses the ones defined below.

Definition 3.4

The support of a message with respect to to a trace , denoted as , is the count of the instances of found in .

Definition 3.5

Given two sequences , and , the forward confidence of given with respect to  a trace is the ratio between the support of and the support of . Formally,

Forward confidence defines the likelihood of occurrence of sequence after occurrence of sequence .

Definition 3.6

Given two sequences , and , the backward confidence of given with respect to a trace is the ratio between the support of and the support of . Formally,

Backward confidence defines the likelihood of occurrence of before occurrence of the sequence . It is not used as common as the forward confidence in previous work. It is included in FlowMiner to allow patterns with branching structures like the one shown in Fig. 2 to be mined. For example, suppose that all three branches of the flow in Fig. 2 are executed. In the resulting trace, the occurrence of message leads to either or to occur next. The forward confidence of given is not . However, their backward confidence is , which allows pattern to be mined.

In this work, we require sequential patterns have either forward or backward confidence as we aim to mine invariants.

Validity of mined patterns with respect to ground truth flows GT is defined below. To facilitate the definition, GT is a set of sequences of messages, one for each path in a flow.

Definition 3.7

Let be a mined pattern. is valid if there exists a sequence such that for every two messages , in and , if , are also in , then in . Otherwise, is invalid.

The above definition indicates that a valid pattern does not possess a temporal ordering between any two messages that cannot be found in all ground truth sequences. Consider the simple example below, where is valid with respect to a GT pattern as temporal dependencies between any pair of events in also prevails in .

The flow specifications as shown in Fig. 2 also define causality relations among messages. For example, when message occurs, it causes either or to happen. The goal of specification mining is to extract such causality. We use interestingness measures as defined above to approximate the causality relations. To make such approximation more accurate, we introduce the concept of structural causality based on the observation: any message in an SoC execution trace is an output of a component in reaction to a previous input message.

Definition 3.8

Messages and satisfy the structural causality property, denoted as , if

The causality as defined above is referred to as structural to differentiate from the functional causality in the flow specifications. As an additional requirement, for a sequence to be regarded as a pattern, every two consecutive messages in that sequence must satisfy the structural causality property. Specifically, the condition below must hold.

Hereafter, we use causality to refer to the structural causality for the discussion of FlowMiner.

4 Mining Framework

The mining framework shown in Fig. 1 is implemented in Algorithm 1. The algorithm works as follows. It takes a set of execution traces over a set of messages as input, and produces a set of sequential patterns as output. First, the input traces are transformed using slicing techniques. Next, the algorithm starts with mining binary patterns (patterns of two messages). The binary patterns are then combined with a set of inference rules to generate patterns of longer lengths. In the last step, redundant patterns are removed. A pattern is redundant if there is another pattern such that the former is either a prefix or suffix of the latter. A visual representation is then produced to ease the analysis and understanding of the mined patterns by users. In the following sections, the first three steps are described.

4.1 Trace Processing

As indicated above, SoC execution traces are highly concurrent as multiple flow instances can be in execution simultaneously at a point of time. The execution of the flow instances are interleaved, which makes correlation among messages a highly challenging task. Therefore, inference of a large number of invalid patterns is a common problem in traditional mining approaches. The work in [17] presents two slicing techniques to break a long trace into a set of shorter sub-traces where messages are correlated more accurately. In FlowMiner, function implements the address-based slicing technique in [17] to achieve better mining performance. In the following, we briefly describe that slicing technique. Readers are referred to [17] for more details.

In SoC execution traces, the message instances often contain runtime information in addition to the static information as defined in the flow specification. One such information is memory addresses, which are common in memory related operations, e.g. CPU initiated memory reads or writes. Therefore, if a flow instance is initiated with a memory address, all message instances of that flow instance have the same address. In other words, if two messages carry different addresses, they should not be correlated, and thus should be separated during mining. The example below illustrates the main idea of the address-based trace slicing. Suppose a trace over 3 messages for the slicing is shown below.

In the above trace, each message instance carries a memory address in parentheses. After the slicing, the trace is broken into two sub-traces as below.

In this example, the potential correlation between and in the original trace is removed in the sliced traces, thus avoiding the false pattern between those two messages to be mined.

1 Input: A set of traces
2 Output: Set of patterns
3 ;
4 ;
5 ;
6 ;
7 ;
8 ;
Algorithm 1 FlowMiner
1 Input: A set of traces
2 Input: A set of events
3 Output: with patterns of length two
4 ;
5 ;
6 ;
7 foreach  do
8       if  then
9             ;
10            
11      if  then
12             ;
13            
Algorithm 2 Mining

4.2 Mining Patterns

Algorithm 2 implements function in Algorithm 1. It takes a set of traces , and returns two sets of binary patterns such that patterns in and have 100% forward and backward confidence, respectively. These binary patterns are building blocks to construct longer patterns. During the mining process, two messages and that satisfy the condition are considered as candidate for patterns. Otherwise, they are ignored even though the confidence between them is . Applying the causality checking leads to great reduction in the number of invalid patterns that could be mined otherwise. See Definition 3.8 for definition of the structural causality.

Algorithm 2 takes a set of traces as input. The more diverse the traces are in terms of how flows instances are interleaved, the better the mining results are in terms of precision. More specifically, two messages that have different temporal orderings in different traces are not considered as patterns. This is because we aim to mine patterns are that invariant over different execution runs.

The forward and backward confidence calculations in Definitions 3.5 and 3.6 are extended to consider a set of traces as follows. Let and be two messages. For each trace , if , the forward confidence of given on , , is undefined. Then, the forward confidence of given over a set of traces, is averaged over all traces where it is defined. The backward confidence over a set of traces can be calculated similarly.

1 Input: Candidate pattern sets ,
2 Output: Longer pattern sets , chained using three chaining cases
3 while  increases do
4      foreach  do
5           if  then
6               
7               
8               
9                while  increases do
10                     foreach  do
11                          if  then
12                              
13                              
14                              
15                               ;
16                               foreach  do
17                                    if  then
18                                        
19                                        
20                                         ;
21                                         foreach  do
22                                              if  then
23                                                   if  then
24                                                       
25                                                        if  then
26                                                            
27                                                            
28                                                            
Algorithm 3 Chaining

4.3 Chaining Patterns

Although binary patterns are interesting, they only provide limited information on SoC executions. In order to gain more comprehensive understanding about the SoC executions, which is beneficial for debugging, it is necessary to extract longer patterns. A chaining rule is described in

Perracotta [15] that constructs longer patterns by combining binary patterns. Specifically, for patterns , , , they can be combined as . However, this approach is only applicable to patterns of events that are strictly alternating. This situation rarely exists in SoC traces where messages from different flows are typically interleaved. Therefore, its effectiveness is limited. Consider the following example for an illustration.

The above two traces can be viewed as being generated by executing the pattern twice in each run. In the first run, the instances of that pattern are executed one after the other. In the second run, both instances are executed concurrently. The chaining rule in Perracotta is able to discover pattern from trace as such pattern alternates in . However, it fails to find such a pattern from as it is unable to mine , , for chaining since those binary patterns are not alternating.

In FlowMiner, several rules, as shown in Algorithm 3, are developed for chaining patterns to form longer ones. They are explained as follows.

(a) (b) (c)
Figure 3: Different flow examples to illustrate differnt chaining rules.
  1. The first rule (line 3–6 in Algorithm 3) chains two patterns from set to form longer patterns. For two patterns and to be chained, they need to share some common sub-sequence as prefix and suffix, respectively. For example in Fig. 3 Flow 1, from the trace resulting from executing the pattern, binary patterns , can be mined with forward confidence. They can be chained to obtain a longer pattern . Suppose that the pattern resulting from chaining rule 1 is . This pattern has the following property.

    Therefore, this pattern is added into set for further chaining. This chaining rule is repeatedly applied until no patterns can be chained and added to .

  2. The chaining rule 2 (line 7–10 in Algorithm 3) is similar to the rule 1 except that it considers patterns from the set . As illustrated in Fig. 3 Flow 2, from the trace resulting from executing the pattern, binary patterns and can be mined with the backward confidence of . Combining these two patterns leads to a new pattern . Suppose that the pattern resulting from chaining rule 2 is . Similarly, the following property holds for the pattern as the result of this rule.

    The pattern is added to the set . This chaining rule is repeatedly applied until no patterns can be chained and added to .

  3. The chaining rule 3 (line 11–15 in Algorithm 3) explores additional opportunities of chaining pattern from patterns in the set to those in the set . This rule is illustrated in Fig. 3 Flow 3. From the trace resulting from executing that pattern, patterns and can be mined with the backward and forward confidence of , respectively. After chaining, the resulting pattern satisfies .

    Suppose that is the result of chaining and where and using the rule 3. Also suppose that . Then, the following property holds for .

    Therefore, the resulting pattern is added into set .

4.4 Evidence-Oriented Chaining

In the previous section, we do not consider chaining patterns in set to patterns in set as we are not able to show the property of the resulting patterns similar to those for the chaining rules described above. However, there are situations that require such chaining. Consider the flows in Fig. 4 as an example. These two flows are simple, however, they share a common message . From the trace resulting from executing these two flows, the following patterns can be mined and . The chaining rules described in the last section cannot chain them together, thus missing the longer patterns in the original specification.

To address that limitation, we develop a new chaining rule (line 17–20 in Algorithm 3) to obtain more interesting and longer patterns combining patterns from to those in . The idea of this new rule is that whether a pattern in can be chained with another pattern in depends on whether the chaining result can explain some previously mined pattern. Specifically, given a pattern , and such that they share some common sequence as suffix and prefix, respectively, if , then they are chained together to form a longer pattern. The reasoning behind this rule is as follows. If we know that leads to in the future for certain, then the pattern by chaining and is an evidence to support such observation. For the example in Fig. 4, this rule can successfully mine the two original flows. The similar case exists when .

Suppose that pattern is the result from using this evidence oriented chaining rule. Then, the following property holds for .

and

Figure 4: An example of two flows that share a common message

When the first three chaining rules are used, the number of patterns may be reduced as the shorter patterns become redundant with respect to the chained patterns, and thus are removed. However, the new chaining rule always increases the total number of mined patterns. One undesirable impacts would be the large number of invalid patterns mined as the result of one invalid binary pattern. In this situation, user inputs would be invaluable to identify invalid binary patterns, and thus avoiding the evidence patterns to be generated from the beginning. In general, users are more capable of determining the validity of binary patterns than the longer ones.

5 Experimental Results

In this section, we evaluate FlowMiner on two different types of traces: synthetic traces and simulation traces of the RTL model of an SoC prototype. The purpose of synthetic traces is to provide a well specified evaluation environment that allows us to gain a better understanding of the strengths and limitations. The RTL simulation provides a more realistic environment for evaluation.

In order to evaluate the proposed mining framework, we compare FlowMiner with Perracotta [15] which is a well-known software API mining tool. Perracotta mines binary alternating patterns from traces that are sequences of software API calls. It also introduces a chaining rule that allows longer alternating patterns to be derived from the shorter ones. We use the code hosted on GitHub [18]. However, that code does not include implementation of the chaining rule. We re-implemented the chaining rule as described in [15]. Perracotta returns patterns with different satisfaction rates, the concept of which is similar to that of confidences defined in this paper. For the comparison, this paper only considers the patterns returned by Perracotta with satisfaction rate. Readers are referred to [12] for more details about Perracotta.

We use precision and recall as the performance metrics. Precision is defined as the ratio between the number of valid patterns mined versus the total number of mined patterns. Recall is defined as the ratio between the number of mined ground truth patterns111These are mined patterns that exactly match some ground truth patterns versus the number of ground truth patterns available. We try to get high recall with a corresponding high precision because if the precision is not high, mined patterns are largely invalid.

5.1 Experiments on Synthetic Traces

In the first set of experiments, we evaluate the FlowMiner over synthetic traces, which provide a clearly defined with known end results for more accurate assessments. These synthetic traces are generated based on ten flows which consist of sequential patterns. These flows are adapted and simplified from cases implemented in industry SoC designs. They include cache coherence protocols, CPU downstream read/write, peripheral upstream read/write, power management, etc. The sequential patterns used for synthetic trace generation are denoted as ground truth (GT).

To generate synthetic traces, instances of each of patterns are created and put in a pool. Then, the patterns in the pool are executed in three different ways, resulting in three different sets of synthetic traces. For the first trace set, a pattern is randomly selected from the pool, and executed atomically to the end before another pattern is selected for execution. In the resulting traces, there is no interleaving among different patterns, and there is a single message in each step of a trace. This trace set is referred to as single-event, non-interleaving. The second trace set, multiple patterns can be active at the same time, however, we still restrict a single message in each step. It is referred to as single-event, interleaved. The third trace set is generated by executing multiple patterns simultaneously, and multiple messages may occur in a single step. This set is referred to as multi-event, interleaved. Each trace set includes traces generated with random orderings among the patterns.

We apply FlowMiner and Perracotta to those three trace sets, and results are reported in Table LABEL:tab:interleaved. The objective is to evaluate the above two approaches by comparing the mined patterns against those

ground truth patterns in terms of precision and recall.

Single-message, non-interleaved traces. The experimental results for this set of traces are shown in row 1 of Table 1. Both approaches perform well in terms precision since the messages in the traces are properly ordered, thus leading to great reduction in the false correlations among messages. However, their performance is not as good in recall scores, which indicates that many ground truth patterns are not mined with either approach. In this respect, FlowMiner still does better than Perracotta by mining more valid patterns. Fig. 5 shows the distributions of mined patterns in terms of their lengths. All the patterns mined with is binary, while FlowMiner mines many more longer patterns, which are more interesting in SoC validation. Note that removes short patterns after they are used to construct longer patterns with Algorithm 3. Therefore, the number of binary patterns found with is actaully larger than what is shown in Fig. 5.

Single-message, interleaved traces. The results for this set of traces are shown in row 2 of Table 1. The comparison is striking as FlowMiner can mine over patterns while Perracotta mines only two. As indicated above, unlike software traces that Perracotta targets, SoC traces are highly concurrent with many patterns interleaved or occurred simultaneously during execution. As a result, Perracotta is not able to extract many valid patterns from SoC traces effectively as the alternating patterns impose a stronger condition on qualification of sequences as patterns.

The precision with is worse. This indicates that a large number of invalid patterns mined from the interleaved traces. Interleaving messages during a SoC execution can introduce temporal orderings among messages that do not exist in the non-interleaved case, and these temporal orderings cause FlowMiner to mine those invalid patterns. On the other hand, the strong condition of the alternating patterns mined with Perracotta can avoid many of those invalid patterns, in this case all of them. In terms of recall, FlowMiner performs better by finding the same number of the ground truth patterns, while Perracotta finds none. Fig. 6 shows the distributions of mined patterns by both of the tools in terms of different lengths. This figure shows the capability of FlowMiner in finding longer patterns from the interleaved traces.

It is interesting to notice that the finds more valid patterns from the interleaved traces than from non-interleaved traces. An explanation is that the message interleavings may introduce additional temporal orderings compared to the non-interleaved case. These additional temporal orderings may cause certain patterns belonging to branching structures of flows to be mined. A side effect of these additional temporal orderings is the larger number of invalid patterns to be found as well.

Traces Tool

#Pattern

mined

Precision Recall
single-message, non-interleaved 40 100% 18.75%
30 100% 6.25%
single-message, interleaved 109 46% 18.75%
2 100% 0.00%
multi-message, interleaved 114 43% 18.75%
Table 1: Mining from synthetic traces
Length 2 3 4 5 6 Total GT
V&F 17 14 17 0 1 49 12
IV&F 6 52 7 0 0 65
Table 2: Mined patterns of different length from multi-message synthetic traces

Multi-message, interleaved traces. This set of traces intends to simulate real SoC execution scenarios. In these traces, there may be multiple messages in each step of execution, and multiple patterns can be active at a time. Perracotta is not able to handle these traces, thus only results with FlowMiner are shown in row 4, Table 1. Table 2 shows the breakdown of the results in terms of length and validity of mined patterns. These results show that concurrency and interleaving together make mining valid and longer patterns a very challenging task where a lot of invalid patterns are often mined. For this reason, the corresponding precision is worse than the precision values in the other two cases. On the other hand, the same number of ground truth patterns, which is , is still found.

Figure 5: The length distributions of mined patterns with FlowMiner and Perracotta from single-message non-interleaved traces.
Figure 6: The length distributions of mined patterns with FlowMiner and Perracotta from single-message interleaved traces. For the bars corresponding to mining results with FlowMiner, the dark red segments show the numbers of valid patterns, while the light pink segments show the numbers of invalid patterns. In the figure, V and IV mean valid and invalid, respectively.

5.2 Experiment on Simulation Traces

Figure 7: An SoC prototype where each communication link is attached with a monitor. There can be multiple links between a pair of components.

In this experiment, we evaluate FlowMiner using the traces generated by simulating a non-trivial SoC design model [17], as shown in Fig. 7. This model has a typical structure of a realistic SoC designs. It contains two CPU cores with private caches. They are connected with a cache coherent bus. There are a number of peripheral IP blocks. All the CPU cores and peripheral blocks are connected a switch-based on-chip network, which can concurrently execute multiple flows simultaneously. This model is specified in RTL VHDL.

The on-chip network of this SoC model implements the flows used for synthetic trace generation. A random test environment is created to simulate the SoC model where CPUs and the peripheral blocks are configured to randomly select a flow to initiate with a random delay between 1 to 10 cycles. The traces from the RTL simulation are obtained, and then abstracted to traces of messages. The difference between these simulation traces and those synthetic ones is that the traces from simulation include runtime information, specifically memory addresses. We would like to find out whether such runtime information can help to improve the mining performance of FlowMiner. Another difference is that the ground truth patterns embedded in the individual simulation traces are unknown as the patterns are randomly selected during simulation. In order to calculate the precision and recall of mining results, we use the set of all ground truth patterns implemented in the SoC design model. However, the recall scores may be pessimistic as not all the ground truth patterns are presented in the traces.

In each simulation run initialized with a different random seed, a total of pattern instances are randomly selected for execution. We collect traces. We then apply address-based slicing, and derive a set of over thousand sliced traces. As the original traces contain multiple messages in certain steps, Tables 3 only shows the mining results with FlowMiner. For sliced traces, both FlowMiner and Perracotta are used, and mining results are shown in Table 4.

Table  3 shows the mined patterns with FlowMiner in terms of lengths and validity. It can be seen that there many more invalid patterns compared to the number of valid patterns. The precision in this case is about , which indicates the challenge of mining from high concurrent SoC traces. On the other hand, FlowMiner is able to mine more ground patterns, and the recall score () is better compared with synthetic experiments.

The mining results from the sliced traces are shown in Table 4. It can be seend that trace slicing has a big impact on mining performance. Compared with results in Table 3, FlowMiner is able to mine more valid patterns while avoiding all invalid patterns, therefore both precision and recall are improved. Compared with Perracotta, FlowMiner mines over twice as many patterns. Again, this is due to the strong condition imposed by mining alternating patterns in Perracotta.

Figure 8: The length distributions of mined patterns with FlowMiner and Perracotta from sliced simulation traces.
Length 2 3 4 5 6 7 8 Total GT
V&F 10 0 17 0 13 0 5 45 20
IV&F 4 12 26 79 213 259 265 859
Table 3: Mined patterns from 224(>1M messages) simulation traces.
#Patterns Mined Precision Recall
55 100% 25%
24 100% 3.12%
Table 4: Mining results from the sliced traces. FlowMiner mines ground truth patterns while Perracotta only mines 2.

5.3 Runtime Analysis

Table 5 gives the runtime information for the experiments performed in sections 5.1 and 5.2. Perracotta is written in Java, and we use the compiled version downloaded from Github. The version we use does nbot implement the chaining rule described in [19], therefore we implement it in Python as described in [20]. On the other hand, FlowMiner is written entirely in Python. All experiments are performed on a Windows PC with a quad Core-i7 processor and  GB memory.

From the table, it can be seen that FlowMiner is slower than Perracotta due to the performance disadvantage of Python compared to Java. Additionally, in FlowMiner a significant amount of time is spent on post-processing where redundant patterns are removed. If that step is removed, the runtime between FlowMiner and Perracotta in all experiments are comparable.

Traces Tools Runtime
Synthetic SM, NI FlowMiner 58
Perracotta 14
SM, I FlowMiner 60
Perracotta 13
MM, I FlowMiner 31
Perracotta
Simulation Original FlowMiner 210
Perracotta 20
Sliced FlowMiner 124
Perracotta 20
Table 5: Runtime comparison between FlowMiner and Perracotta in different experiments. Time is in second. SM, MM, I, and NI refer to "single-message", "multi-message", "interleaved", and "non-interleaved", respectively.

6 Discussions

In this section, we discuss a few observations about FlowMiner.

Mining Complex Patterns.

FlowMiner can discover strong temporal relations among messages observed in a set of traces. It uses forward and backward confidence to mine assertions which hold over all traces as we are interested in mining message flows implemented in SoC communication fabrics. The analysis of experimental results indicates that these two confidence measures, used together, allow flows with branching structures to be fully extracted from traces if those flows do not share common messages in their specifications.

On the other hand, if multiple flows with branching structures and common messages are executed, then branches of those flows may not be completely mined. One examples includes two flows as shown in Figure 2 such that one is memory write for CPU_0 (x=0), while the other is memory write for CPU_1 (x=1). These two flows share messages and for memory access. The pattern may not be mined by FlowMiner even though can be extracted. A typical technique for mining more patterns including those branching ones is to lower the confidence level. Its drawback is the rapid increase in the total number of mined patterns, majority of which are invalid. Additionally, finding a confidence level that balances the numbers of valid and invalid patterns can be tricky. For mining assertions, we focus on developing the pattern chaining techniques instead of manipulating confidence levels in this work.

Invalid Patterns.

Mining excessively large number of invalid patterns is a serious issue to limit specification mining in practical use. The main reason for mining invalid patterns is the incorrect correlations among messages in traces. FlowMiner uses address-based trace slicing and structural causality to exclude incorrect correlations from mining in a certain degree. More such techniques would be beneficial to improving mining precision.

Traces for Mining.

FlowMiner can take the any number of traces of different lengths provided as input for mining. More patterns would be mined from longer traces as those traces may include more information about different flow executions. The number of input traces can have an impact on mining precision as indicated in Section 4.2. Since temporal relations among messages extracted from traces are used approximate the causality relations, more input traces with diverse temporal orderings of flow executions can help to eliminate temporal relations that do not hold for all traces, thus reducing invalid patterns to be mined.

This work considers lossless traces, i.e. all messages of executed flows are captured in traces. To deal with traces with missing messages, we can lower the confidence levels. The side effects of that approach are explained above. Since FlowMiner mines assertions, the patterns that involves missing messages cannot be mined. In simulation environment, we think this may not be an issue. A message can be missing due to 1) a design bug, or 2) incorrect instrumentation. In the former case, a message may be missing only in a few spots in a trace. On the other hand, a message will be always missing if the instrumentation for observation is set up incorrectly. When the mining results are returned to the user for examination, the user can query the database of the mined patterns to filter out patterns that are deemed invalid, and to search for patterns that are expected to hold. If an expected pattern is not included, it signals a problem in the simulation environment. According to the above observation, the confidence levels of the sequence for a missing pattern can reveal the potential cause. If confidence levels of a missing pattern222A missing pattern refers to a sequence that is supposed to be pattern but is not mined due to missing messages. are higher than 0, it is likely caused by a design bug. Otherwise, incorrect instrumentation would be a more likely problem.

7 Conclusion

Developing an effective specification miner is a challenging endeavor. FlowMiner is such an attempt that targets mining patterns about communication behavior over communication fabrics of SoC designs. It takes SoC execution traces that are highly concurrent, and produces a set of patterns, collectively to characterize a message flow specification that describes how various components of an SoC design coordinate with each other to realize system level functions.

We are encouraged by the initial experimental results compared with two similar methods. For future improvements in terms of higher precision and recall, more effective chaining techniques and mining methods are needed. In addition, we plan to experiment FlowMiner on more diverse and realistic SoC designs and find out what and how domain specific information can help to achieve better mining results.

Acknowledgment

The research presented in this paper was partially supported by gifts from the Intel Corporation, and a grant from Cyber Florida.

References

  • [1] Matthew B. Dwyer, George S. Avrunin, and James C. Corbett. Patterns in property specifications for finite-state verification. In Proceedings of the 21st International Conference on Software Engineering, ICSE ’99, pages 411–420, 1999.
  • [2] Glenn Ammons, Rastislav Bodík, and James R. Larus. Mining specifications. In Proceedings of the 29th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL ’02, pages 4–16, 2002.
  • [3] Po-Hsien Chang and Li.-C Wang. Automatic assertion extraction via sequential data mining of simulation traces. In Proceedings of the 2010 Asia and South Pacific Design Automation Conference, ASPDAC ’10, pages 607–612, 2010.
  • [4] Wenchao Li, Alessandro Forin, and Sanjit A. Seshia. Scalable specification mining for verification and diagnosis. In Proceedings of the 47th Design Automation Conference, DAC ’10, pages 755–760, New York, NY, USA, 2010. ACM.
  • [5] Samuel Hertz, David Sheridan, and Shobha Vasudevan. Mining hardware assertions with guidance from static analysis. Trans. Comp.-Aided Des. Integ. Cir. Sys., 32(6):952–965, June 2013.
  • [6] A. Danese, F. Filini, and G. Pravadelli. A time-window based approach for dynamic assertions mining on control signals. In 2015 IFIP/IEEE International Conference on Very Large Scale Integration (VLSI-SoC), pages 246–251, Oct 2015.
  • [7] Alessandro Danese, Tara Ghasempouri, and Graziano Pravadelli. Automatic extraction of assertions from execution traces of behavioural models. In Proceedings of the 2015 Design, Automation & Test in Europe Conference & Exhibition, DATE ’15, pages 67–72, 2015.
  • [8] Alessandro Danese, Nicolò Dalla Riva, and Graziano Pravadelli. A-team: Automatic template-based assertion miner. In Proceedings of the 54th Annual Design Automation Conference 2017, DAC ’17, pages 37:1–37:6, 2017.
  • [9] Ivan Beschastnikh, Yuriy Brun, Sigurd Schneider, Michael Sloan, and Michael D. Ernst. Leveraging existing instrumentation to automatically infer invariant-constrained models. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering, ESEC/FSE ’11, pages 267–277, New York, NY, USA, 2011. ACM.
  • [10] Artur Mrowca, Martin Nocker, Sebastian Steinhorst, and Stephan Günnemann.

    Learning temporal specifications from imperfect traces using bayesian inference.

    In Proceedings of the 56th Annual Design Automation Conference 2019, DAC ’19, pages 96:1–96:6, New York, NY, USA, 2019. ACM.
  • [11] David Lo, Shahar Maoz, and Siau-Cheng Khoo. Mining modal scenario-based specifications from execution traces of reactive systems. In Proceedings of the Twenty-second IEEE/ACM International Conference on Automated Software Engineering, ASE ’07, pages 465–468, 2007.
  • [12] Lingyi Liu and Shobha Vasudevan. Automatic generation of system level assertions from transaction level models. Journal of Electronic Testing, 29(5):669–684, Oct 2013.
  • [13] S. Vasudevan, D. Sheridan, S. Patel, D. Tcheng, B. Tuohy, and D. Johnson. Goldmine: Automatic assertion generation using data mining and static analysis. In 2010 Design, Automation Test in Europe Conference Exhibition (DATE 2010), pages 626–629, March 2010.
  • [14] Tien-Duy B. Le and David Lo. Deep specification mining. In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2018, pages 106–117, New York, NY, USA, 2018. ACM.
  • [15] Jinlin Yang, David Evans, Deepali Bhardwaj, Thirumalesh Bhat, and Manuvir Das. Perracotta: mining temporal api rules from imperfect traces. In Ohio University, 2006.
  • [16] T. B. Le and D. Lo. Beyond support and confidence: Exploring interestingness measures for rule-based specification mining. In 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER), pages 331–340, March 2015.
  • [17] Yuting Cao, Parijat Mukherjee, Mahesh Ketkar, Jin Yang, and Hao Zheng. Mining Message Flows using Recurrent Neural Networks for System-on-Chip Designs. arXiv:2005.01574 [cs], April 2020. arXiv: 2005.01574.
  • [18] ModelInference/perracotta, November 2018. original-date: 2015-12-05T01:09:38Z.
  • [19] Jinlin Yang, David Evans, Deepali Bhardwaj, Thirumalesh Bhat, and Manuvir Das. Perracotta: Mining temporal api rules from imperfect traces. In Proceedings of the 28th International Conference on Software Engineering, ICSE ’06, pages 282–291, New York, NY, USA, 2006. ACM.
  • [20] Jinlin Yang. Automatic Inference and Effective Application of Temporal Specifications. Dissertation, University of Virginia, May 2007.