Lauca: Generating Application-Oriented Synthetic Workloads

12/16/2019 ∙ by Yuming Li, et al. ∙ Singapore Management University 0

The synthetic workload is essential and critical to the performance evaluation of database systems. When evaluating the database performance for a specific application, the similarity between synthetic workload and real application workload determines the credibility of evaluation results. However, the workload currently used for performance evaluation is difficult to have the same workload characteristics as the target application, which leads to inaccurate evaluation results. To address this problem, we propose a workload duplicator (Lauca) that can generate synthetic workloads with highly similar performance metrics for specific applications. To the best of our knowledge, Lauca is the first application-oriented transactional workload generator. By carefully studying the application-oriented synthetic workload generation problem, we present the key workload characteristics (transaction logic and data access distribution) of online transaction processing (OLTP) applications, and propose novel workload characterization and generation algorithms, which guarantee the high fidelity of synthetic workloads. We conduct extensive experiments using workloads from TPC-C, SmallBank and micro benchmarks on both MySQL and PostgreSQL databases, and experimental results show that Lauca consistently generates high-quality synthetic workloads.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Fig. 1: The overall architecture and workflow of Lauca

Performance evaluation is essential to the development of database management systems (DBMSs). For both system and application developers, performance evaluation results are the important information for analyzing problems and making choices. In this paper, we study the problem of generating synthetic workloads for application-oriented database performance evaluation, which refers to the evaluation of database system’s capability for a specific application [25]. The application scenarios include the followings:

Database selection. DBMS is the basic support system for applications, hence the application developer needs to select an appropriate system with satisfactory performance for their target workloads. It is an effective means to evaluate candidate DBMSs by using test workloads constructed in accordance with the actual application workloads.

PoC testing. PoC (proof of concept) testing is indispensable for database vendors, especially some nascent database vendors, when marketing their products to application companies. Building test workloads that are as similar as possible to the counterpart’s applications is the most powerful guarantee for PoC conclusions.

Application-oriented performance optimization. Generally we can find that for application companies, database performance may not be as good as the expectation for online workloads; but for database vendors, developers cannot reproduce the performance problem during optimization because of lack of ideal evaluation workloads.

At present, there are three main ways to evaluate database performance, which are launched by using real online workloads, standard benchmark workloads and synthetic workloads, respectively. However, existing approaches are suboptimal for application-oriented database performance evaluation due to the following deficiencies:

Data privacy issue. Due to privacy concerns, it is often not possible for database vendors to obtain the real online workloads from their clients to do performance evaluation. In fact, for the company’s own testers, data privacy protection is also cumbersome. Therefore, the evaluations have to be done using simulation workloads.

Application oblivious. One can employ standard benchmarks to evaluate database performance without data privacy issues. Nevertheless, standard benchmarks are designed for general-purpose evaluations and their workloads are too general to imitate a specific application, and thus produce inaccurate evaluation results.

There have been some recent works on generating application-oriented synthetic workloads for online analytical processing (OLAP) applications [19, 17]. However, it is still an unexplored area against OLTP applications, where the gap motivates this work.

In this paper, we propose Lauca, a transactional workload generator for application-oriented database performance evaluation. The essential goal is that synthetic workloads generated by Lauca are as similar as possible to real application workloads, so that performance metrics evaluated by Lauca are more informative. In order to achieve this goal, we define the key workload characteristics of OLTP applications, i.e., transaction logic and data access distribution

, which determine the critical runtime behaviors of OLTP applications on database systems, including transaction conflict intensity, deadlock possibility, distributed transaction ratio and cache hit ratio. Transaction logic is first time proposed for catching the potential business logic for fine-grained workload simulation; data access distribution is described from the perspectives of skewness, dynamics and continuity to promise the reality of synthetic workloads.

The workload execution model of real-world applications is also a concern for Lauca. At present, for a considerable proportion of applications, their transaction execution is interactive, where SQL requests are sent one by one; at the same time, some of the transactions are executed using stored procedures [24]. Therefore, Lauca must be able to support these two types of transaction execution models. For this reason, we propose the concept of transaction template to allow users to more easily specify the transactions of target applications.

The overall architecture and workflow of Lauca are presented in Figure 1. We give an overview of Lauca from two aspects, which are synthetic database generation and synthetic workload generation. The inputs of Database Generator are database schema and data characteristics, among which data characteristics are extracted from the real database automatically by Data Characteristics Extractor using simple SQL queries. The inputs of Workload Generator are transaction templates and workload statistics, where transaction templates are the sketches of transactions in the target application and workload statistics mainly include transaction logic and data access distribution. By analyzing workload traces, we can extract the transaction logic and data access distribution for target application workloads. The input of Transaction Logic Analyzer is short-time (e.g., ten minutes) heavy workload traces, which contain all the parameters and return items of each SQL operation. The input of Data Access Distribution Analyzer is long-time (e.g., a full workload cycle) light workload traces, which contain only pivotal SQL parameters. To address the data privacy issue, Lauca is designed to isolate Production Environment and Evaluation Environment, as shown in Figure 1, where components involving real application data are only executed in the production environment by data owners.

To the best of our knowledge, Lauca is the first transactional workload generator for application-oriented database performance evaluation. Extensive experimental results show that transaction logic and data access distribution we proposed can effectively characterize the workloads of OLTP applications, and the deviations in performance metrics between the synthetic workloads generated by Lauca and the real application workloads are consistently less than 10%.

The rest of the paper is organized as follows: Section II provides the preliminaries including problem definition and key workload characteristics considered in this work. Section III briefly introduces the database generation. Section IV gives the formal definition and extraction algorithm of transaction logic. Section V presents three types of data access distribution in Lauca. Section VI shows the concrete implementation of workload generation. Section VII reports the experimental results of Lauca. Section VIII reviews the related studies in the literature. Finally, Section IX gives some discussions and Section X concludes the paper.

Ii Preliminaries

Ii-a Problem Definition

Before giving the formal problem definition, we enumerate the natural requirements of application-oriented database performance evaluation as follows.

Fidelity. The evaluation workloads should be highly similar to the real application workloads. Performance metrics (e.g., throughput, latency, utilizations of various physical resources, etc) obtained by the evaluation are expected to be the same as running in the real application environment. The similarity between evaluation workloads and real application workloads is measured by the deviations in performance metrics. The smaller the deviation, the higher the similarity.

Security. Data privacy protection is the basic requirement for commercial applications, therefore real data and workloads generally cannot be directly used for database performance testing.

Scalability. The target application may have huge data scale and high request concurrency/throughput. It requires that the workload generation toolset is scalable to multiple nodes and can support parallel database and workload generation.

Extensibility. In some cases, the request concurrency and throughput of current real application workload cannot meet the test requirements, so the real application workload needs to be able to be extended by our tool for generating synthetic workloads with desired scales.

Based on these requirements, we formulate the problem of application-oriented synthetic workload generation in Definition 1.

Definition 1

Application-oriented synthetic workload generation: Generating synthetic workloads that are highly similar to the target application workloads, which is not only required to have small deviations in performance metrics on the databases, but also to promise the properties of security, scalability and extensibility.

Fig. 2: Examples of transaction templates

We propose transaction templates that make it easy and simple to specify the transactions of target applications. Transaction template supports two mainstream workload execution models, i.e., interactive and stored procedure. When transactions are executed interactively, SQL operations are sent to the database one by one, and all relevant business logics for the transactions are embedded in application codes. Stored procedures can consolidate and centralize the logics that were originally implemented in application codes, and all applications call the procedures. According to the survey conducted by Pavlo [24], more than half of the actual applications do not use stored procedures. This brings a challenge. It is difficult for testers to extract precise transaction definitions like the stored procedures from long and complex application codes if stored procedures are not used. To solve this problem, the transaction template is proposed, which allows users to define the sketches of transactions in target application without specifying exact relationships among SQL operations. The transaction template is defined as follows.

Definition 2

Transaction template: It is a transaction sketch, which consists of SQL operations and branch/loop structures (if any), where the parameters in SQL operations are symbolized. The SQL operation can be a stored procedure call. All relationships among SQL operations, parameters and return items, as well as the judgment conditions in branch and loop structures are ignored.

Figure 2 presents three sample transaction templates. TX and TX are executed in an interactive manner, while TX is executed as a stored procedure. The parameters in all transaction templates are symbolized. TX contains 4 SQL operations, and TX contains branch and loop structures. If the transaction is executed as a stored procedure, the corresponding transaction template contains only one SQL operation, that is, a stored procedure call, such as TX. In addition, the stored procedure needs to be created in the test database to support evaluation.

Ii-B Workload Characteristics

In order to make the synthetic workloads and the real application workloads have the same execution cost on the same database system and thus have the same performance metrics, we must analyze which workload characteristics need our attention and control during synthetic workload generation. Notice that our work is to simulate workloads for a specific application upon a certain database system. So some factors related to system performance are fixed, such as database schema, transaction templates, the implementation mechanisms of DBMS, etc. We generate the synthetic workloads by instantiating the symbolic parameters in transaction templates.

Fig. 3: Database-side workload characteristics that may be affected by different generation strategies

A database system is mainly divided into three parts as in Figure 3, i.e., SQL Layer, Transaction Layer and Storage Layer. Since we focus on transactional workloads which commonly involve no query optimizations, the generation strategies for synthetic workloads have little effect on the execution cost of SQL layer. In the transaction layer, the workload generation strategy has a significant impact on the execution cost of Concurrency Control module. For a stand-alone database system, the main workload characteristics that may be affected here are transaction conflict intensity and deadlock possibility. If it is a distributed database system, we also need to pay attention to the impact on distributed transaction ratios. For a given database system, transaction log volume is the main performance factor to Logging and Recovery module. And the log volume is determined by transaction templates and data characteristics, which are fixed for a target application. Indexing and Buffer Management are two main modules in the storage layer. When database schema and transaction templates are fixed, the execution cost of indexing is settled. However cache hit ratio in buffer management module is heavily influenced by workload generation strategies, which shall be considered as a key workload characteristic as well.

In order to ensure that the synthetic workload and the real application workload are consistent on these four database-side workload characteristics, we define and manipulate two application-side workload characteristics, namely transaction logic and data access distribution

. Transaction logic depicts the relationship among SQL parameters and return items, and it is a representation of the potential business logic of target applications. The potential business logic, such as data items accessed by different SQL operations in a transaction satisfy a certain correlation in a probability, usually cannot be seen at a glance and need to be obtained by analyzing workload traces. Transaction logic has a big impact on transaction conflict intensity, deadlock possibility and distributed transaction ratio. The technical details about transaction logic are available in Section 

IV. Data access distribution is used to characterize the access distribution of SQL operations on data items. We analyze the workload traces to get data access distributions, namely the data distributions of SQL parameters, for parameter instantiation during the workload generation. Data access distribution has a big impact on transaction conflict intensity, deadlock possibility and cache hit ratio. Section V presents all the details about data access distribution.

Iii Database Generation

All data characteristics needed for database generation are the same as used in Touchstone [17], such as table size, column domain, column cardinality of unique values, etc. The generation of test database is actually the generation of multiple tables, while satisfying primary/foreign key constraints, as well as data characteristics of non-key columns. Introducing database generation is mainly for the completeness of Lauca functionality and the later description of synthetic workload generation, and our database generation is similar to previous work [17].

Fig. 4: Determining the domains of key columns

Without loss of generality, we assume that primary/foreign key columns are integers. The primary key is the identifier of the record and generally has no physical meaning, so we do not need to consider the data characteristics of these key columns. We simply generate primary keys sequentially and generate foreign keys within the referenced primary key domain randomly, which can promise the uniqueness of primary keys and the referential integrity of foreign keys. Before the generation of key columns, there are three steps to determine their domains, with an example shown in Figure 4. Firstly, if the primary key contains only a single column, the domain is , where is the table size (as

1
in Figure 4). Secondly, the domains of foreign key columns can be determined by the domains of referenced primary key columns (as

2
in Figure 4). Thirdly, we settle down the domains of non-foreign key columns in the composite primary key, e.g., column (as

3
in Figure 4). The most usual and reasonable situation is that there is only one non-foreign key column in the composite primary key. And the domain of this column is , where is the domain of one of the foreign key columns within the composite primary key. Other situations are handled similarly. Due to cascaded references, the second and third steps may need to be performed multiple times.

The random column generator [17], which contains a random index generator and a value transformer, is used to generate values for non-key columns, while satisfying the desired data characteristics, especially the cardinality of unique values. The output of random index generator is the integers from 1 to , where is the column cardinality. Given an index, the transformer deterministically maps it to a value in the domain of the column. We adopt different transformers based on the data type of the column. For numeric types, e.g., integer, we simply pick up a linear function which uniformly maps the index to the column domain. For string types, e.g., varchar, there are some seed strings pregenerated randomly, which satisfy the length requirements. We first select a seed string based on the input index, such as the seed string, where is the index and is the number of all seed strings. Then we concatenate the index and the selected seed string as the output value.

In summary, the generation of each table is independent of each other. And for each table, we can achieve parallel data generation with multiple threads on multiple nodes by assigning a primary key range to each thread.

Iv Transaction Logic

The transaction logic proposed in this paper is not an extension of predicate logic in previous work [6], but rather a representation of potential business logic for OLTP applications. Here, we first give an intuitive explanation of why transaction logic matters through a concrete example; then we give a formal definition of transaction logic; finally, we present how to efficiently extract transaction logic from workload traces.

Iv-a Intuitive View

Fig. 5: Transaction template TX from Figure 2

We use the sample transaction template TX in Figure 5 to illustrate the necessity of transaction logic. Referential relationships among tables , and in TX are represented in Figure 4. , , and are double typed non-key columns. is the symbolic parameter of SQL operation. When generating synthetic workloads based on TX, we usually instantiate the symbolic parameters with random values generated according to standard data distributions. However, there may be significant performance differences between such synthetic workloads and real application workloads. Now suppose we have deployed a distributed database, and tables , , and are all hash-partitioned by the first column of primary key. Due to business logics, there are usually some correlations between SQL parameters in real workloads. For example, parameters and are of the same value in a high probability (e.g., 99%) in actually. Thus the first two SQL operations in TX do not bring many distributed transactions. But in the synthetic workloads, the above two parameters are randomly generated, and they are likely to be different, which will lead to serious distributed transactions. As we know, the ratio of distributed transactions has a crucial impact on database performance [14]. More than that, if parameters and in the last two SQL operations of TX are always the same in the real workload, there will be no deadlocks. However, these two parameters may be different during the synthetic workload generation, which can lead to deadlocks.The occurrence of deadlocks can seriously affect the database performance [16]. Through this example, it is intuitive that the potential business logic of the target application, namely the transaction logic we proposed, is an important workload characteristic for synthetic workload generation, and must be guaranteed to be consistent with the real application workload.

Iv-B Definition of Transaction Logic

The workloads of actual applications vary greatly, and the potential business logic is even more difficult to depict. Therefore, it calls for a general framework to model transaction logic that abstracts from different applications. We propose to define the transaction logic using transaction structure information and parameter dependency information. Branches and loops are common structures in transactions. The probability that each branch will be executed and the average numbers that loop operations will be executed have important influence on transaction execution cost. These need to be described in the transaction structure information.

The relationship among SQL parameters and return items in a transaction describes the hidden semantics among/in SQL operations. Even if the transaction is executed as a stored procedure, the relationship among parameters of the stored procedure call may still affect the execution cost of the transaction. There are four types of relationships we care about after the investigation of existing OLTP benchmark workloads and actual application workloads around. Firstly, ‘equal relationship’ is the most common one. For example, two SQL parameters are equal under a certain probability. Secondly, ‘inclusive relationship’ is also familiar. Because the SQL result set may be a set of tuples, the value of a SQL parameter might be an element in returned values of one previous return item. Thirdly, ‘linear relationship’ refers to the relationship between two items, e.g., two parameters, that can be represented by a linear function, and it is a complement and extension to the equal relationship. Fourthly, ‘between relationship’ is proposed for predicates like ‘ ’, in which has a between relationship with . In order to simplify the problem, we only consider the relationship (i.e., dependency) between current parameter and the previous parameters (or return items).

We give the formal definition of transaction logic in Definition 3. Let’s start with some terminology. represents the SQL operation in the transaction template; (resp., ) is the return item (resp., parameter) of ; denotes the average increment of one parameter relative to another; represents the probability that the parameter dependency should be satisfied; are two coefficients for characterizing the linear function. We use the abbreviations ER, IR, LR and BR to denote the equal relationship, inclusive relationship, linear relationship and between relationship, respectively.

Definition 3

Transaction logic: For a transaction template, the transaction logic consists of transaction structure information and parameter dependency information, which are specified as:

  • Transaction structure:

    1. Execution probability of each branch in branch structures.

    2. Average number of executions for operations in loop structures.

  • Parameter dependencies for each parameter :

    1. [, , BR, ]

    2. A list of dep-item, where dep-item {
       [, , ER, ], [, , LR, , ],
       [, , ER, ], [, , LR, , ],
       [, , IR, ] };
      ; ; and if , then .

    3. A list of [, LR, , ], where must be an operation in the loop structure.

Fig. 6: Transaction template TX from Figure 2

Now, we give a more specific description to the transaction logic in Definition 3 using sample transaction templates. The transaction template TX in Figure 6 contains a branch structure and a loop structure. According to the transaction structure information defined in Definition 3, we need to know the execution probability of each branch and the average number of executions of each loop structure. This information can be obtained by analyzing workload traces. For example in TX, the branch execution probability of the first SQL is 40%, and that of the second SQL is 60%; for the loop structure, the average number of executions of the forth SQL is 10.

According to the parameter dependency information defined in Definition 3, there are three sets of dependencies, i.e., PD1-PD3, for each parameter in transaction templates. PD1 is the between dependency. If a parameter has PD1, then both PD2 and PD3 do not need to exist because can be deterministically represented by . For example in Figure 6, in TX has a between dependency [, , BR, 8], and the value of is expressed by ( + 8).

PD2 includes equal, linear and inclusive dependencies. For a parameter , it may have multiple dependencies with previous parameters or return items. Each dependency has a probability value, i.e., , which indicates the degree of dependence. If is 100%, the value of current parameter is completely determined by this dependency. For instance in Section IV-A, PD2 of parameter in TX includes [, , ER, 99%], and there is a dependency [, , ER, 100%] for parameter in TX.

Moreover, since operations in loop structures are usually performed multiple times, we need to consider the value changes of SQL parameters in these operations during loop execution. PD3 is used to represent the linear relationships between the values of the same parameter in successive runs for loop operations. For example in TX, during the loop execution of the fourth SQL, remains unchanged and is increased by 1 each time to ensure the uniqueness of the composite primary key. Therefore, we have PD3 [, LR, 100%, ] and [, LR, 100%, ] for parameters and , respectively. In addition, if a parameter that is not in an operation inside a loop structure, it can only have PD1 or PD2.

Iv-C Extraction Algorithm

The transaction logic is the representation of application business logic, so it generally does not change frequently over time. Therefore, the extraction of transaction logic does not require workload traces across a long time. Since the transaction logic analysis of each transaction template is independent of each other, the following extraction algorithm will be introduced for a single transaction template. Our extraction algorithm consists of six steps. The input is a transaction template and corresponding workload traces of transaction instances.

Step 1: Structure information. By traversing workload traces, we can count the number of executions for each operation in the transaction template, and use it to calculate the execution probability of each branch and the average numbers of executions for loop operations.

Step 2: Identify BR. First, we identify all the parameter pairs, e.g., , , that satisfy the between relationship in transaction templates, and then traverse workload traces to get the average increment, i.e., , for each pair. After that, we construct DP1 for , and the subsequent steps 3-6 can skip the processing of .

Step 3: Collect statistics for ER and IR. For each parameter in the transaction template, we traverse its previous parameters, e.g., (resp., return items, e.g., ), and count the number of transaction instances in which the pairs, e.g., , (resp., , ), satisfy the equal relationship (resp., equal relationship or inclusive relationship). Assuming that there are totally parameters together with return items in the transaction template, there are nearly numbers of above pairs. The complexity of step 3 is O().

Step 4: Collect statistics for LR. LR only involves numeric typed parameters and return items, and the return items must be from the operations based on primary key filtering. Since the calculation of coefficients for LR requires two transaction instances, we randomly select groups of transaction instances (two for each group) from transaction instances. Then we calculate the coefficients for each pair (i.e., , , , ) based on groups of transaction instances. Notice that the LR with coefficients needs to be ignored here, because it has been represented by ER. The complexity of step 4 is O().

Step 5: Determine ER, IR and LR by trade-offs. With the statistics obtained in steps 3-4, we can easily construct DP2 for each parameter . However, there may be a lot of dependencies for each parameter, and some of which may have very small probabilities, so we need to make a trade-off among these dependencies to remove noises and alleviate following calculations. We pick the most important dependencies, such as those with high probability, and ensure that the sum of probabilities of picked dependencies is less than or equal to 1.

Step 6: Construct LR for loop structure. For multiple runs of operations in a loop structure, we use DP3 to characterize the change of parameter values. By traversing workload traces, the value changes in successive runs are counted for each SQL parameter in the loop structure. The calculation of coefficients is similar to the step 4. Then based on the statistics, we construct DP3 and maintain it independently with DP2 for parameters in loop operations.

One thing to note is that if steps 3-4 encounter an operation in the loop structure, only the trace data of the first execution is used. The complexity of our extraction algorithm is dominated by steps 3-4, which is O(+). In practical evaluations, and can be set to tens of thousands, and may be tens or hundreds, so the extraction of transaction logic is very fast. In our experiments, we found that the extraction of transaction logic often takes only a few seconds with and .

V Data Access Distribution

Data access distribution has long been considered as an important application workload characteristic [12]. In this section, we present how to characterize and manipulate the skewness (Section V-A), dynamics (Section V-B) and continuity (Section V-C) of data access distribution for synthetic workload generation.

V-a Skewed Data Access Distribution

Synthetic workload generation is actually the instantiation of symbolic parameters in transaction templates. Therefore the data access distribution of synthetic workload is determined by the values of these instantiated parameters. Our analysis of data access distribution only needs the light workload traces containing pivotal parameters which are used to index the data involved in SQL operations. In the following content, we present S-Dist for describing the skewness of data access, which has a serious impact on database performance.

Fig. 7: An example for illustrating S-Dist

Without loss of generality, we assume the form of predicates which determine the data access distribution, for OLTP application workloads, can be expressed as ‘ ’. We use high frequency items (abbr., HFI) and histogram statistics

(abbr., HS) extracted from workload traces to represent S-Dist for each parameter. Histogram is a commonly used statistical method for density estimation, which are widely used to represent data distribution statistics in industry databases, such as Oracle, MySQL and PostgreSQL. In S-Dist, HFI records

hottest data items (i.e., concrete parameter values) with the highest occurrence frequency. The column domain is evenly divided into intervals, and then the frequency and cardinality of the parameters, except the ones in HFI, falling on each interval are counted for HS. Two statistics, i.e., frequency and cardinality, are used to accurately characterize the access skewness for intervals. For instance, although the frequency of an interval is not very high, the cardinality of this interval is very small, and then parameter values in this interval can still cause high conflicts. Figure 7 is an example S-Dist for an integer typed column with domain [0, 2000]. Both and are here. In HFI, the hottest item is with frequency ; in the first interval of HS, there are unique parameter values with the total access frequency .

Before using S-Dist to instantiate the symbolic parameters, we need first do data transformation for HFI, because the data items in HFI are extracted from real workload traces and may not exist in generated synthetic database. Suppose the column generator for our example is ‘index = ranInt[1, 400], value = index * 5’, where 400 is the column cardinality, as introduced in Section III. We regenerate the data items in HFI using our column generator, as shown in Figure 8, where, for example, data item 57 is replaced by 195. For ensuring that the generated parameters conform to the desired frequency distribution, we calculate a cumulative probability array based on the frequencies of high frequency items and all intervals (‘cumu prob’ in Figure 8). Then, we use random values between 0 and 1, which can be mapped to the cumulative probability array utilizing binary search, to select appropriate parameter values for filling the predicate. In Figure 8, there are two parameter generation examples. The complexity of parameter generation is O(log+), which is dominated by the binary search.

Fig. 8: An example of parameter generation

Additionally, in order to control the cardinality of parameters generated on each interval, we redefine a random index generator for parameter generation, that is ‘index = ranInt[0, cdn) / cdn * cdn + minIdx’, where cdn is the expected cardinality of the generated parameters on the target interval, cdn is the average cardinality for each interval, and minIdx is the minimum index of the target interval. The value transformer remains unchanged. For example in Figure 8, the random index generator for interval 2 is ‘index = ranInt[0, 50) / 50 * 80 + 161’, with cdn = 50, cdn = = 80, minIdx = = 161.

Although the parameter in above example is of integer type, our approach is generic. For all numeric parameters corresponding to non-key columns, the representation of S-Dist and the parameter generation are exactly the same. For parameters on key columns, there are small differences. Since the primary key in our synthetic database is generated sequentially, the domains of key columns in synthetic database may be different from that in real database. Therefore, when collecting statistics for S-Dist, we use the domains of key columns in the real database to divide the intervals and construct HS. But during synthetic workload generation, we use the domains of key columns in the synthetic database to support parameter generation. For string typed parameters, the biggest difference is how we divide the intervals. When constructing HS for a string typed parameter, the interval which it belongs to is calculated by , where is the hash code of the parameter. The parameter generation is similar to numeric types.

V-B Dynamic Data Access Distribution

S-Dist can well depict the skewness of data access distribution throughout the workload cycle. However, if the data access distribution is dynamically changing, S-Dist is inaccurate or even completely wrong. Let’s take a simple example. Suppose there is a table with records and a workload with a period of seconds. In the second, all workload requests only access the record in the table. Assume that the database throughput is stable during this period. At this time, if S-Dist is used to express the entire data access process, we actually find that there is no hot data and the data access distribution is very uniform. Obviously, this is quite different from the fact. So in this section, we propose D-Dist to characterize both dynamics and skewness of data access distribution.

In order to catch such kind of dynamics, we divide the workload trace of a parameter into multiple equal-length time windows according to the log timestamp. For the parameter trace in any time window, we generate its individual S-Dist, and the D-Dist for entire parameter traces is defined as a list of S-Dists. Assuming that the entire workload cycle is one day and the time window size is one second, D-Dist of a parameter is constructed by 24*3600=86400 S-Dists. During the generation of synthetic workloads, we instantiate symbolic parameters using the S-Dist corresponding to the generation time. In addition, for numeric parameters, the used data range in a time window may be much smaller than column domain. In order to improve the accuracy of HS, the intervals can be divided according to the data range of current window when collecting the statistics. Certainly, it is also necessary to use the corresponding index range for each interval when generating the parameters.

V-C Continuous Data Access Distribution

In some applications, the hot degree of data is closely related to time, and the specific manifestation is that data may be accessed continually for a period of time. We call this the continuity of data access distribution. For example, for online food delivery applications, steamed buns are generally ordered frequently in the morning, and coffees may be the favorite one during the afternoon. Previously, D-Dist is defined to catch the skewness of data access in time windows, while ignoring the continuity of data access between successive time windows. When using it to generate synthetic workloads, the data accessed between successive time windows may be completely different, which results in a lower cache hit ratio. So we present C-Dist to characterize the continuity, dynamics, and skewness of data access distribution.

Fig. 9: An example for illustrating C-Dist

We introduce the concept of repetition rate for C-Dist to describe the continuity of data access based on D-Dist. When collecting the statistics, we count the repetition rate between the high frequency items in current time window and those in previous time window, as well as the repetition rates of parameters for all intervals. Figure 9 continues the example in Figure 7 by adding repetition rates for both HFI and HS. In this example, the repetition rate for HFI is 0.6, that is, there are three high frequency items are kept from the previous time window. The repetition rates for five intervals in HS are 0, 0.33, 0.5, 0.46 and 0.56. Assuming that cdn is 15, then there are parameter values in interval 1 that have appeared in the previous time window.

0:  High frequency items and candidate parameters for intervals of previous time window, repetition rate for HFI, repetition rates for HS, cardinalities in HS for intervals, number of high frequency items , number of intervals
0:  High frequency items and candidate parameters for intervals of current time window
1:  Randomly select items from and put them in .
2:  Randomly generate items with the corresponding column generator and put them in .
3:  Initialize an array . The length of is .
4:  Initialize a list all parameters in .
5:  for all  parameter in  do
6:     Identify the interval number of parameter .
7:     if  is valid  then
8:         Put the parameter in ,
9:     end if // is invalid when is not in the current window domain.
10:  end for
11:  for all  interval to  do
12:     Randomly generate parameters belonging to the interval using the column generator and put them in .
13:  end for
14:  return   and
Algorithm 1 Candidate parameter generation

In order to ensure the repetition rates in C-Dist, we need to pregenerate the candidate parameters for each time window. In Algorithm 1, we list the detailed generation process of candidate parameters. In lines 1-2, we generate all the high frequency items that satisfy the expected repetition rate. In lines 3-10, we traverse all the parameters in previous time window and select the repeated parameters for each interval of current time window until the repetition rate on the interval is met. Finally, in lines 11-13, we generate random parameters added to each interval to reach the cardinality requirement. Now for parameter generation against a certain interval, we only need to randomly select a candidate parameter as the output. In addition, if the candidate parameters are generated online during the synthetic workload generation, the workload generator may become the performance bottleneck, thereby affecting the correctness of evaluation results. Therefore, we can generate the candidate parameters for all time windows offline and store them on disk, and then read them as needed when generating the synthetic workloads.

Vi Workload Generation

Given the transaction logic of each transaction template and the data access distribution of each parameter, the Workload Generator in Figure 1 is responsible for generating synthetic workloads satisfying the desired configurations. We present the details of workload generation from three levels: thread model, transaction execution and parameter instantiation.

Thread Model. Users can configurate multiple test nodes for deploying the workload generator and the number of test threads on each node. For each test thread, we establish a separate database connection. The concurrency of synthetic workload is the number of all test threads. Lauca supports two different execution models for test threads to invoke transactions: no-await-in-loop and fixed-throughput. With the no-await-in-loop setting, all test threads repeatedly issue transactions without any think time between requests. In fixed-throughput setting, the user can specify a fixed request throughput or a throughput scale factor. If the throughput scale factor is specified, we multiply the scale factor and the throughput of each time window obtained from workload traces to get the target throughput. The test threads achieve the desired throughput by controlling the think time between transaction requests. Different execution models enable us to build extended synthetic workloads.

Transaction Execution. The test thread invokes different kinds of transactions according to their proportions extracted from workload traces. And the transaction proportions are adjusted periodically with the time window. During the execution of a transaction, the structure information of its transaction logic will be used to determine whether the operations in the branch structure need to be executed, and the number of executions of operations in the loop structure. For the execution of a specific SQL operation, we first instantiate all the symbolic parameters one by one, and then send the operation with concrete parameter values to the test database. After the operation is executed, the result set and the parameters are maintained in an intermediate state for the generation of other parameters in subsequent operations within the same transaction instance.

Parameter Instantiation. When instantiating the parameters, we first guarantee the consistency of transaction logic, and then ensure the data access distribution of synthetic workloads. For a parameter, Case 1: if there is only PD1, the value of this parameter can be calculated directly based on the increment and the value of dependent parameter. Case 2: if there is only PD2, we first attempt to instantiate the parameter by randomly selecting a dependency according to their probabilities. When no dependencies are selected, then we use the data access distribution to instantiate this parameter. Case 3: if there are both PD2 and PD3, the corresponding operation must be in the loop structure. For the first loop execution, we still instantiate the parameter based on PD2 and data access distribution as Case 2. For non-first loop executions, we first attempt to instantiate the parameter using PD3 based on the probability. If no dependency in PD3 is selected, PD2 and data access distribution are then used to instantiate the parameter.

Overall, transaction execution and parameter instantiation are independent of each other for all test threads, so our workload generator can be deployed on multiple nodes to efficiently generate the high concurrency/throughput synthetic workloads while satisfying the desired workload characteristics and configurations.

Vii Experiments

(a) Throughput
(b) Latency
(c) CPU usage
(d) Disk usage
Fig. 10: Deviations in performance metrics for TPC-C workloads on PostgreSQL database
(a) Throughput
(b) Latency
(c) CPU usage
(d) Disk usage
Fig. 11: Deviations in performance metrics for SmallBank workloads on MySQL database

Environment.

Our experiments are conducted on four servers, and each server is equipped with 2 Intel Xeon Silver 4110 @ 2.1 GHz CPUs, 120 GB memory, 4 TB HDD disk configured in RAID-5 and 4 GB RAID cache. The servers are connected using 10 Gigabit Ethernet. The test platforms selected in our experiments are the most popular and advanced open source DBMSs: MySQL (v5.7.24) and PostgreSQL (v10.4), which are also widely used in industry.

Workloads. Two standard benchmarks are used throughout the experiments: TPC-C [10] and SmallBank [1]. TPC-C is one of the most widely used industrial-level OLTP benchmark. It involves nine tables and five types of transactions that simulate the activities found in complex OLTP applications. SmallBank abstracts the operations in banking applications, and it includes three tables and six types of transactions. All transactions of SmallBank perform simple read and update operations on a small number of tuples. We use OLTP-Bench [12], an extensible DBMS benchmarking testbed, to generate workloads of TPC-C and SmallBank, which act as real workloads for comparison. Workload traces are logged by OLTP-Bench and serve as input to Lauca for generating the synthetic workloads. By the way, the original TPC-C implementation of OLTP-Bench artificially binds the test threads to the warehouses, which greatly reduces the conflict of generated workloads on the database. This additional binging is removed in our experiments to make the workload more practical. Yahoo! Cloud Serving Benchmark (YCSB) [9] is a collection of workloads that represent the large-scale web applications. We construct micro benchmark workloads based on YCSB, for simulating workloads with fine-grained control on skewness, dynamics and continuity.

Settings and Setup. In the transaction logic extraction, the number of transaction instances and the group number of transaction instances are all set to . For the data access distribution extraction, the number of items in HFI and the number of intervals in HS are all set to 50, and the time window size is set to 1 second. Both MySQL and PostgreSQL are deployed on a single server, respectively. By default, Lauca/OLTP-Bench/YCSB is deployed on a single server that does not deploy any database system.

Vii-a Fidelity of Synthetic Workloads

The similarity between synthetic workloads and real application workloads is called workload fidelity, which is the most essential target when designing Lauca. It is measured by performance deviations obtained by running synthetic workloads and real application workloads on the same database system.

Figure 10-11 show the deviations in performance metrics between the synthetic workloads generated by Lauca and the real workloads generated by OLTP-Bench, under different scale factors. The concurrency of database requests is the same as the scale factor. There are two groups of experiments, respectively, for the executions of TPC-C workloads on PostgreSQL database and SmallBank workloads on MySQL database. In Figure 10(a) and Figure 11(a), we present the transaction execution throughputs of real workloads and synthetic workloads. From the results we can see that the throughputs of the two workloads are very similar, and the biggest deviation is as low as 6.29% in Figure 10(a) and 6.34% in Figure 11(a) respectively. For average latency and 95% latency metrics, in Figure 10(b) and Figure 11(b), we can see that the synthetic workload is very close to the real workload on both two metrics, with the maximum deviation between the two workloads is only 8.99% and 7.20% respectively. Figure 10(c)-10(d) and Figure 11(c)-11(d) report the CPU and Disk usages of the two workloads. The results show that the resource consumptions for executing synthetic workloads and real workloads on both PostgreSQL and MySQL databases are consistent, which further verify the high fidelity of synthetic workloads generated by Lauca.

TPC-C on MySQL SmallBank on PostgreSQL
Throughput 7.35% 9.44%
Latency -3.11% -8.63%
CPU usage -5.75% 7.50%
Disk usage -2.86% 7.33%
TABLE I: Maximum deviations in performance metrics

We present the maximum deviations in performance metrics for TPC-C workloads on MySQL database and SmallBank workloads on PostgreSQL database in Table  I. The experimental results are the same as in Figure 10-11. Overall, whether for complex workloads of TPC-C or simple workloads of SmallBank on MySQL or PostgreSQL database, the synthetic workloads generated by Lauca are consistent with the real workloads from OLTP-Bench on various performance metrics. It conforms that the synthetic workload generated by Lauca has extremely high fidelity.

Vii-B Exploring Transaction Logic

Simulation of transaction logic of application workloads is an important feature of Lauca. In this section, we demonstrate the impact of transaction logic on transaction semantics, transaction conflict intensity, deadlock possibility and distributed transaction ratio of synthetic workloads. All experiments in this section are run on MySQL database, with the workloads taken from TPC-C benchmark. Both the scale factor and request concurrency are 20. Though transaction logic consists of structure information and parameter dependency information, in which structure information, such as the number of loop executions, has an obvious impact on performance, so here we mainly explore the impact of parameter dependency information on database performance.

Fig. 12: The impact on transaction semantics

Using the TPC-C workload including all five types of transactions, we study how the transaction logic affects the transaction semantics of synthetic workloads. Figure 12 shows the throughputs of real workloads and synthetic workloads respectively generated by OLTP-Bench and Lauca. The transaction throughput is divided into two parts which are success throughput and failure throughput. Success throughput refers to the throughput of successfully executed transactions, while failure throughput refers to the throughput of failed transactions. In Figure 12, when we use all the information in transaction logic (All-On), Lauca presents excellent performance similarity to the real workload; when we turn off the parameter dependency information (Dep-Off), transaction failures increase sharply and the success throughput is much smaller than that of OLTP-Bench. This is because the insert operations in NewOrder transactions do not satisfy the primary/foreign constraints, leading to a large number of transaction rollbacks. And because no new order is generated, Delivery transactions cannot be successfully executed too. Overall, the transaction logic can effectively ensure that the transaction semantics of the synthetic workload are consistent with the real workload.

Fig. 13: The impact on transaction conflict intensity, deadlock possibility and scan data volume

To further explore the impact of transaction logic on other aspects, we focus on three types of transactions that can be successfully executed, namely the Payment, OrderStatus and StockLevel transactions. In Figure 13, we present the transaction throughputs, latencies and deadlock throughputs of workloads generated by OLTP-Bench and Lauca. There are five groups of experimental results for Lauca, which are obtained by turning off different parts of transaction logic. Note that latency here is the average latency of successfully executed transactions. From the results, we can see that when we turn off the equal parameter dependency (Equ-Off), the throughput decreases significantly, the latency increases markedly, and a lot of deadlocks occur. Increased latency indicates that there is more lock waiting time and the transaction conflict is more intensive. Occurrence of these phenomena is because there are three pairs of read and write operations involving the same record in Payment transactions. Under the Equ-Off, read and write operations on the same record are likely to become on different records, which significantly increase the likelihood of transaction conflicts and deadlocks. When we turn off the between parameter dependency (Bet-Off), the latency increases drastically, and the throughput is very low. This is because the scan operation in StockLevel transactions will involve a large amount of data, and normally only about 20 records should be accessed. When we turn off both the equal and between dependencies, the scan operation does not read a large amount of data due to the role of C-Dist, so that the performance metrics of Dep-Off, Equ&Bet-Off and Equ-Off are similar. Overall, the results confirm that manipulating transaction logic enables synthetic workloads with the same transaction conflict intensity, deadlock possibility and scan data volume as real workloads.

Fig. 14: The impact on distributed transaction ratio

The distributed transaction ratio has been proved to have a significant impact on database performance in many works [14, 18, 11]. We simulate a distributed environment on the single-node MySQL database by assuming that data is hash-partitioned into five virtual nodes according to Warehouse ID. If the data in a transaction involves multiple virtual nodes, then the transaction is considered a distributed transaction. We count the distributed transaction ratios of workloads on the application side. Figure 14 shows the distributed transaction ratios of workloads generated by OLTP-Bench and Lauca. Since OrderStatus and StockLevel are read-only transactions, the workloads used in Figure 14 are only Payment transactions. As can be seen from the results, when we turn off the parameter dependency information (Dep-Off), the distributed transaction ratio increases dramatically. This is because the Warehouse ID parameters of four write operations in Payment transactions are randomly generated under Dep-Off, thus there is a high probability of becoming distributed transactions. Overall, the control of transaction logic can effectively guarantee the similarity of distributed transaction ratio between synthetic workload and real workload.

Vii-C Exploring Data Access Distribution

This section demonstrates the ability of proposed data access distributions, namely S-Dist, D-Dist and C-Dist, to depict the skewness, dynamics and continuity of data access. Since the data access distribution of existing benchmark workloads is generally neither dynamic nor continuous, we build the evaluation workloads based on YCSB. All experiments in this section are carried out on MySQL database with a test table from YCSB. The size of test table is , and the concurrency of database request is 20.

Fig. 15: Exploring S-Dist and D-Dist for skewed and dynamic workloads

The evaluation workload in Figure 15 has only one type of transaction. The transaction consists of five pairs of read-write operations, each of which reads a record first and then updates it. The extended YCSB workload runs for 90 seconds, which is divided into three phases, and the data requests of each phase are upon records randomly selected. In the first phase, the data access distribution is Zipf distribution with parameter  = ; the second phase is still the Zipf distribution, but the parameter  = 

; and the third phase is the uniform distribution. Figure 

15 shows the dynamic changes of transaction throughput, latency and deadlock throughput for workloads generated by YCSB and Lauca. Lauca has two groups of results, corresponding to S-Dist and D-Dist. It can be seen from the results that when using D-Dist, the synthetic workload generated by Lauca is dynamically consistent with the real workload generated by YCSB on throughput, latency and deadlock, indicating that D-Dist can well depict the dynamics of workloads. Meanwhile D-Dist is represented by S-Dist in each time window, which also shows that S-Dist can well characterize the skewness of workloads. But the global S-Dist is not working well (grey lines in Figure 15), which is defined for the whole workload time and does not take into account the dynamic changes of the workload.

The evaluation workload in Figure 16 is the single-row update transaction of YCSB, by running 100 seconds with 1 second a time window. The data requests in each time window is based on random records, and the selected records for each time window are 50% coincident with the previous window. The Innodb_buffer_pool_size of MySQL is set to 16 MB. Figure 16 presents the throughputs and Innodb_buffer_pool_reads increments for workloads generated by YCSB and Lauca. Innodb_buffer_pool_reads is the number of logical reads that InnoDB cannot satisfy from the buffer pool, and have to read directly from disk. From the results, we can see that the disk access of D-Dist is significantly higher than that of YCSB, and its throughput is lower. This is because D-Dist is unable to catch the continuity of data access distribution, resulting in data requests in each time window are almost completely different with a low cache hit ratio. The performance of C-Dist is consistent with YCSB, which indicates that C-Dist can well characterize the data access continuity.

Fig. 16: Exploring C-Dist for continuous workloads

Vii-D Performance of Lauca

In this section, we use TPC-C workload traces to study the performance of Lauca, which is deployed on four servers.

Fig. 17: Transaction logic  extraction ( = )
Fig. 18: Data access distribution C-Dist extraction

Figure18 shows the execution time and memory consumptions of transaction logic extraction under different sizes of /. It can be seen from the results that when both and are , the extraction time of transaction logic is only 2.1 seconds and the memory consumption is 1.1 GB. With the increase of and , the execution time and memory consumption increase almost linearly. The experiments in Section VII-A are all conducted when and are set to , and the high fidelity of generated workloads has been proved. Overall, the transaction logic extraction in Lauca is efficient and can be done in a few seconds while guaranteeing the fidelity of synthetic workloads.

Figure18 presents the execution time and memory consumptions of C-Dist extraction for workload traces with different time length. From the results, we can see that the extraction time of C-Dist is linear with the volume of workload traces while the memory consumption is constant. This is because C-Dist is a window-based data access distribution, and the workload trace of a time window can be removed from memory after processing. In Figure18, for TPC-C workload traces with the time length of seconds (transaction throughput is 3610.3 and log volume is 33.8 GB), the extraction time of C-Dist is 678.7 seconds and the memory consumption is 1.2 GB. Since the maximum workload cycle in practical evaluations is generally one day, the result indicates that Lauca can effectively support the performance evaluation of high throughput workloads.

Viii Related Work

There are a number of benchmarks for database performance evaluation in different application areas. For OLAP applications, TPC-H, TPC-DS, and SSB [22] are the frequently-used benchmarks with defined standard database schemas and test queries. And there are TPC-C, TPC-E and SmallBank [1] benchmarks for evaluating the transaction processing capability of database systems. In addition, CH-benCHmark [8] and HTAPBench [7] can provide a unified assessment for hybrid transaction/analytical processing (HTAP) systems. However, the evaluation workloads of these standard benchmarks are abstractions for a class of applications, therefore they are too general to evaluate database performance for a specific application.

In order to obtain elaborate workloads of a target application, workload trace replaying is an optional method. Microsoft SQL Server equips two tools, i.e., SQL Server Profiler [21] and SQL Server Distributed Replay [20], for reproducing production workloads based on the SQL traces. Oracle Database Replay [13] enables users to record workload traces on the production system with minimal performance impact and then to replay a full production workload that has the same concurrency and workload characteristics as the real one. Due to data privacy issues, workload replay is difficult to apply in the practical database performance evaluation because it requires a real database state and original workload traces. Additionally, workload extension (e.g., extending concurrency) is also a problem that is difficult to solve with current replay technologies.

Then workload simulation is necessary and urgent. There are workload-aware data and query generators [4, 3, 19, 17] for database performance evaluation of OLAP applications. The input of these works generally includes database schema, basic data characteristics and size specifications for intermediate results of query trees. The output is a synthetic database instance and instantiated test queries, conforming to the specified data and workload characteristics. Workload analyzers [27, 26] are designed to study and better understand the application workloads, but neither of which can generate synthetic workloads. There are workload generators [15, 2] for database performance benchmarking. Jeong et al. [15] proposes a workload generator for simulating a realistic hardware resource consumption status. NoWog [2] introduces a workload description language for generating synthetic workloads benchmarking NoSQL databases. None of these works [15, 2] can be used to simulate the various workloads of OLTP applications for application-oriented database performance evaluation.

Ix Discussion

Data privacy protection. According to Lauca’s workflow, only workload description information and workload statistics are exposed to testers during the evaluation. Neither the data in real databases nor the logged workload traces are visible to testers. Components in Lauca that involve real application data are manipulated by data owners in the production environment. Moreover, we can also anonymize the table names, column names and transaction names in both workload description information and workload statistics. But at present, we cannot prevent doing reverse engineering on workload statistics to get sensitive information.

Limitations of transaction logic. The transaction logic in Definition 3 aims to support the common workloads in real world applications. There are still three deficiencies in our current work. Firstly, the relationships in Definition 3 are not complete. For example, the relationship between two parameters may be represented by a quadratic function, which cannot be covered currently. Secondly, only the relationship of two data items is considered at present, without considering the relationship among more data items. Thirdly, the importance of different relationships may vary greatly and it is difficult to quantify them. These issues are left to be addressed in future work.

Choosing the window size and number of intervals. The setting of the time window size depends on how often the target workload changes. If the workload changes frequently, the window size should be set to a small value, otherwise it can be set to a larger value. We recommend setting the time window size to 1 second so that even second-level workload changes can be captured. Histogram is widely used to represent data distribution statistics in industry databases, such as Oracle, MySQL and PostgreSQL. At present, the number of intervals is selected by experiments. We can apply existing academic and industrial methods [5, 23] to choose the number of intervals in future work.

X Conclusion

In this paper, we presented Lauca, a transactional workload generator for application-oriented database performance evaluation. Lauca uses transaction logic to depict the potential business logic of target applications, and data access distribution to characterize the access skewness, dynamics and continuity. Our results on various workloads and popular databases show that Lauca consistently generates high-quality synthetic workloads.

References

  • [1] M. Alomari, M. Cahill, A. Fekete, and U. Rohm (2008) The cost of serializability on platforms that use snapshot isolation. In 2008 IEEE 24th International Conference on Data Engineering, pp. 576–585. Cited by: §VII, §VIII.
  • [2] P. Ameri, N. Schlitter, J. Meyer, and A. Streit (2016) NoWog: a workload generator for database performance benchmarking. In 2016 IEEE 14th Intl Conf on Dependable, Autonomic and Secure Computing, 14th Intl Conf on Pervasive Intelligence and Computing, 2nd Intl Conf on Big Data Intelligence and Computing and Cyber Science and Technology Congress (DASC/PiCom/DataCom/CyberSciTech), pp. 666–673. Cited by: §VIII.
  • [3] A. Arasu, R. Kaushik, and J. Li (2011) Data generation using declarative constraints. In Proceedings of the 2011 ACM SIGMOD International Conference on Management of data, pp. 685–696. Cited by: §VIII.
  • [4] C. Binnig, D. Kossmann, E. Lo, and M. T. Özsu (2007) QAGen: generating query-aware test databases. In Proceedings of the 2007 ACM SIGMOD international conference on Management of data, pp. 341–352. Cited by: §VIII.
  • [5] L. Birgé and Y. Rozenholc (2006) How many bins should be put in a regular histogram. ESAIM: Probability and Statistics 10, pp. 24–45. Cited by: §IX.
  • [6] A. J. Bonner and M. Kifer (1998) A logic for programming database transactions. In Logics for databases and information systems, pp. 117–166. Cited by: §IV.
  • [7] F. Coelho, J. Paulo, R. Vilaça, J. Pereira, and R. Oliveira (2017) Htapbench: hybrid transactional and analytical processing benchmark. In Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering, pp. 293–304. Cited by: §VIII.
  • [8] R. Cole, F. Funke, L. Giakoumakis, W. Guy, A. Kemper, S. Krompass, H. Kuno, R. Nambiar, T. Neumann, M. Poess, et al. (2011) The mixed workload ch-benchmark. In Proceedings of the Fourth International Workshop on Testing Database Systems, pp. 8. Cited by: §VIII.
  • [9] B. F. Cooper, A. Silberstein, E. Tam, R. Ramakrishnan, and R. Sears (2010) Benchmarking cloud serving systems with ycsb. In Proceedings of the 1st ACM symposium on Cloud computing, pp. 143–154. Cited by: §VII.
  • [10] T. P. P. Council (1992)(Website) External Links: Link Cited by: §VII.
  • [11] C. Curino, E. Jones, Y. Zhang, and S. Madden (2010) Schism: a workload-driven approach to database replication and partitioning. Proceedings of the VLDB Endowment 3 (1-2), pp. 48–57. Cited by: §VII-B.
  • [12] D. E. Difallah, A. Pavlo, C. Curino, and P. Cudre-Mauroux (2013) Oltp-bench: an extensible testbed for benchmarking relational databases. Proceedings of the VLDB Endowment 7 (4), pp. 277–288. Cited by: §V, §VII.
  • [13] L. Galanis, S. Buranawatanachoke, R. Colle, B. Dageville, K. Dias, J. Klein, S. Papadomanolakis, L. L. Tan, V. Venkataramani, Y. Wang, et al. (2008) Oracle database replay. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pp. 1159–1170. Cited by: §VIII.
  • [14] R. Harding, D. Van Aken, A. Pavlo, and M. Stonebraker (2017) An evaluation of distributed concurrency control. Proceedings of the VLDB Endowment 10 (5), pp. 553–564. Cited by: §IV-A, §VII-B.
  • [15] H. J. Jeong and S. H. Lee (2005) A workload generator for database system benchmarks.. In iiWAS, pp. 813–822. Cited by: §VIII.
  • [16] N. Krivokapić, A. Kemper, and E. Gudes (1999) Deadlock detection in distributed database systems: a new algorithm and a comparative performance analysis. The VLDB journal 8 (2), pp. 79–100. Cited by: §IV-A.
  • [17] Y. Li, R. Zhang, X. Yang, Z. Zhang, and A. Zhou (2018) Touchstone: generating enormous query-aware test databases. In 2018 USENIX Annual Technical Conference ( 18), pp. 575–586. Cited by: §I, §III, §III, §VIII.
  • [18] Q. Lin, P. Chang, G. Chen, B. C. Ooi, K. Tan, and Z. Wang (2016) Towards a non-2pc transaction management in distributed database systems. In Proceedings of the 2016 International Conference on Management of Data, pp. 1659–1674. Cited by: §VII-B.
  • [19] E. Lo, N. Cheng, W. W. Lin, W. Hon, and B. Choi (2014) MyBenchmark: generating databases for query workloads. The VLDB Journal—The International Journal on Very Large Data Bases 23 (6), pp. 895–913. Cited by: §I, §VIII.
  • [20] Microsoft (2017)(Website) External Links: Link Cited by: §VIII.
  • [21] Microsoft (2018)(Website) External Links: Link Cited by: §VIII.
  • [22] P. O’Neil, E. O’Neil, X. Chen, and S. Revilak (2009) The star schema benchmark and augmented fact table indexing. In Technology Conference on Performance Evaluation and Benchmarking, pp. 237–252. Cited by: §VIII.
  • [23] Oracle (2017)(Website) External Links: Link Cited by: §IX.
  • [24] A. Pavlo (2017) What are we doing with our lives?: nobody cares about our concurrency control research. In Proceedings of the 2017 ACM International Conference on Management of Data, pp. 3–3. Cited by: §I, §II-A.
  • [25] M. Seltzer, D. Krinsky, K. Smith, and X. Zhang (1999) The case for application-specific benchmarking. In Proceedings of the Seventh Workshop on Hot Topics in Operating Systems, pp. 102–107. Cited by: §I.
  • [26] Q. T. Tran, K. Morfonios, and N. Polyzotis (2015) Oracle workload intelligence. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, pp. 1669–1681. Cited by: §VIII.
  • [27] P. S. Yu, M. Chen, H. Heiss, and S. Lee (1992) On workload characterization of relational database environments. IEEE transactions on Software Engineering (4), pp. 347–355. Cited by: §VIII.