VerdictDB: Universalizing Approximate Query Processing

by   Yongjoo Park, et al.

Despite 25 years of research in academia, approximate query processing (AQP) has had little industrial adoption. One of the major causes for this slow adoption is the reluctance of traditional vendors to make radical changes to their legacy codebases, and the preoccupation of newer vendors (e.g., SQL-on-Hadoop products) with implementing standard features. On the other hand, the few AQP engines that are available are each tied to a specific platform and require users to completely abandon their existing databases---an unrealistic expectation given the infancy of the AQP technology. Therefore, we argue that a universal solution is needed: a database-agnostic approximation engine that will widen the reach of this emerging technology across various platforms. Our proposal, called VerdictDB, uses a middleware architecture that requires no changes to the backend database, and thus, can work with all off-the-shelf engines. Operating at the driver-level, VerdictDB intercepts analytical queries issued to the database and rewrites them into another query that, if executed by any standard relational engine, will yield sufficient information for computing an approximate answer. VerdictDB uses the returned result set to compute an approximate answer and error estimates, which are then passed on to the user or application. However, lack of access to the query execution layer introduces significant challenges in terms of generality, correctness, and efficiency. This paper shows how VerdictDB overcomes these challenges and delivers up to 171 times speedup (18.45 times on average) for a variety of existing engines, such as Impala, Spark SQL, and Amazon Redshift while incurring less than 2.6


page 1

page 2

page 3

page 4


Scripting Relational Database Engine Using Transducer

We allow database user to script a parallel relational database engine w...

Empowering In-Memory Relational Database Engines with Native Graph Processing

The plethora of graphs and relational data give rise to many interesting...

LevelHeaded: Making Worst-Case Optimal Joins Work in the Common Case

Pipelines combining SQL-style business intelligence (BI) queries and lin...

Parallelization of XPath Queries using Modern XQuery Processors

A practical and promising approach to parallelizing XPath queries was pr...

MonetDBLite: An Embedded Analytical Database

While traditional RDBMSes offer a lot of advantages, they require signif...

Pragmatic approach to structured data querying via natural language interface

As the use of technology increases and data analysis becomes integral in...

Approximating Aggregated SQL Queries With LSTM Networks

Despite continuous investments in data technologies, the latency of quer...

Please sign up or login with your details

Forgot password? Click here to reset