Log In Sign Up

EntropyDB: A Probabilistic Approach to Approximate Query Processing

by   Laurel Orr, et al.

We present EntropyDB, an interactive data exploration system that uses a probabilistic approach to generate a small, query-able summary of a dataset. Departing from traditional summarization techniques, we use the Principle of Maximum Entropy to generate a probabilistic representation of the data that can be used to give approximate query answers. We develop the theoretical framework and formulation of our probabilistic representation and show how to use it to answer queries. We then present solving techniques, give two critical optimizations to improve preprocessing time and query execution time, and explore methods to reduce query error. Lastly, we experimentally evaluate our work using a 5 GB dataset of flights within the United States and a 210 GB dataset from an astronomy particle simulation. While our current work only supports linear queries, we show that our technique can successfully answer queries faster than sampling while introducing, on average, no more error than sampling and can better distinguish between rare and nonexistent values. We also discuss extensions that can allow for data updates and linear queries over joins.


page 13

page 19


Approximate Query Processing for Group-By Queries based on Conditional Generative Models

The Group-By query is an important kind of query, which is common and wi...

Approximation with Error Bounds in Spark

We introduce a sampling framework to support approximate computing with ...

To Ship or Not to (Function) Ship (Extended version)

Sampling is often used to reduce query latency for interactive big data ...

Interactive Summarization and Exploration of Top Aggregate Query Answers

We present a system for summarization and interactive exploration of hig...

Approximating Aggregated SQL Queries With LSTM Networks

Despite continuous investments in data technologies, the latency of quer...

Combining Aggregation and Sampling (Nearly) Optimally for Approximate Query Processing

Sample-based approximate query processing (AQP) suffers from many pitfal...

Random Sampling for Group-By Queries

Random sampling has been widely used in approximate query processing on ...