Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings

02/27/2020
by   Tarique Siddiqui, et al.
0

Query processing over big data is ubiquitous in modern clouds, where the system takes care of picking both the physical query execution plans and the resources needed to run those plans, using a cost-based query optimizer. A good cost model, therefore, is akin to better resource efficiency and lower operational costs. Unfortunately, the production workloads at Microsoft show that costs are very complex to model for big data systems. In this work, we investigate two key questions: (i) can we learn accurate cost models for big data systems, and (ii) can we integrate the learned models within the query optimizer. To answer these, we make three core contributions. First, we exploit workload patterns to learn a large number of individual cost models and combine them to achieve high accuracy and coverage over a long period. Second, we propose extensions to Cascades framework to pick optimal resources, i.e, number of containers, during query planning. And third, we integrate the learned cost models within the Cascade-style query optimizer of SCOPE at Microsoft. We evaluate the resulting system, Cleo, in a production environment using both production and TPC-H workloads. Our results show that the learned cost models are 2 to 3 orders of magnitude more accurate, and 20X more correlated with the actual runtimes, with a large majority (70 substantial improvements in latency as well as resource usage.

READ FULL TEXT
research
10/24/2022

Deploying a Steered Query Optimizer in Production at Microsoft

Modern analytical workloads are highly heterogeneous and massively compl...
research
06/15/2019

Query and Resource Optimizations: A Case for Breaking the Wall in Big Data Systems

Modern big data systems run on cloud environments where resources are sh...
research
07/05/2022

Fine-Grained Modeling and Optimization for Intelligent Resource Management in Big Data Processing

Big data processing at the production scale presents a highly complex en...
research
10/05/2021

Phoebe: A Learning-based Checkpoint Optimizer

Easy-to-use programming interfaces paired with cloud-scale processing en...
research
04/07/2019

Neo: A Learned Query Optimizer

Query optimization is one of the most challenging problems in database s...
research
04/10/2023

COOOL: A Learning-To-Rank Approach for SQL Hint Recommendations

Query optimization is a pivotal part of every database management system...
research
12/18/2018

DeepLens: Towards a Visual Data Management System

Advances in deep learning have greatly widened the scope of automatic co...

Please sign up or login with your details

Forgot password? Click here to reset