Resource Utilization Monitoring for Raw Data Query Processing

by   Mayank Patel, et al.

Scientific experiments, simulations, and modern applications generate large amounts of data. Data is stored in raw format to avoid the high loading time of traditional database management systems. Researchers have proposed many techniques to improve query execution time for raw data and reduce data loading time for traditional systems. The core of all the proposed techniques is efficient utilization of resources by processing only required data or reducing operations on data. The processed data caching in the main memory or disk can resolve this issue and avoid repeated processing of data. However, limitations of resources like main memory space, storage IO speeds, and additional storage space requirements on disk need to be considered to provide reliable and scalable solutions for cloud or in-house deployments. This paper presents improvements to the raw data query processing framework by integrating a resource monitoring module. The experiments were performed using a scientific dataset known Sloan Digital Sky Survey (SDSS). Analysis of monitored resources revealed that sampling queries had the lowest resource utilization. The PostgresRAW can answer simple 0-JOIN queries faster than PostgreSQL. While one or more JOIN complex queries need to be answered using PostgreSQL to reduce workload execution time (WET). The results section discusses resource requirements of simple, complex, and sampling type queries. The result analysis of query types and resource utilization patterns assisted in proposing Query Complexity Aware (QCA) and Resource Utilization Aware (RUA) data partitioning techniques for raw engines and DBMS to reduce cost or data to result time.


page 1

page 3

page 4

page 7


Query Complexity Based Optimal Processing of Raw Data

The paper aims to find an efficient way for processing large datasets ha...

gSMat: A Scalable Sparse Matrix-based Join for SPARQL Query Processing

Resource Description Framework (RDF) has been widely used to represent i...

Join Query Optimization Techniques for Complex Event Processing Applications

Complex event processing (CEP) is a prominent technology used in many mo...

Progressive Evaluation of Queries over Tagged Data

Modern information systems often collect raw data in the form of text, i...

Progressive Evaluation of Queries over Untagged Data

Modern information systems often collect raw data in the form of text, i...

TripleID-Q: RDF Query Processing Framework using GPU

Resource Description Framework (RDF) data represents information linkage...

Code Generation Techniques for Raw Data Processing

The motivation of the current study was to design an algorithm that can ...

Please sign up or login with your details

Forgot password? Click here to reset