biggy: An Implementation of Unified Framework for Big Data Management System

10/22/2018
by   Yao Wu, et al.
0

Various tools, softwares and systems are proposed and implemented to tackle the challenges in big data on different emphases, e.g., data analysis, data transaction, data query, data storage, data visualization, data privacy. In this paper, we propose datar, a new prospective and unified framework for Big Data Management System (BDMS) from the point of system architecture by leveraging ideas from mainstream computer structure. We introduce five key components of datar by reviewing the cur- rent status of BDMS. Datar features with configuration chain of pluggable engines, automatic dataflow on job pipelines, intelligent self-driving system management and interactive user interfaces. Moreover, we present biggy as an implementation of datar with manipulation details demonstrated by four running examples. Evaluations on efficiency and scalability are carried out to show the performance. Our work argues that the envisioned datar is a feasible solution to the unified framework of BDMS, which can manage big data pluggablly, automatically and intelligently with specific functionalities, where specific functionalities refer to input, storage, computation, control and output of big data.

READ FULL TEXT

page 1

page 7

research
01/15/2022

Characterizing Big Data Management

Big data management is a reality for an increasing number of organizatio...
research
12/17/2019

Securing Big Data systems, A cybersecurity management discussion

This paper explores the essential areas of cybersecurity management for ...
research
08/02/2018

Diversification on Big Data in Query Processing

Recently, in the area of big data, some popular applications such as web...
research
07/30/2019

A performance comparison of Dask and Apache Spark for data-intensive neuroimaging pipelines

In the past few years, neuroimaging has entered the Big Data era due to ...
research
07/03/2017

Version 0.1 of the BigDAWG Polystore System

A polystore system is a database management system (DBMS) composed of in...
research
02/22/2020

BAD to the Bone: Big Active Data at its Core

Virtually all of today's Big Data systems are passive in nature, respond...
research
06/23/2021

Mr. Plotter: Unifying Data Reduction Techniques in Storage and Visualization Systems

As the rate of data collection continues to grow rapidly, developing vis...

Please sign up or login with your details

Forgot password? Click here to reset