DeepAI AI Chat
Log In Sign Up

A Survey on Spark Ecosystem for Big Data Processing

by   Shanjiang Tang, et al.
Nankai University
National University of Singapore

With the explosive increase of big data in industry and academic fields, it is necessary to apply large-scale data processing systems to analysis Big Data. Arguably, Spark is state of the art in large-scale data computing systems nowadays, due to its good properties including generality, fault tolerance, high performance of in-memory data processing, and scalability. Spark adopts a flexible Resident Distributed Dataset (RDD) programming model with a set of provided transformation and action operators whose operating functions can be customized by users according to their applications. It is originally positioned as a fast and general data processing system. A large body of research efforts have been made to make it more efficient (faster) and general by considering various circumstances since its introduction. In this survey, we aim to have a thorough review of various kinds of optimization techniques on the generality and performance improvement of Spark. We introduce Spark programming model and computing system, discuss the pros and cons of Spark, and have an investigation and classification of various solving techniques in the literature. Moreover, we also introduce various data management and processing systems, machine learning algorithms and applications supported by Spark. Finally, we make a discussion on the open issues and challenges for large-scale in-memory data processing with Spark.


page 3

page 6

page 7

page 15

page 18

page 19

page 20

page 21


Evaluating Serverless Architecture for Big Data Enterprise Applications

In this paper, we investigate serverless computing for performing large ...

A Survey on Geographically Distributed Big-Data Processing using MapReduce

Hadoop and Spark are widely used distributed processing frameworks for l...

Big Data application in congestion detection and classification using Apache spark

With the era of big data, an explosive amount of information is now avai...

Performance Evaluation of Big Data Processing Strategies for Neuroimaging

Neuroimaging datasets are rapidly growing in size as a result of advance...

Faasm: Lightweight Isolation for Efficient Stateful Serverless Computing

Serverless computing is an excellent fit for big data processing because...

Exoshuffle: Large-Scale Shuffle at the Application Level

Shuffle is a key primitive in large-scale data processing applications. ...

A Survey on Large-scale Machine Learning

Machine learning can provide deep insights into data, allowing machines ...