Analyzing Big Datasets of Genomic Sequences: Fast and Scalable Collection of k-mer Statistics

07/04/2018
by   Umberto Ferraro Petrillo, et al.
0

Distributed approaches based on the map-reduce programming paradigm have started to be proposed in the bioinformatics domain, due to the large amount of data produced by the next-generation sequencing techniques. However, the use of map-reduce and related Big Data technologies and frameworks (e.g., Apache Hadoop and Spark) does not necessarily produce satisfactory results, in terms of both efficiency and effectiveness. We discuss how the development of distributed and Big Data management technologies has affected the analysis of large datasets of biological sequences. Moreover, we show how the choice of different parameter configurations and the careful engineering of the software with respect to the specific framework under consideration may be crucial in order to achieve good performance, especially on very large amounts of data. We choose k-mers counting as a case study for our analysis, and Spark as the framework to implement FastKmer, a novel approach for the extraction of k-mer statistics from large collection of biological sequences, with arbitrary values of k. One of the most relevant contributions of FastKmer is the introduction of a module for balancing the statistics aggregation workload over the nodes of a computing cluster, in order to overcome data skew while allowing for a fully exploitation of the underly- ing distributed architecture. We also present the results of a comparative experimental analysis showing that our approach is currently the fastest among the ones based on Big Data technologies, while exhibiting a very good scalability. We provide evidence that the usage of technologies such as Hadoop or Spark for the analysis of big datasets of biological sequences is productive only if the architectural details and the peculiar aspects of the considered framework are carefully taken into account for the algorithm design and implementation.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/23/2018

Goodness-of-Fit Tests for Large Datasets

Nowadays, data analysis in the world of Big Data is connected typically ...
research
07/07/2021

Burrows Wheeler Transform on a Large Scale: Algorithms Implemented in Apache Spark

With the rapid growth of Next Generation Sequencing (NGS) technologies, ...
research
07/02/2016

A Distributed Deep Representation Learning Model for Big Image Data Classification

This paper describes an effective and efficient image classification fra...
research
07/20/2020

A Big Data Approach for Sequences Indexing on the Cloud via Burrows Wheeler Transform

Indexing sequence data is important in the context of Precision Medicine...
research
07/16/2017

Performance Evaluation of Distributed Computing Environments with Hadoop and Spark Frameworks

Recently, due to rapid development of information and communication tech...
research
07/20/2020

Collaborative Cloud Computing Framework for Health Data with Open Source Technologies

The proliferation of sensor technologies and advancements in data collec...
research
05/30/2018

A Survey of the State-of-the-Art Parallel Multiple Sequence Alignment Algorithms on Multicore Systems

Evolutionary modeling applications are the best way to provide full info...

Please sign up or login with your details

Forgot password? Click here to reset