SasCsvToolkit – A versatile parallel 'bag-of-tasks' job submission application on heterogeneous and homogeneous platforms for Big Data Analytics such as for Biomedical Inform

02/24/2021
by   Abhishek Narain Singh, et al.
0

Background: The need for big data analysis requires being able to process large data which are being held fine-tuned for usage by corporate. It is only very recently that the need for big data has caught attention for low budget corporate groups and academia who typically do not have money and resources to buy expensive licenses of big data analysis platforms such as SAS. The corporate continue to work on SAS data format largely because of systemic organizational history and that the prior codes have been built on them. The data-providers continue to thus provide data in SAS formats. Acute sudden need has arisen because of this gap of data being in SAS format and the coders not having a SAS expertise or training background as the economic and inertial forces acting of having shaped these two class of people have been different. Method: We analyze the differences and thus the need for SasCsvToolkit which helps to generate a CSV file for a SAS format data so that the data scientist can then make use of his skills in other tools that can process CSVs such as R, SPSS, or even Microsoft Excel. At the same time, it also provides conversion of CSV files to SAS format. Apart from this, a SAS database programmer always struggles in finding the right method to do a database search, exact match, substring match, except condition, filters, unique values, table joins and data mining for which the toolbox also provides template scripts to modify and use from command line. Results: The toolkit has been implemented on SLURM scheduler platform as a `bag-of-tasks` algorithm for parallel and distributed workflow though serial version has also been incorporated.

READ FULL TEXT
research
07/19/2022

Big Data and Education: using big data analytics in language learning

Working with big data using data mining tools is rapidly becoming a tren...
research
06/11/2018

A Cost-based Storage Format Selector for Materialization in Big Data Frameworks

Modern big data frameworks (such as Hadoop and Spark) allow multiple use...
research
10/23/2018

Goodness-of-Fit Tests for Large Datasets

Nowadays, data analysis in the world of Big Data is connected typically ...
research
05/25/2021

Big data and big values: When companies need to rethink themselves

In order to face the complexity of business environments and detect prio...
research
11/09/2019

DataSist: A Python-based library for easy data analysis, visualization and modeling

A large amount of data is produced every second from modern information ...
research
08/25/2018

Hyperscaling Internet Graph Analysis with D4M on the MIT SuperCloud

Detecting anomalous behavior in network traffic is a major challenge due...

Please sign up or login with your details

Forgot password? Click here to reset