A Spark ML driven preprocessing approach for deep learning based scholarly data applications

11/04/2019
by   Samiya Khan, et al.
0

Big data has found applications in multiple domains. One of the largest sources of textual big data is scientific documents and papers. Big scholarly data have been used in numerous ways to create innovative applications such as collaborator discovery, expert finding and research management systems. With the advent of advanced machine and deep learning techniques, the accuracy and novelty of such applications have risen manifold. However, the biggest challenge in the development of deep learning models for scholarly applications in cloud based environment is the underutilization of resources because of the excessive time taken by textual preprocessing. This paper presents a preprocessing pipeline that makes use of Spark for data ingestion and Spark ML for pipelining preprocessing tasks. The evaluation of the proposed work is done using a case study, which uses LSTM based text summarization for generating title or summary from abstract of any research. The ingestion, preprocessing and cumulative time for the proposed approach (P3SAPP) is much lower than the conventional approach (CA), which manifests in reduction of costs as well.

READ FULL TEXT
research
10/14/2018

DPASF: A Flink Library for Streaming Data preprocessing

Data preprocessing techniques are devoted to correct or alleviate errors...
research
10/08/2021

Big Machinery Data Preprocessing Methodology for Data-Driven Models in Prognostics and Health Management

Sensor monitoring networks and advances in big data analytics have guide...
research
12/31/2019

Hiding Information in Big Data based on Deep Learning

The current approach of information hiding based on deep learning model ...
research
08/21/2017

nuts-flow/ml: data pre-processing for deep learning

Data preprocessing is a fundamental part of any machine learning applica...
research
10/22/2019

Toward Automated Website Classification by Deep Learning

In recent years, the interest in Big Data sources has been steadily grow...
research
12/20/2019

Robust Data Preprocessing for Machine-Learning-Based Disk Failure Prediction in Cloud Production Environments

To provide proactive fault tolerance for modern cloud data centers, exte...
research
02/12/2018

Embedding Complexity In the Data Representation Instead of In the Model: A Case Study Using Heterogeneous Medical Data

Electronic Health Records have become popular sources of data for second...

Please sign up or login with your details

Forgot password? Click here to reset