Development details and computational benchmarking of DEPAM

by   D. Cazau, et al.

In the big data era of observational oceanography, passive acoustics datasets are becoming too high volume to be processed on local computers due to their processor and memory limitations. As a result there is a current need for our community to turn to cloud-based distributed computing. In this paper we present a scalable computation chain for FFT (Fast Fourier Transform)-based features (e.g., Power Spectral Density) based on the Apache frameworks Hadoop and Spark. These features are at the core of many different types of acoustic analysis where the need of processing data at scale with speed is evident, e.g. serving as long-term averaged learning representations of soundscapes to identify periods of acoustic interest. In addition to a complete description of our system implementation, we also provide a computational benchmark comparing our system to different programming languages (Matlab, Python) in standalone executions, and evaluate its scalability using the speed up metric. Our current results show that our system obtains near-linear scalability in its distributed configuration for ou tested dataset, and more surprisingly, is even slightly more performant with equivalent Matlab and Python-based workflows when executed on a single node.


pPython for Parallel Python Programming

pPython seeks to provide a parallel capability that provides good speed-...

UniGPS: A Unified Programming Framework for Distributed Graph Processing

The industry and academia have proposed many distributed graph processin...

Modeling browser-based distributed evolutionary computation systems

From the era of big science we are back to the "do it yourself", where y...

Do we still need fuzzy classifiers for Small Data in the Era of Big Data?

The Era of Big Data has forced researchers to explore new distributed so...

Distributed ReliefF based Feature Selection in Spark

Feature selection (FS) is a key research area in the machine learning an...

Triclustering in Big Data Setting

In this paper, we describe versions of triclustering algorithms adapted ...

Please sign up or login with your details

Forgot password? Click here to reset