Development details and computational benchmarking of DEPAM
share
In the big data era of observational oceanography, passive acoustics datasets are becoming too high volume to be processed on local computers due to their processor and memory limitations. As a result there is a current need for our community to turn to cloud-based distributed computing. In this paper we present a scalable computation chain for FFT (Fast Fourier Transform)-based features (e.g., Power Spectral Density) based on the Apache frameworks Hadoop and Spark. These features are at the core of many different types of acoustic analysis where the need of processing data at scale with speed is evident, e.g. serving as long-term averaged learning representations of soundscapes to identify periods of acoustic interest. In addition to a complete description of our system implementation, we also provide a computational benchmark comparing our system to different programming languages (Matlab, Python) in standalone executions, and evaluate its scalability using the speed up metric. Our current results show that our system obtains near-linear scalability in its distributed configuration for ou tested dataset, and more surprisingly, is even slightly more performant with equivalent Matlab and Python-based workflows when executed on a single node.
READ FULL TEXT