Online Analysis of High-Volume Data Streams in Astroparticle Physics

10/26/2020
by   Katharina Morik, et al.
0

Experiments in high-energy astroparticle physics produce large amounts of data as continuous high-volume streams. Gaining insights from the observed data poses a number of challenges to data analysis at various steps in the analysis chain of the experiments. Machine learning methods have already cleaved their way selectively at some particular stages of the overall data mangling process. In this paper we investigate the deployment of machine learning methods at various stages of the data analysis chain in a gamma-ray astronomy experiment. Aiming at online and real-time performance, we build up on prominent software libraries and discuss the complete cycle of data processing from raw-data capturing to high-level classification using a data-flow based rapid-prototyping environment. In the context of a gamma-ray experiment, we review user requirements in this interdisciplinary setting and demonstrate the applicability of our approach in a real-world setting to provide results from high-volume data streams in real-time performance.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/27/2020

FACT-Tools - Processing High-Volume Telescope Data

Several large experiments such as MAGIC, FACT, VERITAS, HESS or the upco...
research
07/04/2019

Development of a data infrastructure for a global data and analysis center in astroparticle physics

Nowadays astroparticle physics faces a rapid data volume increase. Meanw...
research
10/27/2020

Distributed Real-Time Data Stream Analysis for CTA

Once completed, the Cherenkov Telescope Array (CTA) will be able to map ...
research
03/24/2023

PeakNet: Bragg peak finding in X-ray crystallography experiments with U-Net

Serial crystallography at X-ray free electron laser (XFEL) sources has e...
research
04/28/2022

Standardized Evaluation of Machine Learning Methods for Evolving Data Streams

Due to the unspecified and dynamic nature of data streams, online machin...
research
06/22/2021

Real-Time XFEL Data Analysis at SLAC and NERSC: a Trial Run of Nascent Exascale Experimental Data Analysis

X-ray scattering experiments using Free Electron Lasers (XFELs) are a po...
research
12/26/2021

Acoustic scene classification using auditory datasets

The approach used not only challenges some of the fundamental mathematic...

Please sign up or login with your details

Forgot password? Click here to reset