A distributed data warehouse system for astroparticle physics

12/05/2018
by   Minh-Duc Nguyen, et al.
0

A distributed data warehouse system is one of the actual issues in the field of astroparticle physics. Famous experiments, such as TAIGA, KASCADE-Grande, produce tens of terabytes of data measured by their instruments. It is critical to have a smart data warehouse system on-site to store the collected data for further distribution effectively. It is also vital to provide scientists with a handy and user-friendly interface to access the collected data with proper permissions not only on-site but also online. The latter case is handy when scientists need to combine data from different experiments for analysis. In this work, we describe an approach to implementing a distributed data warehouse system that allows scientists to acquire just the necessary data from different experiments via the Internet on demand. The implementation is based on CernVM-FS with additional components developed by us to search through the whole available data sets and deliver their subsets to users' computers.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 3

page 4

08/05/2019

Data Aggregation In The Astroparticle Physics Distributed Data Storage

German-Russian Astroparticle Data Life Cycle Initiative is an internatio...
07/16/2019

Distributed data storage for modern astroparticle physics experiments

The German-Russian Astroparticle Data Life Cycle Initiative is an intern...
01/03/2019

Landscape of Big Medical Data: A Pragmatic Survey on Prioritized Tasks

Big medical data poses great challenges to life scientists, clinicians, ...
04/29/2022

Data+Shift: Supporting visual investigation of data distribution shifts by data scientists

Machine learning on data streams is increasingly more present in multipl...
12/15/2021

EDAssistant: Supporting Exploratory Data Analysis in Computational Notebooks with In-Situ Code Search and Recommendation

Using computational notebooks (e.g., Jupyter Notebook), data scientists ...
04/18/2017

HEPData: a repository for high energy physics data

The Durham High Energy Physics Database (HEPData) has been built up over...
02/09/2004

Self-Organising Networks for Classification: developing Applications to Science Analysis for Astroparticle Physics

Physics analysis in astroparticle experiments requires the capability of...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.