A hybrid architecture for astronomical computing

01/19/2018
by   Changhua Li, et al.
0

With many large science equipment constructing and putting into use, astronomy has stepped into the big data era. The new method and infrastructure of big data processing has become a new requirement of many astronomers. Cloud computing, Map/Reduce, Hadoop, Spark, etc. many new technology has sprung up in recent years. Comparing to the high performance computing(HPC), Data is the center of these new technology. So, a new computing architecture infrastructure is necessary, which can be shared by both HPC and big data processing. Based on Astronomy Cloud project of Chinese Virtual Observatory (China-VO), we have made much efforts to optimize the designation of the hybrid computing platform. which include the hardware architecture, cluster management, Job and Resource scheduling.

READ FULL TEXT
research
12/03/2018

Resource Management and Scheduling for Big Data Applications in Cloud Computing Environments

This chapter presents software architectures of the big data processing ...
research
05/09/2023

High-throughput Cotton Phenotyping Big Data Pipeline Lambda Architecture Computer Vision Deep Neural Networks

In this study, we propose a big data pipeline for cotton bloom detection...
research
03/02/2022

Providing A Compiler Technology-Based Alternative For Big Data Application Infrastructures

The unprecedented growth of data volumes has caused traditional approach...
research
08/11/2020

DV-ARPA: Data Variety Aware Resource Provisioning for Big Data Processing in Accumulative Applications

In Cloud Computing, the resource provisioning approach used has a great ...
research
05/01/2018

SAGE: Percipient Storage for Exascale Data Centric Computing

We aim to implement a Big Data/Extreme Computing (BDEC) capable system i...
research
07/06/2018

The SAGE Project: a Storage Centric Approach for Exascale Computing

SAGE (Percipient StorAGe for Exascale Data Centric Computing) is a Europ...
research
04/05/2018

Big enterprise registration data imputation: Supporting spatiotemporal analysis of industries in China

Big, fine-grained enterprise registration data that includes time and lo...

Please sign up or login with your details

Forgot password? Click here to reset