Scaling Big Data Platform for Big Data Pipeline

02/11/2019
by   Rebecca Wild, et al.
0

Monitoring and Managing High Performance Computing (HPC) systems and environments generate an ever growing amount of data. Making sense of this data and generating a platform where the data can be visualized for system administrators and management to proactively identify system failures or understand the state of the system requires the platform to be as efficient and scalable as the underlying database tools used to store and analyze the data. In this paper we will show how we leverage Accumulo, d4m, and Unity to generate a 3D visualization platform to monitor and manage the Lincoln Laboratory Supercomputer systems and how we have had to retool our approach to scale with our systems.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 2

11/06/2018

Defining Big Data Analytics Benchmarks for Next Generation Supercomputers

The design and construction of high performance computing (HPC) systems ...
06/29/2021

Scalable Traffic Predictive Analysis using GPU in Big Data

The paper adopts parallel computing systems for predictive analysis in b...
06/23/2021

Mr. Plotter: Unifying Data Reduction Techniques in Storage and Visualization Systems

As the rate of data collection continues to grow rapidly, developing vis...
09/02/2018

A Serverless Tool for Platform Agnostic Computational Experiment Management

Neuroscience has been carried into the domain of big data and high perfo...
02/22/2020

BAD to the Bone: Big Active Data at its Core

Virtually all of today's Big Data systems are passive in nature, respond...
08/06/2021

Scalable Analysis for Covid-19 and Vaccine Data

This paper explains the scalable methods used for extracting and analyzi...
06/30/2017

From Big Data to Big Displays: High-Performance Visualization at Blue Brain

Blue Brain has pushed high-performance visualization (HPV) to complement...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

References