Deployment of Elastic Virtual Hybrid Clusters Across Cloud Sites

by   Miguel Caballer, et al.

Virtual clusters are widely used computing platforms than can be deployed in multiple cloud platforms. The ability to dynamically grow and shrink the number of nodes has paved the way for customised elastic computing both for High Performance Computing and High Throughput Computing workloads. However, elasticity is typically restricted to a single cloud site, thus hindering the ability to provision computational resources from multiple geographically distributed cloud sites. To this aim, this paper introduces an architecture of open-source components that coherently deploy a virtual elastic cluster across multiple cloud sites to perform large-scale computing. These hybrid virtual elastic clusters are automatically deployed and configured using an Infrastructure as Code (IaC) approach on a distributed hybrid testbed that spans different organizations, including on-premises and public clouds, supporting automated tunneling of communications across the cluster nodes with advanced VPN topologies. The results indicate that cluster-based computing of embarrassingly parallel jobs can benefit from hybrid virtual clusters that aggregate computing resources from multiple cloud back-ends and bring them together into a dedicated, albeit virtual network. The work presented in this article has been partially funded by the European Union's (EU) Horizon 2020 research project DEEP Hybrid-DataCloud (grant agreement No 777435).


page 10

page 17

page 21

page 25


An Automated Implementation of Hybrid Cloud for Performance Evaluation of Distributed Databases

A Hybrid cloud is an integration of resources between private and public...

Distributed Log Analysis on the Cloud Using MapReduce

In this paper we describe our work on designing a web based, distributed...

FedApp: a Research Sandbox for Application Orchestration in Federated Clouds using OpenStack

Multi-cluster federation is envisioned to be the next-generation cloud i...

Virtualizing the Stampede2 Supercomputer with Applications to HPC in the Cloud

Methods developed at the Texas Advanced Computing Center (TACC) are desc...

Metabolomics in the Cloud: Scaling Computational Tools to Big Data

Background: Metabolomics datasets are becoming increasingly large and co...

An efficient deception architecture for cloud-based virtual networks

Emerging deceptive systems present a new promise for the uprising securi...

Nefele: Process Orchestration for the Cloud

Virtualization, either at OS- or hardware level, plays an important role...