DeepAI AI Chat
Log In Sign Up

Improving the Effective Utilization of Supercomputer Resources by Adding Low-Priority Containerized Jobs

by   Julia Dubenskaya, et al.

We propose an approach to utilize idle computational resources of supercomputers. The idea is to maintain an additional queue of low-priority non-parallel jobs and execute them in containers, using container migration tools to break the execution down into separate intervals. We propose a container management system that can maintain this queue and interact with the supercomputer scheduler. We conducted a series of experiments simulating supercomputer scheduler and the proposed system. The experiments demonstrate that the proposed system increases the effective utilization of supercomputer resources under most of the conditions, in some cases significantly improving the performance.


page 1

page 2

page 3

page 4


Differential Approximation and Sprinting for Multi-Priority Big Data Engines

Today's big data clusters based on the MapReduce paradigm are capable of...

Two stage cluster for resource optimization with Apache Mesos

As resource estimation for jobs is difficult, users often overestimate t...

Adaptive parallelism with RMI: Idle high-performance computing resources can be completely avoided

In practice, standard scheduling of parallel computing jobs almost alway...

Algorithms for Grey-Weighted Distance Computations

With the increasing size of datasets and demand for real time response f...

A Resourceful Coordination Approach for Multilevel Scheduling

HPC users aim to improve their execution times without particular regard...

Efficient Scheduling for Scalable Bioinformatics Analysis Platform with Microservices

With the advancement of biology and computer science, amount of bioinfor...

New Recruiter and Jobs: The Largest Enterprise Data Migration at LinkedIn

In August 2019, we introduced to our members and customers the idea of m...