AI Chat AI Image Generator AI Video Text to Speech

TonY: An Orchestrator for Distributed Machine Learning Jobs

03/24/2019

∙

by Anthony Hsu, et al.

∙

∙

Training machine learning (ML) models on large datasets requires considerable computing power. To speed up training, it is typical to distribute training across several machines, often with specialized hardware like GPUs or TPUs. Managing a distributed training job is complex and requires dealing with resource contention, distributed configurations, monitoring, and fault tolerance. In this paper, we describe TonY, an open-source orchestrator for distributed ML jobs built at LinkedIn to address these challenges.

Anthony Hsu
1 publication
Keqiu Hu
2 publications
Jonathan Hung
1 publication
Arun Suresh
2 publications
Zhe Zhang
16 publications

research

∙ 02/13/2018

SLAQ: Quality-Driven Scheduling for Distributed Machine Learning

Training machine learning (ML) models with large datasets can incur sign...

0 Haoyu Zhang, et al. ∙

research

∙ 06/09/2022

HDTorch: Accelerating Hyperdimensional Computing with GP-GPUs for Design Space Exploration

HyperDimensional Computing (HDC) as a machine learning paradigm is highl...

0 William Andrew Simon, et al. ∙

research

∙ 01/31/2023

Partitioning Distributed Compute Jobs with Reinforcement Learning and Graph Neural Networks

From natural language processing to genome sequencing, large-scale machi...

0 Christopher W. F. Parsonson, et al. ∙

research

∙ 06/28/2019

Asymptotic Network Independence in Distributed Optimization for Machine Learning

We provide a discussion of several recent results which have overcome a ...

0 Alex Olshevsky, et al. ∙

research

∙ 12/08/2014

MLitB: Machine Learning in the Browser

With few exceptions, the field of Machine Learning (ML) research has lar...

0 Edward Meeds, et al. ∙

research

∙ 05/04/2022

SMLT: A Serverless Framework for Scalable and Adaptive Machine Learning Design and Training

In today's production machine learning (ML) systems, models are continuo...

6 Ahsan Ali, et al. ∙

research

∙ 05/30/2018

Predictive Performance Modeling for Distributed Computing using Black-Box Monitoring and Machine Learning

In many domains, the previous decade was characterized by increasing dat...

0 Carl Witt, et al. ∙