Global Update Tracking: A Decentralized Learning Algorithm for Heterogeneous Data

05/08/2023
by   Sai Aparna Aketi, et al.
0

Decentralized learning enables the training of deep learning models over large distributed datasets generated at different locations, without the need for a central server. However, in practical scenarios, the data distribution across these devices can be significantly different, leading to a degradation in model performance. In this paper, we focus on designing a decentralized learning algorithm that is less susceptible to variations in data distribution across devices. We propose Global Update Tracking (GUT), a novel tracking-based method that aims to mitigate the impact of heterogeneous data in decentralized learning without introducing any communication overhead. We demonstrate the effectiveness of the proposed technique through an exhaustive set of experiments on various Computer Vision datasets (CIFAR-10, CIFAR-100, Fashion MNIST, and ImageNette), model architectures, and network topologies. Our experiments show that the proposed method achieves state-of-the-art performance for decentralized learning on heterogeneous data via a 1-6% improvement in test accuracy compared to other existing techniques.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/09/2021

Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data

Decentralized training of deep learning models is a key element for enab...
research
08/13/2018

COLA: Communication-Efficient Decentralized Linear Learning

Decentralized machine learning is a promising emerging paradigm in view ...
research
10/01/2019

The Non-IID Data Quagmire of Decentralized Machine Learning

Many large-scale machine learning (ML) applications need to train ML mod...
research
02/09/2021

Consensus Control for Decentralized Deep Learning

Decentralized training of deep learning models enables on-device learnin...
research
02/23/2022

Better Modelling Out-of-Distribution Regression on Distributed Acoustic Sensor Data Using Anchored Hidden State Mixup

Generalizing the application of machine learning models to situations wh...
research
08/25/2023

Heterogeneous Decentralized Machine Unlearning with Seed Model Distillation

As some recent information security legislation endowed users with uncon...
research
09/28/2022

Neighborhood Gradient Clustering: An Efficient Decentralized Learning Method for Non-IID Data Distributions

Decentralized learning over distributed datasets can have significantly ...

Please sign up or login with your details

Forgot password? Click here to reset