Federated Training of Dual Encoding Models on Small Non-IID Client Datasets

09/30/2022
by   Raviteja Vemulapalli, et al.
0

Dual encoding models that encode a pair of inputs are widely used for representation learning. Many approaches train dual encoding models by maximizing agreement between pairs of encodings on centralized training data. However, in many scenarios, datasets are inherently decentralized across many clients (user devices or organizations) due to privacy concerns, motivating federated learning. In this work, we focus on federated training of dual encoding models on decentralized data composed of many small, non-IID (independent and identically distributed) client datasets. We show that existing approaches that work well in centralized settings perform poorly when naively adapted to this setting using federated averaging. We observe that, we can simulate large-batch loss computation on individual clients for loss functions that are based on encoding statistics. Based on this insight, we propose a novel federated training approach, Distributed Cross Correlation Optimization (DCCO), which trains dual encoding models using encoding statistics aggregated across clients, without sharing individual data samples. Our experimental results on two datasets demonstrate that the proposed DCCO approach outperforms federated variants of existing approaches by a large margin.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/01/2021

Implicit Model Specialization through DAG-based Decentralized Federated Learning

Federated learning allows a group of distributed clients to train a comm...
research
12/24/2020

Decentralized Federated Learning via Mutual Knowledge Transfer

In this paper, we investigate the problem of decentralized federated lea...
research
08/18/2021

Learning Federated Representations and Recommendations with Limited Negatives

Deep retrieval models are widely used for learning entity representation...
research
09/14/2022

Federated Pruning: Improving Neural Network Efficiency with Federated Learning

Automatic Speech Recognition models require large amount of speech data ...
research
05/01/2023

Scalable Data Point Valuation in Decentralized Learning

Existing research on data valuation in federated and swarm learning focu...
research
03/17/2021

Bias-Free FedGAN: A Federated Approach to Generate Bias-Free Datasets

Federated Generative Adversarial Network (FedGAN) is a communication-eff...
research
10/11/2021

Dual Attention-Based Federated Learning for Wireless Traffic Prediction

Wireless traffic prediction is essential for cellular networks to realiz...

Please sign up or login with your details

Forgot password? Click here to reset