Federated Word2Vec: Leveraging Federated Learning to Encourage Collaborative Representation Learning

04/19/2021

∙

Large scale contextual representation models have significantly advanced NLP in recent years, understanding the semantics of text to a degree never seen before. However, they need to process large amounts of data to achieve high-quality results. Joining and accessing all these data from multiple sources can be extremely challenging due to privacy and regulatory reasons. Federated Learning can solve these limitations by training models in a distributed fashion, taking advantage of the hardware of the devices that generate the data. We show the viability of training NLP models, specifically Word2Vec, with the Federated Learning protocol. In particular, we focus on a scenario in which a small number of organizations each hold a relatively large corpus. The results show that neither the quality of the results nor the convergence time in Federated Word2Vec deteriorates as compared to centralised Word2Vec.

READ FULL TEXT

Federated Word2Vec: Leveraging Federated Learning to Encourage Collaborative Representation Learning

A Real-time Contribution Measurement Method for Participants in Federated Learning

Adaptive Parameterization of Deep Learning Models for Federated Learning

From Distributed Machine Learning to Federated Learning: A Survey

A Privacy-Preserving Approach to Extraction of Personal Information through Automatic Annotation and Federated Learning

Collaborative Domain Blocking: Using federated NLP To Detect Malicious Domains

Federated Learning for Appearance-based Gaze Estimation in the Wild

Low-Latency Cooperative Spectrum Sensing via Truncated Vertical Federated Learning

Federated Word2Vec: Leveraging Federated Learning to Encourage Collaborative Representation Learning

Related Research

A Real-time Contribution Measurement Method for Participants in Federated Learning

Adaptive Parameterization of Deep Learning Models for Federated Learning

From Distributed Machine Learning to Federated Learning: A Survey

A Privacy-Preserving Approach to Extraction of Personal Information through Automatic Annotation and Federated Learning

Collaborative Domain Blocking: Using federated NLP To Detect Malicious Domains

Federated Learning for Appearance-based Gaze Estimation in the Wild

Low-Latency Cooperative Spectrum Sensing via Truncated Vertical Federated Learning