FedSpeech: Federated Text-to-Speech with Continual Learning

10/14/2021
by   Ziyue Jiang, et al.
0

Federated learning enables collaborative training of machine learning models under strict privacy restrictions and federated text-to-speech aims to synthesize natural speech of multiple users with a few audio training samples stored in their devices locally. However, federated text-to-speech faces several challenges: very few training samples from each speaker are available, training samples are all stored in local device of each user, and global model is vulnerable to various attacks. In this paper, we propose a novel federated learning architecture based on continual learning approaches to overcome the difficulties above. Specifically, 1) we use gradual pruning masks to isolate parameters for preserving speakers' tones; 2) we apply selective masks for effectively reusing knowledge from tasks; 3) a private speaker embedding is introduced to keep users' privacy. Experiments on a reduced VCTK dataset demonstrate the effectiveness of FedSpeech: it nearly matches multi-task training in terms of multi-speaker speech quality; moreover, it sufficiently retains the speakers' tones and even outperforms the multi-task training in the speaker similarity experiment.

READ FULL TEXT
research
08/06/2020

Improving on-device speaker verification using federated learning with privacy

Information on speaker characteristics can be useful as side information...
research
12/07/2021

Multi-speaker Emotional Text-to-speech Synthesizer

We present a methodology to train our multi-speaker emotional text-to-sp...
research
10/12/2022

Federated Continual Learning for Text Classification via Selective Inter-client Transfer

In this work, we combine the two paradigms: Federated Learning (FL) and ...
research
03/26/2021

Continual Speaker Adaptation for Text-to-Speech Synthesis

Training a multi-speaker Text-to-Speech (TTS) model from scratch is comp...
research
09/09/2021

A distillation-based approach integrating continual learning and federated learning for pervasive services

Federated Learning, a new machine learning paradigm enhancing the use of...
research
06/22/2019

Keyword Spotting for Hearing Assistive Devices Robust to External Speakers

Keyword spotting (KWS) is experiencing an upswing due to the pervasivene...
research
09/07/2023

Privacy-preserving Continual Federated Clustering via Adaptive Resonance Theory

With the increasing importance of data privacy protection, various priva...

Please sign up or login with your details

Forgot password? Click here to reset