Distributed Supervised Learning using Neural Networks

by   Simone Scardapane, et al.

Distributed learning is the problem of inferring a function in the case where training data is distributed among multiple geographically separated sources. Particularly, the focus is on designing learning strategies with low computational requirements, in which communication is restricted only to neighboring agents, with no reliance on a centralized authority. In this thesis, we analyze multiple distributed protocols for a large number of neural network architectures. The first part of the thesis is devoted to a definition of the problem, followed by an extensive overview of the state-of-the-art. Next, we introduce different strategies for a relatively simple class of single layer neural networks, where a linear output layer is preceded by a nonlinear layer, whose weights are stochastically assigned in the beginning of the learning process. We consider both batch and sequential learning, with horizontally and vertically partitioned data. In the third part, we consider instead the more complex problem of semi-supervised distributed learning, where each agent is provided with an additional set of unlabeled training samples. We propose two different algorithms based on diffusion processes for linear support vector machines and kernel ridge regression. Subsequently, the fourth part extends the discussion to learning with time-varying data (e.g. time-series) using recurrent neural networks. We consider two different families of networks, namely echo state networks (extending the algorithms introduced in the second part), and spline adaptive filters. Overall, the algorithms presented throughout the thesis cover a wide range of possible practical applications, and lead the way to numerous future extensions, which are briefly summarized in the conclusive chapter.


page 21

page 25


A Tutorial on Distributed (Non-Bayesian) Learning: Problem, Algorithms and Results

We overview some results on distributed learning with focus on a family ...

Evolino for recurrent support vector machines

Traditional Support Vector Machines (SVMs) need pre-wired finite time wi...

Neural Networks Regularization Through Representation Learning

Neural network models and deep models are one of the leading and state o...

Inferring linear and nonlinear Interaction networks using neighborhood support vector machines

In this paper, we consider modelling interaction between a set of variab...

Distributed Weight Balancing in Directed Topologies

This doctoral thesis concerns novel distributed algorithms for weight ba...

Collaborative Learning over Wireless Networks: An Introductory Overview

In this chapter, we will mainly focus on collaborative training across w...

Please sign up or login with your details

Forgot password? Click here to reset