Distributed Learning in the Non-Convex World: From Batch to Streaming Data, and Beyond

by   Tsung-Hui Chang, et al.

Distributed learning has become a critical enabler of the massively connected world envisioned by many. This article discusses four key elements of scalable distributed processing and real-time intelligence — problems, data, communication and computation. Our aim is to provide a fresh and unique perspective about how these elements should work together in an effective and coherent manner. In particular, we provide a selective review about the recent techniques developed for optimizing non-convex models (i.e., problem classes), processing batch and streaming data (i.e., data types), over the networks in a distributed manner (i.e., communication and computation paradigm). We describe the intuitions and connections behind a core set of popular distributed algorithms, emphasizing how to trade off between computation and communication costs. Practical issues and future research directions will also be discussed.



There are no comments yet.


page 7


Scaling-up Distributed Processing of Data Streams for Machine Learning

Emerging applications of machine learning in numerous areas involve cont...

Information Processing by Nonlinear Phase Dynamics in Locally Connected Arrays

Research toward powerful information processing systems that circumvent ...

FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Data

Federated Learning (FL) has become a popular paradigm for learning from ...

DFOGraph: An I/O- and Communication-Efficient System for Distributed Fully-out-of-Core Graph Processing

With the magnitude of graph-structured data continually increasing, grap...

A Survey of Coded Distributed Computing

Distributed computing has become a common approach for large-scale compu...

Distributed Approximate Distance Oracles

Data structures that allow efficient distance estimation (distance oracl...

Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updates

It has been experimentally observed that the efficiency of distributed t...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.