Optimization for Large-Scale Machine Learning with Distributed Features and Observations

10/31/2016
by   Alexandros Nathan, et al.
0

As the size of modern data sets exceeds the disk and memory capacities of a single computer, machine learning practitioners have resorted to parallel and distributed computing. Given that optimization is one of the pillars of machine learning and predictive modeling, distributed optimization methods have recently garnered ample attention in the literature. Although previous research has mostly focused on settings where either the observations, or features of the problem at hand are stored in distributed fashion, the situation where both are partitioned across the nodes of a computer cluster (doubly distributed) has barely been studied. In this work we propose two doubly distributed optimization algorithms. The first one falls under the umbrella of distributed dual coordinate ascent methods, while the second one belongs to the class of stochastic gradient/coordinate descent hybrid methods. We conduct numerical experiments in Spark using real-world and simulated data sets and study the scaling properties of our methods. Our empirical evaluation of the proposed algorithms demonstrates the out-performance of a block distributed ADMM method, which, to the best of our knowledge is the only other existing doubly distributed optimization algorithm.

READ FULL TEXT

page 7

page 13

research
03/29/2018

A Stochastic Large-scale Machine Learning Algorithm for Distributed Features and Observations

As the size of modern data sets exceeds the disk and memory capacities o...
research
03/24/2023

Predictive modeling for limited distributed targets

Many forecasting applications have a limited distributed target variable...
research
07/19/2019

ASYNC: Asynchronous Machine Learning on Distributed Systems

ASYNC is a framework that supports the implementation of asynchronous ma...
research
01/26/2023

Distributed Optimization Methods for Multi-Robot Systems: Part II – A Survey

Although the field of distributed optimization is well-developed, releva...
research
08/28/2023

Distributed Dual Coordinate Ascent with Imbalanced Data on a General Tree Network

In this paper, we investigate the impact of imbalanced data on the conve...
research
10/25/2019

The Scalability for Parallel Machine Learning Training Algorithm: Dataset Matters

To gain a better performance, many researchers put more computing resour...
research
10/13/2017

DSCOVR: Randomized Primal-Dual Block Coordinate Algorithms for Asynchronous Distributed Optimization

Machine learning with big data often involves large optimization models....

Please sign up or login with your details

Forgot password? Click here to reset