Exploiting the Structure: Stochastic Gradient Methods Using Raw Clusters

02/05/2016
by   Zeyuan Allen-Zhu, et al.
0

The amount of data available in the world is growing faster than our ability to deal with it. However, if we take advantage of the internal structure, data may become much smaller for machine learning purposes. In this paper we focus on one of the fundamental machine learning tasks, empirical risk minimization (ERM), and provide faster algorithms with the help from the clustering structure of the data. We introduce a simple notion of raw clustering that can be efficiently computed from the data, and propose two algorithms based on clustering information. Our accelerated algorithm ClusterACDM is built on a novel Haar transformation applied to the dual space of the ERM problem, and our variance-reduction based algorithm ClusterSVRG introduces a new gradient estimator using clustering. Our algorithms outperform their classical counterparts ACDM and SVRG respectively.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/08/2016

Stochastic dual averaging methods using variance reduction techniques for regularized empirical risk minimization problems

We consider a composite convex minimization problem associated with regu...
research
10/22/2019

The Practicality of Stochastic Optimization in Imaging Inverse Problems

In this work we investigate the practicality of stochastic gradient desc...
research
02/13/2020

Gradient tracking and variance reduction for decentralized optimization and machine learning

Decentralized methods to solve finite-sum minimization problems are impo...
research
02/22/2023

Stochastic Approximation Beyond Gradient for Signal Processing and Machine Learning

Stochastic approximation (SA) is a classical algorithm that has had sinc...
research
10/05/2018

Network Distance Based on Laplacian Flows on Graphs

Distance plays a fundamental role in measuring similarity between object...
research
09/07/2021

COCO Denoiser: Using Co-Coercivity for Variance Reduction in Stochastic Convex Optimization

First-order methods for stochastic optimization have undeniable relevanc...
research
11/28/2021

Approximate Inference via Clustering

In recent years, large-scale Bayesian learning draws a great deal of att...

Please sign up or login with your details

Forgot password? Click here to reset