Optimization for Supervised Machine Learning: Randomized Algorithms for Data and Parameters

08/26/2020
by   Filip Hanzely, et al.
0

Many key problems in machine learning and data science are routinely modeled as optimization problems and solved via optimization algorithms. With the increase of the volume of data and the size and complexity of the statistical models used to formulate these often ill-conditioned optimization tasks, there is a need for new efficient algorithms able to cope with these challenges. In this thesis, we deal with each of these sources of difficulty in a different way. To efficiently address the big data issue, we develop new methods which in each iteration examine a small random subset of the training data only. To handle the big model issue, we develop methods which in each iteration update a random subset of the model parameters only. Finally, to deal with ill-conditioned problems, we devise methods that incorporate either higher-order information or Nesterov's acceleration/momentum. In all cases, randomness is viewed as a powerful algorithmic tool that we tune, both in theory and in experiments, to achieve the best results. Our algorithms have their primary application in training supervised machine learning models via regularized empirical risk minimization, which is the dominant paradigm for training such models. However, due to their generality, our methods can be applied in many other fields, including but not limited to data science, engineering, scientific computing, and statistics.

READ FULL TEXT
research
09/26/2019

Randomized Iterative Methods for Linear Systems: Momentum, Inexactness and Gossip

In the era of big data, one of the key challenges is the development of ...
research
06/22/2021

Faster Randomized Methods for Orthogonality Constrained Problems

Recent literature has advocated the use of randomized methods for accele...
research
09/02/2021

Data science and Machine learning in the Clouds: A Perspective for the Future

As we are fast approaching the beginning of a paradigm shift in the fiel...
research
06/22/2022

On a class of geodesically convex optimization problems solved via Euclidean MM methods

We study geodesically convex (g-convex) problems that can be written as ...
research
08/24/2020

Data-Driven Aerospace Engineering: Reframing the Industry with Machine Learning

Data science, and machine learning in particular, is rapidly transformin...
research
09/09/2015

Statistical Inference, Learning and Models in Big Data

The need for new methods to deal with big data is a common theme in most...
research
01/26/2023

Open Problems in Applied Deep Learning

This work formulates the machine learning mechanism as a bi-level optimi...

Please sign up or login with your details

Forgot password? Click here to reset