Swarm-Based Gradient Descent Method for Non-Convex Optimization

11/30/2022
by   Jingcheng Lu, et al.
0

We introduce a new Swarm-Based Gradient Descent (SBGD) method for non-convex optimization. The swarm consists of agents, each is identified with a position, x, and mass, m. The key to their dynamics is communication: masses are being transferred from agents at high ground to low(-est) ground. At the same time, agents change positions with step size, h=h(x,m), adjusted to their relative mass: heavier agents proceed with small time-steps in the direction of local gradient, while lighter agents take larger time-steps based on a backtracking protocol. Accordingly, the crowd of agents is dynamically divided between `heavier' leaders, expected to approach local minima, and `lighter' explorers. With their large-step protocol, explorers are expected to encounter improved position for the swarm; if they do, then they assume the role of `heavy' swarm leaders and so on. Convergence analysis and numerical simulations in one-, two-, and 20-dimensional benchmarks demonstrate the effectiveness of SBGD as a global optimizer.

READ FULL TEXT

page 22

page 25

research
07/23/2023

Swarm-Based Optimization with Random Descent

We extend our study of the swarm-based gradient descent method for non-c...
research
11/15/2016

The Power of Normalization: Faster Evasion of Saddle Points

A commonly used heuristic in non-convex optimization is Normalized Gradi...
research
10/11/2022

Multi-Agent Distributed and Decentralized Geometric Task Allocation

We consider the general problem of geometric task allocation, wherein a ...
research
08/19/2019

Second-Order Guarantees of Stochastic Gradient Descent in Non-Convex Optimization

Recent years have seen increased interest in performance guarantees of g...
research
11/12/2019

A Distributed Online Convex Optimization Algorithm with Improved Dynamic Regret

In this paper, we consider the problem of distributed online convex opti...
research
02/02/2022

Dynamic Time Slot Allocation Algorithm for Quadcopter Swarms

A swarm of quadcopters can perform cooperative tasks, such as monitoring...
research
03/19/2019

How to Make Swarms Open-Ended? Evolving Collective Intelligence Through a Constricted Exploration of Adjacent Possibles

We propose an approach of open-ended evolution via the simulation of swa...

Please sign up or login with your details

Forgot password? Click here to reset