On the Comparison between Cyclic Sampling and Random Reshuffling

04/25/2021
by   Xinmeng Huang, et al.
14

When applying a stochastic/incremental algorithm, one must choose the order to draw samples. Among the most popular approaches are cyclic sampling and random reshuffling, which are empirically faster and more cache-friendly than uniform-iid-sampling. Cyclic sampling draws the samples in a fixed, cyclic order, which is less robust than reshuffling the samples periodically. Indeed, existing works have established worst case convergence rates for cyclic sampling, which are generally worse than that of random reshuffling. In this paper, however, we found a certain cyclic order can be much faster than reshuffling and one can discover it at a low cost! Studying and comparing different sampling orders typically require new analytic techniques. In this paper, we introduce a norm, which is defined based on the sampling order, to measure the distance to solution. Applying this technique on proximal Finito/MISO algorithm allows us to identify the optimal fixed ordering, which can beat random reshuffling by a factor up to log(n)/n in terms of the best-known upper bounds. We also propose a strategy to discover the optimal fixed ordering numerically. The established rates are state-of-the-art compared to previous works.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/23/2018

The largest number of weights in cyclic codes

Upper and lower bounds on the largest number of weights in a cyclic code...
research
10/10/2019

Understanding Limitation of Two Symmetrized Orders by Worst-case Complexity

It was recently found that the standard version of multi-block cyclic AD...
research
11/02/2021

Equivalent Versions of Total Flow Analysis

Total Flow Analysis (TFA) is a method for conducting the worst-case anal...
research
06/20/2018

Hulls of Cyclic Codes over Z_4

The hulls of linear and cyclic codes over finite fields have been of int...
research
02/23/2022

On Z_p^rZ_p^rZ_p^s-Additive Cyclic Codes

In this paper, we introduce ℤ_p^rℤ_p^rℤ_p^s-additive cyclic codes for r≤...
research
01/21/2023

Matchings under distance constraints II

This paper introduces the d-distance b-matching problem, in which we are...
research
09/03/2020

Equal partners do better in defensive alliances

Cyclic dominance offers not just a way to maintain biodiversity, but als...

Please sign up or login with your details

Forgot password? Click here to reset