    Linear Asymptotic Convergence of Anderson Acceleration: Fixed-Point Analysis

We study the asymptotic convergence of AA(m), i.e., Anderson acceleration with window size m for accelerating fixed-point methods x_k+1=q(x_k), x_k ∈ R^n. Convergence acceleration by AA(m) has been widely observed but is not well understood. We consider the case where the fixed-point iteration function q(x) is differentiable and the convergence of the fixed-point method itself is root-linear. We identify numerically several conspicuous properties of AA(m) convergence: First, AA(m) sequences {x_k} converge root-linearly but the root-linear convergence factor depends strongly on the initial condition. Second, the AA(m) acceleration coefficients β^(k) do not converge but oscillate as {x_k} converges to x^*. To shed light on these observations, we write the AA(m) iteration as an augmented fixed-point iteration z_k+1 =Ψ(z_k), z_k ∈ R^n(m+1) and analyze the continuity and differentiability properties of Ψ(z) and β(z). We find that the vector of acceleration coefficients β(z) is not continuous at the fixed point z^*. However, we show that, despite the discontinuity of β(z), the iteration function Ψ(z) is Lipschitz continuous and directionally differentiable at z^* for AA(1), and we generalize this to AA(m) with m>1 for most cases. Furthermore, we find that Ψ(z) is not differentiable at z^*. We then discuss how these theoretical findings relate to the observed convergence behaviour of AA(m). The discontinuity of β(z) at z^* allows β^(k) to oscillate as {x_k} converges to x^*, and the non-differentiability of Ψ(z) allows AA(m) sequences to converge with root-linear convergence factors that strongly depend on the initial condition. Additional numerical results illustrate our findings.

Authors

07/04/2020

On the Asymptotic Linear Convergence Speed of Anderson Acceleration, Nesterov Acceleration, and Nonlinear GMRES

We consider nonlinear convergence acceleration methods for fixed-point i...
09/29/2021

Anderson Acceleration as a Krylov Method with Application to Asymptotic Convergence Analysis

Anderson acceleration is widely used for accelerating the convergence of...
05/19/2021

Enabling fast convergence of the iterated penalty Picard iteration with O(1) penalty parameter for incompressible Navier-Stokes via Anderson acceleration

This paper considers an enhancement of the classical iterated penalty Pi...
02/28/2020

Convergence analysis of adaptive DIIS algorithms with application to electronic ground state calculations

This paper deals with a general class of algorithms for the solution of ...
09/18/2019

Fixed Point Analysis of Douglas-Rachford Splitting for Ptychography and Phase Retrieval

Douglas-Rachford Splitting (DRS) methods based on the proximal point alg...
07/06/2020

Quantifying the asymptotic linear convergence speed of Anderson Acceleration applied to ADMM

We explain how Anderson Acceleration (AA) speeds up the Alternating Dire...
02/10/2020

Anderson Acceleration Using the H^-s Norm

Anderson acceleration (AA) is a technique for accelerating the convergen...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.