DeepAI AI Chat
Log In Sign Up

RecurJac: An Efficient Recursive Algorithm for Bounding Jacobian Matrix of Neural Networks and Its Applications

10/28/2018
by   Huan Zhang, et al.
Microsoft
0

The Jacobian matrix (or the gradient for single-output networks) is directly related to many important properties of neural networks, such as the function landscape, stationary points, (local) Lipschitz constants and robustness to adversarial attacks. In this paper, we propose a recursive algorithm, RecurJac, to compute both upper and lower bounds for each element in the Jacobian matrix of a neural network with respect to network's input, and the network can contain a wide range of activation functions. As a byproduct, we can efficiently obtain a (local) Lipschitz constant, which plays a crucial role in neural network robustness verification, as well as the training stability of GANs. Experiments show that (local) Lipschitz constants produced by our method is of better quality than previous approaches, thus providing better robustness verification results. Our algorithm has polynomial time complexity, and its computation time is reasonable even for relatively large networks. Additionally, we use our bounds of Jacobian matrix to characterize the landscape of the neural network, for example, to determine whether there exist stationary points in a local neighborhood. Source code available at https://github.com/huanzhang12/RecurJac-Jacobian-Bounds.

READ FULL TEXT

page 1

page 2

page 3

page 4

05/12/2021

LipBaB: Computing exact Lipschitz constant of ReLU networks

The Lipschitz constant of neural networks plays an important role in sev...
07/01/2021

Boosting Certified ℓ_∞ Robustness with EMA Method and Ensemble Model

The neural network with 1-Lipschitz property based on ℓ_∞-dist neuron ha...
10/13/2022

Efficiently Computing Local Lipschitz Constants of Neural Networks via Bound Propagation

Lipschitz constants are connected to many properties of neural networks,...
02/28/2020

Automatic Perturbation Analysis on General Computational Graphs

Linear relaxation based perturbation analysis for neural networks, which...
08/08/2022

Robust Training and Verification of Implicit Neural Networks: A Non-Euclidean Contractive Approach

This paper proposes a theoretical and computational framework for traini...
03/06/2023

A Unified Algebraic Perspective on Lipschitz Neural Networks

Important research efforts have focused on the design and training of ne...
02/10/2020

Polynomial Optimization for Bounding Lipschitz Constants of Deep Networks

The Lipschitz constant of a network plays an important role in many appl...