DeepAI AI Chat
Log In Sign Up

Notes on Latent Structure Models and SPIGOT

07/24/2019
by   André F. T. Martins, et al.
Unbabel Inc.
0

These notes aim to shed light on the recently proposed structured projected intermediate gradient optimization technique (SPIGOT, Peng et al., 2018). SPIGOT is a variant of the straight-through estimator (Bengio et al., 2013) which bypasses gradients of the argmax function by back-propagating a surrogate "gradient." We provide a new interpretation to the proposed gradient and put this technique into perspective, linking it to other methods for training neural networks with discrete latent variables. As a by-product, we suggest alternate variants of SPIGOT which will be further explored in future work.

READ FULL TEXT

page 1

page 2

page 3

page 4

06/15/2021

Coupled Gradient Estimators for Discrete Latent Variables

Training models with discrete latent variables is challenging due to the...
10/05/2020

Understanding the Mechanics of SPIGOT: Surrogate Gradients for Latent Structure Learning

Latent structure models are a powerful tool for modeling language data: ...
10/24/2018

Notes on asymptotics of sample eigenstructure for spiked covariance models with non-Gaussian data

These expository notes serve as a reference for an accompanying post Mor...
05/12/2018

Backpropagating through Structured Argmax using a SPIGOT

We introduce the structured projection of intermediate gradients optimiz...
10/05/2019

Straight-Through Estimator as Projected Wasserstein Gradient Flow

The Straight-Through (ST) estimator is a widely used technique for back-...
08/02/2018

Likelihood-free inference with an improved cross-entropy estimator

We extend recent work (Brehmer, et. al., 2018) that use neural networks ...
11/27/2020

Eigenvalue-corrected Natural Gradient Based on a New Approximation

Using second-order optimization methods for training deep neural network...