DeepAI AI Chat
Log In Sign Up

Kernel Deformed Exponential Families for Sparse Continuous Attention

by   Alexander Moreno, et al.
University of Michigan
Georgia Institute of Technology

Attention mechanisms take an expectation of a data representation with respect to probability weights. This creates summary statistics that focus on important features. Recently, (Martins et al. 2020, 2021) proposed continuous attention mechanisms, focusing on unimodal attention densities from the exponential and deformed exponential families: the latter has sparse support. (Farinhas et al. 2021) extended this to use Gaussian mixture attention densities, which are a flexible class with dense support. In this paper, we extend this to two general flexible classes: kernel exponential families and our new sparse counterpart kernel deformed exponential families. Theoretically, we show new existence results for both kernel exponential and deformed exponential families, and that the deformed case has similar approximation capabilities to kernel exponential families. Experiments show that kernel deformed exponential families can attend to multiple compact regions of the data domain.


page 1

page 2

page 3

page 4


Sparse and Continuous Attention Mechanisms

Exponential families are widely used in machine learning; they include m...

Extended Plugin Densities for Curved Exponential Families

Extended plugin densities are proposed as predictive densities for curve...

Sparse Continuous Distributions and Fenchel-Young Losses

Exponential families are widely used in machine learning; they include m...

Approximations for STERGMs Based on Cross-Sectional Data

Temporal exponential-family random graph models (TERGMs) are a flexible ...

Estimating Densities with Non-Parametric Exponential Families

We propose a novel approach for density estimation with exponential fami...

E-values for k-Sample Tests With Exponential Families

We develop and compare e-variables for testing whether k samples of data...

Learning Exponential Families in High-Dimensions: Strong Convexity and Sparsity

The versatility of exponential families, along with their attendant conv...