Global Autoregressive Models for Data-Efficient Sequence Learning

1 Introduction

Neural sequential text generation models have become the standard in NLP applications such as language modelling, NLG, machine translation. When enough data is available, these models can be trained end-to-end with impressive results. Generally, inference and training proceed in an auto-regressive manner, namely, the next decoded symbol is predicted by a

locally normalized

conditional distribution (the “softmax”). This has several advantages: (i) the probability of the sequence is already normalized, by the chain-rule over local decisions, (ii) max-likelihood (ML) training is easy, because the log-likelihood of the full sequence is simply the sum of local CE (cross-entropy) losses, (iii) exact sampling of full sequences from the model distribution is directly obtained through a sequence of local sampling decisions.

However, these autoregressive models (AMs) tend to suffer from a form of myopia. They have difficulty accounting for global properties of the predicted sequences, from overlooking certain aspects of the semantic input in NLG to duplicating linguistic material or producing “hallucinations” in MT, and generally through being unable to account for long-distance consistency requirements that would be obvious for a human reader.¹¹1

To borrow terminology from Reinforcement Learning (RL)

Sutton and Barto (2018)

, such NLP models work by “imitation learning”, without any representation of “objectives” to be realized. While this defect can be mitigated in the presence of large training sets, it can become serious when this condition is not met.

The main contributions of this paper are as follows.

First, we propose a hybrid seq2seq formalization, the Global Autoregressive Model (GAM), that combines a local autoregressive component with a global log-linear component, allowing the use of a priori

features to compensate for the lack of training data. GAMs are related both to the class of Energy-Based Models (EBM) and to that of Exponential Families (EF), and inherit some important properties from those: an intimate relationship between training and sampling (EBM); the identity of empirical and model expectations at maximum-likelihood; convexity of log-likelihood (EF).

Second, we propose a training procedure in two steps. In the first step, we train through max-likelihood a GAM, which however is unnormalized and improper for fast inference or evaluation. In the second step, we use this GAM to train (by distillation) a second autoregressive model that approximates the normalized distribution associated with the GAM, and can be used for fast inference and evaluation.

Third, we demonstrate the ability of GAMs to be data-efficient, namely, to exploit the original data better than a standard autoregressive model. In order to clarify the core techniques and issues, we design a simple class of synthetic data, consisting of random binary strings containing “motifs” (specific substrings) that we can manipulate in different ways. We show that, in limited data conditions, GAMs are able to exploit the features to obtain final autoregressive models that perform better than the original ones.

The remainder of the paper is structured as follows. In Section 2, we provide some background about autoregressive models, energy-based models, and log-linear models. In Section 3, we introduce GAMs. In section 4, we describe our focus on synthetic data. In Section 5, we explain our training procedure. In Section 6, we comment on related work. In Section 7, we describe our experiments. In Section 8, we provide an analysis of our results. We conclude with a discussion in Section 9. Note that some additional explanations and experiments are provided in the Supplementary Material, indicated by [SM].

2 Background

Autoregressive models (AM)

These are currently the standard for neural seq2seq processing, with such representatives as RNN/LSTMs Hochreiter and Schmidhuber (1997); Sutskever et al. (2014), ConvS2S Gehring et al. (2017), Transformer Vaswani et al. (2017)). Formally, they are defined though a distribution $r_{η} (x | C)$ , where $C$ is an input (aka Context, e.g. a source sentence in Machine Translation (MT)), and $x$ is a target sequence (e.g. a target sentence in MT). We have:

r_{η} (x | C) ≐ \prod i s_{η} (x_{i} | x_{1}, \dots, x_{i - 1}, C),

where each $s_{η} (x_{i} | x_{1}, \dots, x_{i - 1}, C)$

is a normalized conditional probability over the next symbol of the sequence, computed by a neural network (NN) with parameters

η

. The local normalization of the incremental probabilities implies the overall normalization of the distribution

r_{η} (x | C)

, and consequently, the possibility of directly sampling from it and evaluating the likelihood of training sequences.

Energy-Based Models (EBM)

EBMs are a generic class of models, characterized by an energy function $U_{η} (x | C)$ computed by a NN parametrized by $η$ LeCun et al. (2006)

. Equivalently, they can be seen as directly defining a potential (an unnormalized probability distribution)

P_{η} (x | C) = e^{- U_{η} (x | C)}

, and indirectly the normalized distribution

p_{η} (x | C) = 1 / Z_{η} (C) P_{η} (x | C)

, with

Z_{η} (C) = \sum_{x} P_{η} (x | C)

. A fundamental property of these models is that, for max-likelihood training, the SGD updates can be computed through the formula:²²2See (LeCun et al., 2006, p. 15), and [SM] for a derivation.

	$\nabla_{η} log p_{η} (x \| C)$	$= \nabla_{η} log P_{η} (x \| C)$		(1)
		$- E_{x \sim p_{η} (\cdot \| C)} \nabla_{η} log P_{η} (x \| C),$

which, in principle, reduces the problem of training with unnormalized potentials to the problem of sampling from them.

Log-Linear Models / Exponential Families

Log-Linear models Jebara (2013) are the conditional version of Exponential Families Jordan (2010). The general form of a log-linear model (for the discrete case) is as follows:

p_{λ} (x | C) = 1 / Z_{λ} (C) μ (x; C) e^{⟨ λ, ϕ (x; C) ⟩},

with $Z_{λ} (C) = \sum_{x} μ (x; C) e^{⟨ λ (C), ϕ (x; C) ⟩}$ . Here $ϕ (x; C)$

is a vector of predefined real features of the pair

(x, C)

, which is combined by scalar product with a real vector of weights

λ

of the same dimension;

μ (x; C)

is an arbitrary “base measure”, which is fixed. These models, which allow to introduce prior knowledge through features and have nice formal properties (see below), were mainstream in NLP before the revival of neural approaches.

3 Proposal: GAMs

We now define Global Autoregressive Models (GAMs). These are hybrid seq2seq models that exploit both local autoregressive properties as well as global properties of the full target sequence. A GAM is an unnormalized distribution $P_{η} (x | C)$ over sequences $x$ , parametrized by a vector $η = η_{1} \oplus η_{2}$ :

P_{η} (x | C) = r_{η_{1}} (x | C) \cdot e^{⟨ λ_{η_{2}} (C), ϕ (x; C) ⟩} .

(2)

Here $r_{η_{1}} (x | C)$ is an autoregressive seq2seq model for generating $x$ from input $C$ , parametrized by $η_{2}$ ; $ϕ (x; C)$ is a vector of predefined real features of the pair $(x, C)$ , which is combined by a scalar product with a real vector $λ_{η_{2}} (C)$ of the same dimension, computed over the input $C$ by a network parametrized by $η_{2}$ . The normalized distribution associated with the GAM is $p_{η} (x | C) = \frac{P_{η} (x | C)}{Z_{η} (C)}$ , where $Z_{η} (C) = \sum_{x} P_{η} (x | C)$ .

GAMs appear promising for the following reasons:

[leftmargin=*]
Features $ϕ (x; C)$ provide a simple way to draw attention of the model to potentially useful aspects that may be difficult for the AM component to discover on its own from limited data.
GAMs are an instance of EBMs, where the potential $P_{η} (x | C)$ is the product of the an AM potential $r_{η_{1}} (x | C)$ with a “log-linear” potential $e^{⟨ λ_{η_{2}} (C), ϕ (x; C) ⟩}$ . Here the gradient relative to the log-linear part takes the especially simple form:

$\nabla_{η_{2}} log p_{η} (x | C)$ $= ϕ (x; C)$ (3)

$- E_{x \sim p_{η} (\cdot | C)} ϕ (x; C) .$
Log-linear models, on their own, while great at expressing prior knowledge, are not as good as AM models at discovering unforeseen regularities in the data. Also, they are typically problematic to train from a log-likelihood perspective, because sampling from them is often unfeasible. GAMs address the first issue through the $r$ component, and alleviate the second issue by permitting the use of $r$ as a powerful “proposal” (aka “surrogate”) distribution in importance sampling and related approaches, as we will see.

4 Experimental focus

While the motivation for GAMs ultimately lies in practical NLP applications such as those evoked earlier, in this paper we aim to understand some of their capabilities and training techniques in simple and controllable conditions. We focus on the unconditional (i.e. language modelling) case, and on synthetic data. Our setup is as follows:

[leftmargin=*]
We consider an underlying process $p_{t r u e}$ that generates binary sequences according to a well-defined and flexible process. In this paper we use PFSAs (Probabilistic Finite State Automata) to impose the presence or absence of sub-strings (“motifs”) anywhere in the generated data, exploiting the intersection properties of automata.
Due to the dynamic programming properties of PFSAs, it is possible to compute the true entropy $H (p_{t r u e}) = - \sum_{x} p_{t r u e} (x) log p_{t r u e} (x)$ of the process (see [SM]), as well as other quantities (Partition Functions, Mean sequence length); it is also possible to generate training ( $D$ ), validation ( $V$ ), and test data ( $T$ ) in arbitrary quantities.
We employ an unconditional GAM of the simple form:

$p_{λ} (x)$ $≐ \frac{P_{λ} (x)}{Z_{λ}}, with Z_{λ} ≐ \sum x P_{λ} (x) and$

$P_{λ} (x)$ $≐ r (x) \cdot e^{⟨ λ, ϕ (x) ⟩},$ (4)

where $r$ is trained on $D$ and then kept fixed, and where $λ$ is then trained on top of $r$ , also on $D$ .

It should be noted that with $r$ fixed in this way, this formulation exactly corresponds to the definition of an exponential family Jordan (2010), with $r$ as base measure. In such models, we have two important properties: (i) the log-likelihood of the data is convex relative to the parameters $λ$ , and thus a local maximum is also global; (ii) the max-likelihood value $λ^{*}$ has the property that the model expectation $E_{x \sim p_{λ^{*}} (\cdot)} ϕ (x)$ is equal to the empirical expectation $| D |^{- 1} \sum_{x \in D} ϕ (x)$
(“Moment Matching” property of exponential families).
We are specially interested in the relative data-efficiency of the GAM compared to the AM $r$ : namely the ability of the GAM to recover a lower perplexity approximation of $p_{t r u e}$ than $r$ , especially in small training-set conditions.

5 Training procedure

Two-stage training

We consider a two-stage training procedure (see Fig. 1).

Training-1

This consists in training the model $P_{λ}$ on $D$ . This is done by first training $r$ on $D$ in the standard way (by cross-entropy) and then by training $λ$ by SGD with the formula (adapted from (3)):

\nabla_{λ} log p_{λ} (x) = ϕ (x) - E_{x \sim p_{λ} (\cdot)} ϕ (x) .

(5)

The main difficulty then consists in computing an estimate of the model moments

E_{x \sim p_{λ} (\cdot)} ϕ (x)

. In our experiments, we compare two Monte-Carlo approaches Robert and Casella (2005) for addressing this problem: (i) Rejection Sampling (rs), using

r

as the proposal distribution and (ii) Self-Normalized Importance Sampling (snis) (Y. Bengio and J. S. Senecal, 2008), also using

r

as the proposal.

Rejection sampling is performed as follows. We use $r (x)$ as the proposal, and $P_{λ} (x) = r (x) e^{λ \cdot ϕ (x)}$ as the unnormalized target distribution; for any specific $λ$ , because our features are bounded between $0$ and $1$ , we can easily upper-bound the ratio $\frac{P_{λ} (x)}{r (x)} = e^{λ \cdot ϕ (x)}$ by a number $β$ ; we then sample $x$ from $r$ , compute the ratio $ρ (x) = \frac{P_{λ} (x)}{β r (x)} \leq 1$ , and accept $x$ with probability $ρ (x)$ . The accepted samples are unbiased samples from $p_{λ} (x)$ and can be used to estimate model moments.

Snis also uses the proposal distribution $r$ , but does not require an upper-bound, and is directly oriented towards the computation of expectations. In this case, we sample a number of points $x_{1}, \dots, x_{N}$ from $r$ , compute “importance ratios” $w (x_{i}) = \frac{P_{λ} (x_{i})}{r (x_{i})}$ , and estimate $E_{x \sim p_{λ} (\cdot)} ϕ (x)$ through $^E = \frac{\sum_{i} w (x_{i}) ϕ (x_{i})}{\sum_{i} w (x_{i})}$ . The estimate is biased for a given $N$ , but consistent (that is, it converges to the true $E$ for $N \to \infty$ ).

Training-2

While Training-1 results in a well-defined model $P_{λ} (x)$ , which may fit the data closely in principle, we should not conclude that $P_{λ} (x)$ is convenient to use for inference — namely, in language modeling, efficiently sampling from its normalized version $p_{λ} (x)$ ; as seriously, because of the partition factor $Z_{λ}$ , it is also not obvious to evaluate the perplexity of $P_{λ} (x)$ on test data. In order to do both, one approach consists in using a distillation technique Hinton et al. (2015), where, during training, one expends generous time towards producing a set of samples from $P_{λ}$ , for instance by Monte-Carlo (e.g. Rejection Sampling) techniques, and where this set (which may be arbitrarily larger than the original $D$ ) is in turn used to train a new autoregressive model $π_{θ} (x)$ , which can then be used directly for sampling or for computing data likelihood. This is the approach that we use in our current experiments, again using the original $r (x)$ as a proposal distribution.

Cyclical training

In the case of small $| D |$ , the proposal distribution $r$ is weak and as a result the distillation process, based on rejection sampling, can be slow. To address this issue, we also consider a cyclical training regime that updates the proposal distribution after distilling each batch of samples, with the intention of reducing the rejection rate. Once the process of distillation is finished, we use the aggregated samples to train the final $π_{θ}$ . The two-stage training procedure is a variant of the cyclical one, with a fixed proposal (see the Algorithm 1 for more details).

1:function train(

D, V, T, f t

, DsSize, tReg, mode)

▹

DsSize - distilled dataset size; tReg

\in {r s, s n i s_{r}}

r←\textsctrainRNN(D,V,optAdam)

▹

initialize and then train RNN

Pλ←\textsctrainGAM(r,D,V,tReg,ft)

▹

train

λ

for a given proposal

r

4: if mode

=

‘two_stage’ then

▹

Training-2: distill in one step

˜D, ˜V, ––←\textscdistillBatch(Pλ,DsSize)

6: else if mode

=

‘cyclic’ then

▹

Cyclic-training: distill in several steps

˜ D \leftarrow {}; ˜ V \leftarrow {}; {flag}_{λ} \leftarrow False

8: while

| ˜ D | < DsSize

▹

proceed to the distillation process

˜DB, ˜VB, accptRate←\textscdistillBatch(Pλ,bSize)

▹

accptRate - acceptance rate of

r s

during distillation

10:

˜ D

.insert(

{˜ D}_{B}

);

˜ V

.insert(

{˜ V}_{B}

)

11: if not

{flag}_{λ}

then

12:

r←\textscsingleUpdateRNN(r,˜DB,optAdam)

▹

improve proposal

r

13:

Pλ←\textsctrainGAM(r,D,V,tReg,ft)

▹

train

λ

for a given proposal

r

14:

flagλ←\textscearlyStopping$d$(acceptRate)

▹

check if acceptance rate has stopped improving

15:

˜ D

.insert(

D

);

˜ V

.insert(

V

)

▹

add true data to the distilled one

16:

πθ←\textsctrainRNN(˜D,˜V,% optAdam)

17: return

π_{θ}

18:function trainGAM(

P_{λ}

, D, V, tReg,

f t

)

▹

Training-1

19:

α_{0} \leftarrow 10

▹

initial learning rate

20:

target_mom←\textscgetMoments(D,V,ft)

▹

empirical moments of the given dataset

21: while not earlyStopping(

ℓ_{1}_m o m

) do

▹

check if

ℓ_{1}_m o m

has stopped improving

22: model_mom

\leftarrow [0] \times | f t |

▹

accumulate the model’s moments

23:

αt←α01+\#epoch

24: for

b \in range (# updatesPerEpoch)

25:

mean\_mom←\textscgetMomentsGAM(Pλ,D,V,tReg,ft)

▹

use

r s

s n i s_{r}

to compute

E_{x \sim p_{λ} (\cdot)} ϕ (x)

26: model_mom

←(model\_mom+mean\_mom/(b−1))⋅b−1b

▹

moving average

27:

∇λ←target\_mom−mean\_mom

▹

use Eq. 5 to compute gradients

28:

λ \leftarrow λ + α_{t} \cdot \nabla_{λ}

29:

ℓ1_mom←∥target\_mom−model\_mom∥1

30: return

P_{λ}

Algorithm 1 Training

6 Related Work

Hoang et al. (2018), working in a NMT context, have a similar motivation to ours. They first train an autoregressive seq2seq model (Transformer in their case) on bilingual data, then attempt to control global properties of the generated sequences through the introduction of a priori

features. They interpolate the training of the autoregressive model with training of a Moment Matching component which tries to equate the features expectations of the model with those of the data. Contrarily to our approach, they do not directly try to maximize likelihood in an integrated model.

Andor et al. (2016) consider transition-based neural networks, and contrast local to global normalization of decision sequences, showing how the global approach avoids the label bias problem in such tasks as tagging or parsing. They focus on inference as maximization, e.g. finding the best sequence of tags for a sequence of words, and consistent with that objective, their training procedure exploits a beam-search approximation. By contrast, our focus is on inference as sampling in a language modelling perspective, on the complementarity between auto-regressive models and log-linear models, and on the relations between training and sampling in energy-based models.

Figure 2: Cross-entropy in nats per character and frequency of sampling motif, depending on $| D |$ . Two-stage Training. Features $d_{0}, d_{1}, d_{2}, d_{3}$ are on for all panels ( ${f t}_{[4 : 7]} = {1111}$ ). Panel (a): pure $D$ , features $m_{+ 0}$ (super-motif) and $m_{/ 2}$ (sub-motif) on; (b): pure $D$ , $m$ (motif) and $m_{/ 2}$ (sub-motif) on; (c) pure $D$ , $m$ on; (d) mixture $D$ , $m$ on. The plain lines represent cross-entropy, the dashed lines motif frequency.

7 Experiments

We conduct a series of experiments on synthetic data to illustrate our approach.

Synthetic data

To assess the impact of GAMs, we focus on distributions $p_{t r u e} (x)$ that are likely to be well approximated by the AM $r (x)$ in the presence of large data. The first class of distributions is obtained through a PFSA that filters binary strings of fixed length $n = 30$ , $0$ ’s and $1$

’s being equally probable (white-noise strings), through the condition that they contain a specific substring (“motif”) anywhere; here the relative frequency of sequences containing the motif among all sequences varies from

\sim 0.01

\sim 0.001

We also consider mixtures of two PFSAs (motif/anti-motif): the first (with mixture prob. $0.9$ ) produces white-noise strings containing the motif and the second (with mixture prob. $0.1$ ) strings excluding the motif.

From these processes we produce a training set $D$ , of size $| D |$ varying between $5 \cdot 10^{2}$ and $2 \cdot 10^{4}$ , a validation set $V$ of size $0.25 \cdot | D |$ (but never smaller than $5 \cdot 10^{2}$ or bigger than $2 \cdot 10^{3}$ ) and a test set $T$ of fixed size $5 \cdot 10^{3}$ .

Features

In a real world scenario, prior knowledge about the true process will involve, along with predictive features, a number of noisy and useless features. By training the $λ$ parameters to match the empirical moments, the GAM will learn to distinguish between these types. In order to simulate this situation we consider feature vectors over our artificial data that involve both types.

With $x$ the full string and $m$ the fixed motif used in constructing the training data, we consider variations among the 7 binary features in the set $F$ :

F = {m, m_{+ 0}, m_{/ 2}, d_{0}, d_{1}, d_{2}, d_{3}},

where $m = 0$ iff the motif $m$ appears in $x$ , $m_{+ 0} = 0$ iff the motif followed by a zero (“super-motif”) appears in $x$ , $m_{/ 2} = 0$ iff an initial section of the motif (“sub-motif”, roughly half the size of $m$ ) appears in $x$ . These three features are chosen because they have some correlation with the process for generating the training data. By contrast, the four remaining features are “distractors”: $d_{0} = 0$ iff $x$ begins with a $0$ , $d_{1} = 0$ (resp. $d_{2} = 0$ , $d_{3} = 0$ ) iff a certain random, but fixed, string of similar length to $m$ (resp. of larger length, of smaller length) appears in $x$ . We test different configurations of these features for training $λ$ , and document the use/non-use of features with a bit-vector $f t$ of length $| F |$ , for instance $f t = 0111111$ means that all features are exploited, apart from $m$ .

Implementation aspects

Autoregressive models

The AMs are implemented in PyTorch

Paszke et al. (2017) using a 2-layered LSTM Hochreiter and Schmidhuber (1997)

with hidden-state size 200. The input is presented through one-hot encodings over the vocabulary

V = {0, 1, ⟨ EOS ⟩}

. These LSTMs are optimized with Adam Kingma and Ba (2014), with learning rate

α = 0.001

, and with early stopping (patience

= 20

) over a validation set.

Training: Two-Stage and Cyclical

The implementation is described in (Algorithm 1). Here we provide some additional details.

Training-1 For training $P_{λ} (x)$ we test two regimes in Eq. 5, namely $r s$ and $s n i s_{r}$ ; in both cases, we first train $r (x)$ on the whatever $D$ is available, and use it as the proposal distribution. During $r s$ , we compute the model’s expectation over 10 accepted samples, update the $λ$ ’s according to (5), and iterate. During $s n i s_{r}$ , we keep a buffer of the last $5 \cdot 10^{4}$ samples from $r (x)$ to compute the weighted average of the feature moments. For the training of $λ$ ’s, we use a basic SGD optimization with learning rate $α(\#epoch)=α01+\#epoch,α0=10$ . To assess the quality of $P_{λ} (x)$ for early stopping during training, we use the distance between the empirical and model moments:

ℓ_{1}_m o m = ∥ ∥ ∥ \frac{1}{| D |} \sum d \in D ϕ (d) - E_{x \sim p_{λ} (\cdot)} ϕ (x) {∥ ∥ ∥}_{1} .

(6)

Training-2 and Cyclical Training When distilling from $P_{λ}$ in Training-2, we use a single proposal $r$ , and systematically produce a distilled dataset of size $DsSize = 2 \cdot 10^{4}$ , which corresponds to the highest value of $| D |$ among those considered for training $r$ . In Cyclical Training, the distillation process is performed in several stages, with an evolving $r$ for improving the rejection rate.

$\| D \|$	m; $mtf\_frqrsmtf\_frqsnisr$	m; $\frac{CE (r s)}{CE (s n i s_{r})}$	m; $\frac{time (r s)}{time (s n i s_{r})}$	mam; $mtf\_frqrsmtf\_frqsnisr$	mam; $\frac{CE (r s)}{CE (s n i s_{r})}$	mam; $\frac{time (r s)}{time (s n i s_{r})}$
500	0.998	0.967	2.92	0.997	1.003	4.7
1000	1.009	0.973	2.038	0.77	1.07	3.638
5000	0.995	0.967	0.756	1.12	0.99	1.365
10000	1.134	0.956	1.514	1.011	1.002	1.005
20000	1.497	0.961	0.938	0.965	1.005	0.975

Table 1: Comparison of the time for Training-1 in

r s

and

s n i s_{r}

; for motif

10001011111000

;

f t = 1011111

;

H (p_{t r u e}) = 0.449

with pure

D

(m) and

f t = 1001111

;

H (p_{t r u e}) = 0.482

with mixture of motif-anti-motif

D

(mam).

8 Results

Cross-entropy comparison

We conduct experiments to compare the cross-entropy (measured in nats) between the initial AM $r (x)$ relative to the test set $T$ and the final AM $π_{θ} (x)$ also relative to $T$ ; we vary the size of $| D | \in {0.5, 1, 5, 10, 20} \cdot 10^{3}$ , the regimes (tReg) for Training-1 ( $r s$ or $s n i s_{r}$ ), the features employed, the rarity of the motifs. Figure 2 depicts the resulting curves at the end of the two-stage training (plain lines).

Here we show only a few experiments (a more extensive set is provided in the [SM]).

We observe that, for a small dataset size $| D |$ , there is a big gap between the CE of $r (x)$ and the CE of $π_{θ} (x)$ . As $| D |$ increases, these cross-entropies become closer to one another, but a large gap persists for $| D | = 5000$ .

We note that the presence of the “fully-predictive” feature $m$ results in a $π_{θ} (x)$ that has CE very close to the theoretical entropy, even in low $| D |$ regimes, where $r$ on its own is very weak.³³3The CE of a model relative to the true underlying process (approximated by the test set $T$ ) can never be below the entropy of this process, due to the KL-divergence being non-negative. Thus, not only is the distilled AM much better than the initial AM, but this is an indication that $P_{λ}$ itself (for which the cross-entropy is more difficult to compute exactly) is a good approximation of the true process.

By contrast, if the $m$ feature is absent, then, while $π_{θ}$ is still better than $r$ in low $| D |$ regimes, it cannot reach the theoretical entropy in such regimes, because features such as $m_{0 +}$ and $m_{/ 2}$ can only partially model the data. With large $| D |$ , on the other hand, $r$ on itself does a good job at predicting the data, and $P_{λ}$ adds little on top of its $r$ component.

Finally, we note that the two regimes for training $P_{λ} (x)$ , $r s$ and $s n i s_{r}$ , result in $π_{θ}$ ’s with similar accuracies.

We also observe that with a good performance of $π_{θ} (x)$ , the moments of motif feature on the distilled dataset are close to the true ones (see [SM] Figure 4, 5, 7).

These trends are consistent across the experiments with different motifs, as can be checked with the more extensive plots and statistics in the [SM] (Figure 4, 5, 7 and Table 3).

Motif frequencies

In order to assess the predictive properties of obtained AMs, we also compare the frequency of motifs in strings sampled from $r$ and from $π_{θ}$ ( $2 \cdot 10^{3}$ samples in total). From Figure 2 we see that when vary $| D |$ , the frequency of motifs (dashed lines) is aligned with the CE performance. Namely, $π_{θ}$ produces a higher fraction of strings with motif than $r$ when $| D |$ is small ( $| D | \in {0.5, 1, 5} \cdot 10^{3}$ ).
Detailed illustration To provide more intuition, we provide an illustration from one experiment in Table 2.

1	$t r u e$	$101100010111110001000001001001$
2	$r$	$0111110 0001011111000 - --------------- - 1110001011$
3	$π_{θ}$	$111010100010111110000111111100$
4	$f t$	$[m,_,_, d_{0}, d_{1}, d_{2}, d_{3}]$
5	$λ$ ’s	$[- 10.1,_,_, - 0.15, - 0.06, 0.0, - 0.14]$
6	mom $t r u e$	$[0.0,_,_, 0.47, 0.99, 1.0, 0.91]$
7	mom $r$	$[0.95,_,_, 0.53, 0.99, 1.0, 0.91]$
8	mom $π_{θ}$	$[0.0006,_,_, 0.43, 0.99, 0.99, 0.91]$
9	CEs	$t r u e$ : 0.45, $r$ : 0.56, $π_{θ}$ : 0.47
10	motif freqs	$t r u e$ : 1.0, $r$ : 0.045, $π_{θ}$ : 0.959

Table 2: Illustration. Setting is from Fig. 2, panel (c): n =30, motif = 10001011111000 (always present in

D

), ft = 1001111,

| D |

= 5000, rs used for Training-1. Lines 1,2,3 show one example from

t r u e, r, π_{θ}

respectively; with training set of size 5000,

r

is only able to generate the motif a fraction of the time (0.045, see line 10), but is better able to generate some submotifs (underlined);

π_{θ}

generates the motif frequently (0.959), as illustrated on line 3. With the features from

f t

(line 4), Training-1 produces a

P_{λ}

with first feature

λ_{m}

strongly negative (line 5), meaning that

P_{λ}

strongly penalizes the absence of the motif; the “distractor” features

d_{0}, d_{1}, d_{2}, d_{3}

get a weight close to

0

, meaning that they have little predictive power in combination with feature

m

. It is visible from lines 6,7,8 that

π_{θ}

is much better able to approximate the true feature expectations than

r

. Finally (line 9), the CE of

π_{θ}

relative to the test set is close to the true entropy of the process, while that of

r

is much further away.

Mixture $D_{m a m}$ vs pure $D_{m}$

In our experiments, the strings in $D_{m a m}$ (motif-anti-motif) contain a motif with $p = 0.9$ . However, if not all of the samples in $D_{m a m}$ contain the motif, then the motif feature itself is not fully predictive. It can be seen in panel (d) of Figure 2 that the $π_{θ}$ achieved with $P_{λ}$ trained on mixture $D_{m a m}$ has consistent behaviour with the results obtained on the pure $D_{m}$ of panels (a,b,c).

Regimes in Training-1

For training GAM we consider two methods, $s n i s_{r}$ and $r s$ . As described in the previous sections, their impact on $P_{λ}$ leads to $π_{θ}$ ’s that have similar CE’s and motif frequencies. Despite such resemblance in terms of accuracy, these two methods differ in terms of speed (see Table 1). Namely, when $r$ is close to white noise due to small $| D |$ , then for the rare events $r s$ rejects most samples not containing the motif due to the effect of the log linear term and negative value of the component $λ_{m}$ corresponding to the $m$ feature, while $s n i s_{r}$ is able to exploit all samples. Despite being faster than $r s$ , $s n i s_{r}$ remains competitive in terms of CE.

Cyclical vs two-stage training

We conducted a small experiment to compare the performance of cyclical training with two-stage training in terms of speed and accuracy for a fixed motif $m$ and features $f t$ (see [SM] Table 4, Figure 3). We observed that CEs of the obtained $π_{θ}$ ’s were about the same for different values of $| D |$ and Training-1 regimes. On the other hand, there was no systematic improvement in the training speed of one method over the other.

9 Discussion

The basic idea behind GAMs is very simple. First, we extend the representational power of the autoregressive model $r$ by multiplying by a log-linear potential, obtaining an unnormalized model $P_{λ}$ (Training-1). Then we try to “project” this extended representation again to an autoregressive model $π_{θ}$ (Training-2). Our results showed that, under favorable prior knowledge conditions, the final $π_{θ}$ was able to perform as well, when trained on small data, as the standard $r$ , trained on large data. During our experiments, we noticed that training $P_{λ}$ was actually easier than training $π_{θ}$ from it. Intuitively, the small number of parameters to be fitted in the log-linear model requires less work and fewer data than the training of an autoregressive component.⁴⁴4At a deeper level, there are extreme situations where the $P_{λ}$ obtained at the end of Training-1 can perfectly represent the true process, but where no autoregressive model can actually fit $P_{λ}$ : one way to obtain such situations consists in generating binary strings that satisfy a certain cryptographic predicate, associated with a specific feature; the importance of this feature can be easily detected through Training-1, but an autoregressive model has no chance of generalizing from distilled or true data, even in large quantities.

It is interesting to relate our study to certain aspects of Reinforcement Learning (RL).

First, consider Training-2. There, we have a “score” $P_{λ}$ that we are trying to approximate through an autoregressive model $π_{θ}$ , which is basically a sequential “policy”. The main difference with RL is that we are not trying to find a policy that maximizes the score (which would be a bad idea for language modelling, as it would tend to concentrate the mass on a few sequences), but one that approximates $P_{λ}$ in a distributional sense; our current distillation technique is only one way to approach this problem, but other techniques more in the spirit of RL are possible, a direction that we leave for future work.

Second, consider Training-1. Our approach, consisting in suggesting to the model a number of prior features, might look too easy and suspicious. But notice that in RL, one would typically directly provide to the model an externally defined reward, a very strong form of prior knowledge. Here, instead, we “only” indicate to the models which features it might attend to, and Training-1 then determines the “reward” $P_{λ}$ through max-likelihood, a milder form of prior knowledge, more respectful for what the data has to say.

References

Andor et al. (2016) Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016. Globally Normalized Transition-Based Neural Networks.
Bellemare et al. (2017) Marc G. Bellemare, Will Dabney, and Rémi Munos. 2017. A Distributional Perspective on Reinforcement Learning. arXiv:1707.06887 [cs, stat]. ArXiv: 1707.06887.
Carrasco (1997) Rafael C. Carrasco. 1997. Accurate computation of the relative entropy between stochastic regular grammars. Theoretical Informatics and Applications, 31:437–444.
Cortes et al. (2008) Corinna Cortes, Mehryar Mohri, Ashish Rastogi, and Michael Riley. 2008. On the computation of the relative entropy of probabilistic automata. Int. J. Found. Comput. Sci., 19(1):219–242.
Gehring et al. (2017) Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. CoRR. Cite arxiv:1705.03122.
Hinton et al. (2015) Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. CoRR, abs/1503.02531.
Hoang et al. (2018) Cong Duy Vu Hoang, Ioan Calapodescu, and Marc Dymetman. 2018. Moment Matching Training for Neural Machine Translation: A Preliminary Study.
Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780.
Jebara (2013) Tony Jebara. 2013. Log-Linear Models, Logistic Regression and Conditional Random Fields.
Jordan (2010) Michael I. Jordan. 2010. Chapter 8 The exponential family : Basics.
Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
LeCun et al. (2006) Yann LeCun, Sumit Chopra, Raia Hadsell, Marc’Aurelio Ranzato, and Fu Jie Huang. 2006. A Tutorial on Energy-Based Learning. Predicting Structured Data, pages 191–246.
Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop.
Robert and Casella (2005) Christian P. Robert and George Casella. 2005. Monte Carlo Statistical Methods (Springer Texts in Statistics). Springer-Verlag, Berlin, Heidelberg.
Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 3104–3112.
Sutton and Barto (2018) Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction, second edition. The MIT Press.
Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 6000–6010.
Y. Bengio and J. S. Senecal (2008) Y. Bengio and J. S. Senecal. 2008. Adaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language Model. Ieee Transactions on Neural Networks, 19(4):713–722.

Appendix A Supplementary Material

a.1 SGD in Energy-based models

The formula 1 is fundamental for studying the SGD behavior of Energy-based models, and for convenience, we provide a derivation here.

If we define $Z_{η} (C) ≐ \sum_{x} P_{η} (x | C)$ , we find that:

	$\nabla_{η} log Z_{η} (C)$	$= \nabla_{η} log \sum x P_{η} (x \| C)$
		$= \frac{1}{Z_{η} (C)} \nabla_{η} \sum x P_{η} (x \| C)$
		$= \frac{1}{Z_{η} (C)} \sum x \nabla_{η} P_{η} (x \| C)$
		$= \frac{1}{Z_{η} (C)} \sum x P_{η} (x \| C) \nabla_{η} log P_{η} (x \| C)$
		$= \sum x p_{η} (x \| C) \nabla_{η} log P_{η} (x \| C)$
		$= E_{x \sim p_{η} (\cdot \| C)} \nabla_{η} log P_{η} (x \| C) .$

We then have:

	$\nabla_{η} log p_{η} (x \| C)$	$= \nabla_{η} log P_{η} (x \| C) - \nabla_{η} log Z_{η} (C)$
		$= \nabla_{η} log P_{η} (x \| C)$
		$- E_{x \sim p_{η} (\cdot \| C)} \nabla_{η} log P_{η} (x \| C) .$

a.2 The relevance of finite state automata; connections and differences with Reinforcement Learning

The way our synthetic data is produced through FSA’s may look contrived, but there are good motivations for using automata in such a study as ours.

Consider the following problem: you are given some RNN $r$ that produces sequences $x$ over a vocabulary $V$ , with probabilities $r$ but you would like to filter out sequences that do not contain a specific symbol $a$ , while preserving the relative probabilities of sequences provided by the RNN: $p_{f i l t e r e d} (x) \propto P_{f i l t e r e d} (x) = r (x) I [a \in x]$ . There appears to be no obvious way to realize $p_{f i l t e r e d}$ through an RNN, apart from techniques similar to what we have been describing in our discussion of Training-2.

The situation is completely different with FSA’s. If you have a PFSA (Probabilistic FSA) $r_{p f s a}$ generating sequences $x$ , then you can intersect $r_{p f s a}$ with an automaton that accepts all sequences containing at least one $a$ , and re-normalize the intersection through dynamic programming, obtaining a new PFSA that generates the filtered distribution.⁵⁵5And this is exactly what we do to produce training data for our experiments, but using a binary sequence (motif $m$ ) instead of a single symbol $a$ . Such dynamic programming, with the capacity to anticipate properties that need to be satisfied on the global sequence, is unavailable in the RNN world.

With RNNs, the situation is reminiscent of RL, with a reward associated with having observed an $a$ during the production of the sequence. But a standard RL approach would mean that we would try to maximize $P_{f i l t e r e d} (x)$ , without taking into consideration the original $r (x)$ that we are filtering from. To be correct, we need to find a policy $π_{θ} (x)$ (similar to an RNN), that tries to approximate $p_{f i l t e r e d} (x)$ in a distributional sense, not in a maximization sense (see (Bellemare et al., 2017) for related considerations). This is what we try to do in Training-2, using motifs as our main case-study, instead of a single symbol $a$ (which would not make sense for binary strings).

The advantages of using PFSAs in our study are multiple. They provide a well-understood comparison point to the more complex techniques that need to be deployed for autoregressive models. From an operational viewpoint, they also permit, through dynamic programming, to perform various calculations of interest for our study, such as sampling datasets of arbitrary size and computing exact entropy and partition functions that can serve as comparison points for the results obtained with GAMs. In the present paper, we only exploited PFSA’s in the context of motifs, but they provide a much larger class of models that could serve to expand our understanding of sequence-based energy based models.

a.2.1 Computing the Entropy of a PFSA

As mentioned earlier, one advantage of using weigthed finite-state automata for generating synthetic data is that some important quantities, such as entropy, mean sequence length, or partition function can be computed by dynamic programming.

Here we only derive a simple iterative method for computing the entropy of a PFSA, the other computations are very similar.⁶⁶6For another technique, and for extensions to the computation of relative entropy, see Carrasco (1997).

We consider a PFSA with transitions of the form $(q, l, q^{'}, w)$ , where $q, q^{'}$ are states, $l$ is the label of the transition from $q$ to $q^{'}$ (in our case $l \in {0, 1}$ ), and $w$ is the probability of the transition. The fact that the automaton is probabilistic, instead of simply weighted, means that the sum of $w$ ’s associated with transitions starting at $q$ is equal to $1$ . We further assume that the automaton is deterministic, namely that given $q$ and $l$ uniquely determines the next state $q^{'}$ .⁷⁷7The case of non-deterministic probabilistic automata appears much more difficult Cortes et al. (2008).

The entropy $H (q)$ of a state $q$ is defined as $H (q) ≐ - \sum_{x ≻ q} p (x | q) log p (x | q)$ , where $x ≻ q$ denotes a sequence of labels $x$ that ends in a final state of the automaton, for which $p (x | q)$ is computed in the obvious way. The entropy of the automaton as a whole is then defined as $H (q_{s})$ , where $q_{s}$ is the initial state of the automaton.

Lemma The entropies of states satisfy the fixpoint equation:

H (q) = \sum (q, l, q^{'}, w) - w log w + w H (q^{'}) .

(7)

Proof. Let’s denote by $q_{l}$ the state obtained from $q$ by following label $l$ . We have:

	$H (q)$	$= - \sum x ≻ q p (x \| q) log p (x \| q)$
		$= - \sum l \cdot y ≻ q p (l \cdot y \| q) log p (l \cdot y \| q)$
		$= - \sum l \cdot y ≻ q p (l \| q) p (y \| q_{l}) [log p (l \| q) + log p (y \| q_{l})]$
		$= - \sum l p (l \| q) log p (l \| q) \sum y ≻ q_{l} p (y \| q_{l})$
		$- \sum l p (l \| q) \sum y ≻ q_{l} p (y \| q_{l}) log p (y \| q_{l})$
		$= - \sum l p (l \| q) log p (l \| q) + \sum l p (l \| q) H (q_{l})$
		$= \sum (q, l, q^{'}, w) - w log w + w H (q^{'}) .$

It is possible to show that the state entropies actually correspond to the least fixpoint of equation (7), and this allows a simple iterative algorithm for computing the state entropies: at time $t = 0$ , for all states $q$ , we define $H^{t = 0} (q) ≐ 0$ , and then we iterate until convergence:

H^{t + 1} (q) = \sum (q, l, q^{'}, w) - w log w + w H^{t} (q^{'}) .

a.3 Additional Experiments and Results

(See next pages)

tReg	$\| D \|$	m: $\frac{CE (T, r)}{CE (T, π_{θ})}$	m: $\frac{CE (T, π_{θ})}{H (p_{t r u e})}$	m: $mtf\_frq(πθ)mtf\_frq(r)$	mam: $\frac{CE (T, r)}{CE (T, π_{θ})}$	mam: $\frac{CE (T, π_{θ})}{H (p_{t r u e})}$	mam: $mtf\_frq(πθ)mtf\_frq(r)$
$r s$	500	$1.24 \pm 0.07$	$1.19 \pm 0.07$	$[32.0, 392.0]$	$1.23 \pm 0.03$	$1.16 \pm 0.03$	$[59.26, 433.33]$
$r s$	1000	$1.24 \pm 0.07$	$1.16 \pm 0.07$	$[23.87, 653.33]$	$1.21 \pm 0.03$	$1.14 \pm 0.03$	$[26.29, 233.33]$
$r s$	5000	$1.18 \pm 0.08$	$1.09 \pm 0.05$	$[3.59, 206.67]$	$1.16 \pm 0.05$	$1.08 \pm 0.04$	$[7.32, 130.0]$
$r s$	10000	$1.08 \pm 0.1$	$1.04 \pm 0.02$	$[0.89, 196.0]$	$1.02 \pm 0.03$	$1.04 \pm 0.03$	$[1.0, 4.97]$
$r s$	20000	$0.99 \pm 0.01$	$1.02 \pm 0.01$	$[0.81, 1.76]$	$0.99 \pm 0.0$	$1.02 \pm 0.0$	$[0.85, 1.04]$

Table 3: Overall statistics: for

D_{m}, m o t i f \in {10001010001, 01011101101, 001001100111, 1011100111001

10001011111000}, f t \in {1001111, 1011111, 0111111}

and

D_{m a m}

m o t i f \in {01011101101, 001001100111, 1011100111001,

100010100011, 10001011111000}

f t \in {1001111}

$\| D \|$	$\frac{{CE}_{r s}}{{CE}_{r s} (cycl)}$	$\frac{{time}_{r s}}{{time}_{r s} (cycl)}$	$\frac{{CE}_{s n i s_{r}}}{{CE}_{s n i s_{r}} (cycl)}$	$\frac{{time}_{s n i s_{r}}}{{time}_{s n i s_{r}} (cycl)}$
500	1.02	1.21	1.02	1.51
1000	1.0	1.48	1.08	2.04
5000	1.04	0.57	1.0	0.57
10000	0.98	1.47	1.02	0.45
20000	0.99	2.65	1.0	0.28

Table 4: Cyclical training vs two stage training for motif

10001011111000, D_{m}, f t = 1001111

;

CE is short for CE (T, π_{θ}) .

	$\nabla_{η} log p_{η} (x \| C)$	$= \nabla_{η} log P_{η} (x \| C)$		(1)
		$- E_{x \sim p_{η} (\cdot \| C)} \nabla_{η} log P_{η} (x \| C),$

	$\nabla_{η} log Z_{η} (C)$	$= \nabla_{η} log \sum x P_{η} (x \| C)$
		$= \frac{1}{Z_{η} (C)} \nabla_{η} \sum x P_{η} (x \| C)$
		$= \frac{1}{Z_{η} (C)} \sum x \nabla_{η} P_{η} (x \| C)$
		$= \frac{1}{Z_{η} (C)} \sum x P_{η} (x \| C) \nabla_{η} log P_{η} (x \| C)$
		$= \sum x p_{η} (x \| C) \nabla_{η} log P_{η} (x \| C)$
		$= E_{x \sim p_{η} (\cdot \| C)} \nabla_{η} log P_{η} (x \| C) .$

	$\nabla_{η} log p_{η} (x \| C)$	$= \nabla_{η} log P_{η} (x \| C) - \nabla_{η} log Z_{η} (C)$
		$= \nabla_{η} log P_{η} (x \| C)$
		$- E_{x \sim p_{η} (\cdot \| C)} \nabla_{η} log P_{η} (x \| C) .$

	$H (q)$	$= - \sum x ≻ q p (x \| q) log p (x \| q)$
		$= - \sum l \cdot y ≻ q p (l \cdot y \| q) log p (l \cdot y \| q)$
		$= - \sum l \cdot y ≻ q p (l \| q) p (y \| q_{l}) [log p (l \| q) + log p (y \| q_{l})]$
		$= - \sum l p (l \| q) log p (l \| q) \sum y ≻ q_{l} p (y \| q_{l})$
		$- \sum l p (l \| q) \sum y ≻ q_{l} p (y \| q_{l}) log p (y \| q_{l})$
		$= - \sum l p (l \| q) log p (l \| q) + \sum l p (l \| q) H (q_{l})$
		$= \sum (q, l, q^{'}, w) - w log w + w H (q^{'}) .$

	$\nabla_{η_{2}} log p_{η} (x \| C)$	$= ϕ (x; C)$		(3)
		$- E_{x \sim p_{η} (\cdot \| C)} ϕ (x; C) .$

	$p_{λ} (x)$	$≐ \frac{P_{λ} (x)}{Z_{λ}}, with Z_{λ} ≐ \sum x P_{λ} (x) and$
	$P_{λ} (x)$	$≐ r (x) \cdot e^{⟨ λ, ϕ (x) ⟩},$		(4)

Global Autoregressive Models for Data-Efficient Sequence Learning

Authors

ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translation

Pairwise likelihood estimation of latent autoregressive count models

Surprisal-Triggered Conditional Computation with Neural Networks

Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference using a Delta Posterior

Mixtures of Sparse Autoregressive Networks

Autoregressive Modeling is Misspecified for Some Sequence Distributions

Autoregressive Text Generation Beyond Feedback Loops

This week in AI

1 Introduction

2 Background

Autoregressive models (AM)

Energy-Based Models (EBM)

Log-Linear Models / Exponential Families

3 Proposal: GAMs

4 Experimental focus

5 Training procedure

Two-stage training

Training-1

Training-2

Cyclical training

6 Related Work

7 Experiments

Synthetic data

Features

Implementation aspects

Autoregressive models

Training: Two-Stage and Cyclical

8 Results

Cross-entropy comparison

Motif frequencies

Mixture Dmam vs pure Dm

Regimes in Training-1

Cyclical vs two-stage training

9 Discussion

References

Appendix A Supplementary Material

a.1 SGD in Energy-based models

a.2 The relevance of finite state automata; connections and differences with Reinforcement Learning

a.2.1 Computing the Entropy of a PFSA

a.3 Additional Experiments and Results

Mixture $D_{m a m}$ vs pure $D_{m}$