1 Introduction
Many machine learning tasks involve joint prediction of a set of variables. For instance, semantic image segmentation infers the class label for every pixel in an image. To address joint prediction, it is common to use deep nets which model probability distributions independently over the variables (
e.g., the pixels). The downside: correlations between different variables aren’t modeled explicitly.A number of techniques, such as Structured SVMs (Tsochantaridis et al., 2005), MaxMargin Markov Nets (Taskar et al., 2003) and Deep Structured Models (Chen et al., 2015b; Schwing and Urtasun, 2015), directly model relations between output variables. However, modeling the correlations between a large number of variables is computationally expensive and therefore generally impractical. As an attempt to address some of the fallbacks of classical highorder structured prediction techniques, Structured Prediction Energy Networks (SPENs) were introduced (Belanger and McCallum, 2016; Belanger et al., 2017). SPENs assign a score to an entire prediction, which allows them to harness global structure. Additionally, because these models do not represent structure explicitly, complex relations between variables can be learned while maintaining tractability of inference. However, SPENs have their own set of downsides: Belanger and McCallum (2016) mention, and we can confirm, that it is easy to overfit SPENs to the training data. Additionally, the inference techniques developed for SPENs do not enforce structural constraints among output variables, hence they cannot support structured scores and discrete losses. An attempt to combine locally structured scores with joint prediction was introduced very recently by Graber et al. (2018). However, Graber et al. (2018) require the score function to take a specific, restricted form, and inference is formulated as a difficulttosolve saddlepoint optimization problem.
To address these concerns, we develop a new model which we refer to as ‘Graph Structured Prediction Energy Network’ (GSPEN). Specifically, GSPENs combine the capabilities of classical structured prediction models and SPENs and have the ability to explicitly model local structure when known or assumed, while providing the ability to learn an unknown or more global structure implicitly. Additionally, the proposed GSPEN formulation generalizes the approach by Graber et al. (2018). Concretely, inference in GSPENs is a maximization of a generally nonconcave function w.r.t. structural constraints, for which we develop two inference algorithms.
We show the utility of GSPENs by comparing to related techniques on several tasks: optical character recognition, image tagging, multilabel classification, and named entity recognition. In general, we show that GSPENs are able to outperform other models. Our implementation is available at
https://github.com/cgraber/GSPEN.2 Background
Let represent the input provided to a model, such as a sentence or an image. In this work, we consider tasks where the outputs take the form , i.e
., they are vectors where the
th variable’s domain is the discrete and finite set . In general, the number of variables which are part of the configuration can depend on the observation . However, for readability only, we assume all contain entries, i.e., we drop the dependence of the output space on input .All models we consider consist of a function , which assigns a score to a given configuration conditioned on input and is parameterized by weights . Provided an input , the inference problem requires finding the configuration that maximizes this score, i.e., .
To find the parameters of the function
, it is common to use a Structured Support Vector Machine (SSVM) (a.k.a. MaxMargin Markov Network) objective
(Tsochantaridis et al., 2005; Taskar et al., 2003): given a multiset of data points comprised of an input and the corresponding groundtruth configuration , a SSVM attempts to find weights which maximize the margin between the scores assigned to the groundtruth configuration and the inference prediction:(1) 
Hereby, is a taskspecific and often discrete loss, such as the Hamming loss, which steers the model towards learning a margin between correct and incorrect outputs. Due to addition of the taskspecific loss to the model score , we often refer to the maximization task within Eq. (1) as lossaugmented inference. The procedure to solve lossaugmented inference depends on the considered model, which we discuss next.
Unstructured Models. Unstructured models, such as feedforward deep nets, assign a score to each label of variable which is part of the configuration , irrespective of the label choice of other variables. Hence, the final score function is the sum of individual scores , one for each variable:
(2) 
Because the scores for each output variable do not depend on the scores assigned to other output variables, the inference assignment is determined efficiently by independently finding the maximum score for each variable . The same is true for lossaugmented inference, assuming that the loss decomposes into a sum of independent terms as well.
Classical Structured Models. Classical structured models incorporate dependencies between variables by considering functions that take more than one output space variable as input, i.e., each function depends on a subset of the output variables. We refer to the subset of variables via and use to denote the corresponding function. The overall score for a configuration is a sum of these functions, i.e.,
(3) 
Hereby, is a set containing all of the variable subsets which are required to compute . The variable subset relations between functions , i.e., the structure, is often visualized using factor graphs or, generally, Hasse diagrams.
This formulation allows to explicitly model relations between variables, but it comes at the price of more complex inference which is NPhard (Shimony, 1994) in general. A number of approximations to this problem have been developed and utilized successfully (see Sec. 5 for more details), but the complexity of these methods scales with the size of the largest region . For this reason, these models commonly consider only unary and pairwise regions, i.e., regions with one or two variables.
Inference, i.e
., maximization of the score, is equivalent to the integer linear program
(4) 
where each represents a marginal probability vector for region and represents the set of whose marginal distributions are globally consistent, which is often called the marginal polytope. Adding an entropy term over the probabilities to the inference objective transforms the problem from maximum aposteriori (MAP) to marginal inference, and pushes the predictions to be more uniform (Wainwright and Jordan, 2008; Niculae et al., 2018)
. When combined with the learning procedure specified above, this entropy provides learning with the additional interpretation of maximum likelihood estimation
(Wainwright and Jordan, 2008). The training objective then also fits into the framework of FenchelYoung Losses (Blondel et al., 2018).For computational reasons, it is common to relax the marginal polytope to the local polytope , which is the set of all probability vectors that marginalize consistently for the factors present in the graph (Wainwright and Jordan, 2008). Since the resulting marginals are no longer globally consistent, i.e
., they are no longer guaranteed to arise from a single joint distribution, we write the predictions for each region using
instead of and refer to them using the term “beliefs.” Additionally, the entropy term is approximated using fractional entropies Heskes et al. (2003) such that it only depends on the factors in the graph, in which case it takes the form .Structured Prediction Energy Networks. Structured Prediction Energy Networks (SPENs) (Belanger and McCallum, 2016) were motivated by the desire to represent interactions between larger sets of output variables without incurring a high computational cost. The SPEN score function takes the following form:
(5) 
where is a learned feature representation of the input , each is a onehot vector, and is a function that takes these two terms and assigns a score. This representation of the labels, i.e., , is used to facilitate gradientbased optimization during inference. More specifically, inference is formulated via the program:
(6) 
where each is constrained to lie in the dimensional probability simplex . This task can be solved using any constrained optimization method. However, for nonconcave the inference solution might only be approximate.
NLStruct. SPENs do not support score functions that contain a structured component. In response, Graber et al. (2018) introduced NLStruct, which combines a classical structured score function with a nonlinear transformation applied on top of it to produce a final score. Given a set as defined previously, the NLStruct score function takes the following form:
(7) 
where is a vectorized form of the score function for a classical structured model, is a vector containing all marginals, ‘’ is the Hadamard product, and is a scalarvalued function.
For this model, inference is formulated as a constrained optimization problem, where :
(8) 
Forming the Lagrangian of this program and rearranging leads to the saddlepoint inference problem
(9) 
Notably, maximization over is solved using techniques developed for classical structured models^{1}^{1}1As mentioned, solving the maximization over tractably might require relaxing the marginal polytope to the local marginal polytope . For brevity, we will not repeat this fact whenever an inference problem of this form appears throughout the rest of this paper., and the saddlepoint problem is optimized using the primaldual algorithm of Chambolle and Pock (2011), which alternates between updating , , and .
3 Graph Structured Prediction Energy Nets
Graph Structured Prediction Energy Networks (GSPENs) generalize all aforementioned models. They combine both a classical structured component as well as a SPENlike component to score an entire set of predictions jointly. Additionally, the GSPEN score function is more general than that for NLStruct, and includes it as a special case. After describing the formulation of both the score function and the inference problem (Sec. 3.1), we discuss two approaches to solving inference (Sec. 3.2 and Sec. 3.3) that we found to work well in practice. Unlike the methods described previously for NLStruct, these approaches do not require solving a saddlepoint optimization problem.
3.1 GSPEN Model
The GSPEN score function is written as follows:
where vector contains one marginal per region per assignment of values to that region. This formulation allows for the use of a structured score function while also allowing
to score an entire prediction jointly. Hence, it is a combination of classical structured models and SPENs. For instance, we can construct a GSPEN model by summing a classical structured model and a multilayer perceptron that scores an entire label vector, in which case the score function takes the form
. Of course, this is one of many possible score functions that are supported by this formulation. Notably, we recover the NLStruct score function if we use and let .Given this model, the inference problem is
(10) 
As for classical structured models, the probabilities are constrained to lie in the marginal polytope. In addition we also consider a fractional entropy term over the predictions, leading to
(11) 
In the classical setting, adding an entropy term relates to Fenchel duality (Blondel et al., 2018)
. However, the GSPEN inference objective does not take the correct form to use this reasoning. We instead view this entropy as a regularizer for the predictions: it pushes predictions towards a uniform distribution, smoothing the inference objective, which we empirically observed to improve convergence. The results discussed below indicate that adding entropy leads to betterperforming models. Also note that it is possible to add a similar entropy term to the SPEN inference objective, which is mentioned by
Belanger and McCallum (2016) and Belanger et al. (2017).For inference in GSPEN, SPEN procedures cannot be used since they do not maintain the additional constraints imposed by the graphical model, i.e., the marginal polytope . We also cannot use the inference procedure developed for NLStruct, as the GSPEN score function does not take the same form. Therefore, in the following, we describe two inference algorithms that optimize the program while maintaining structural constraints.
3.2 FrankWolfe Inference
The FrankWolfe algorithm (Frank and Wolfe, 1956) is suitable because the objectives in Eqs. (10, 11) are nonlinear while the constraints are linear. Specifically, using (Frank and Wolfe, 1956), we compute a linear approximation of the objective at the current iterate, maximize this linear approximation subject to the constraints of the original problem, and take a step towards this maximum.
In Algorithm 1 we detail the steps to optimize Eq. (10). In every iteration we first calculate the gradient of the score function with respect to the marginals/beliefs using the current prediction as input. We denote this gradient using . The gradient of
depends on the specific function used and is computed via backpropagation. If entropy is part of the objective, an additional term of
is added to this gradient.Next we find the maximizing beliefs which is equivalent to inference for classical structured prediction: the constraint space is identical and the objective is a linear function of the marginals/beliefs. Hence, we solve this inner optimization using one of a number of techniques referenced in Sec. 5.
Convergence guarantees for FrankWolfe have been proven when the overall objective is concave, continuously differentiable, and has bounded curvature (Jaggi, 2013), which is the case when has these properties with respect to the marginals. This is true even when the inner optimization is only solved approximately, which is often the case due to standard approximations used for structured inference. When is nonconcave, convergence can still be guaranteed, but only to a local optimum (LacosteJulien, 2016). Note that entropy has unbounded curvature, therefore its inclusion in the objective precludes convergence guarantees. Other variants of the FrankWolfe algorithm exist which improve convergence in certain cases (Krishnan et al., 2015; LacosteJulien and Jaggi, 2015). We defer a study of these properties to future work.
3.3 Structured Entropic Mirror Descent
Mirror descent, another constrained optimization algorithm, is analogous to projected subgradient descent, albeit using a more general distance beyond the Euclidean one (Beck and Teboulle, 2003). This algorithm has been used in the past to solve inference for SPENs, where entropy was used as the link function and by normalizing over each coordinate independently (Belanger and McCallum, 2016). We similarly use entropy in our case. However, the additional constraints in form of the polytope require special care.
We summarize the structured entropic mirror descent inference for the proposed model in Algorithm 2. Each iteration of mirror descent updates the current prediction and dual vector in two steps: (1) is updated based on the current prediction . Using as the link function, this update step takes the form . As mentioned previously, the gradient of can be computed using backpropagation; (2) is updated by computing the maximizing argument of the Fenchel conjugate of the link function evaluated at . More specifically, is updated via
(12) 
which is identical to classical structured prediction.
When the inference objective is concave and Lipschitz continuous (i.e., when has these properties), this algorithm has also been proven to converge (Beck and Teboulle, 2003). Unfortunately, we are not aware of any convergence results if the inner optimization problem is solved approximately and if the objective is not concave. In practice, though, we did not observe any convergence issues during experimentation.
3.4 Learning GSPEN Models
GSPENs assign a score to an input and a prediction . An SSVM learning objective is applicable, which maximizes the margin between the scores assigned to the correct prediction and the inferred result. The full SSVM learning objective with added lossaugmented inference is summarized in Fig. 1. The learning procedure consists of computing the highestscoring prediction using one of the inference procedures described in Sec. 3.2 and Sec. 3.3 for each example in a minibatch and then updating the weights of the model towards making better predictions.
4 Experiments
Struct  SPEN  NLStruct  GSPEN  

OCR (size 1000)  0.40 s  0.60 s  68.56 s  8.41 s 
Tagging  18.85 s  30.49 s  208.96 s  171.65 s 
Bibtex  0.36 s  11.75 s  –  13.87 s 
Bookmarks  6.05 s  94.44 s  –  234.33 s 
NER  29.16 s  –  –  99.83 s 
To demonstrate the utility of our model and to compare inference and learning settings, we report results on the tasks of optical character recognition (OCR), image tagging, multilabel classification, and named entity recognition (NER). For each experiment, we use the following baselines: Unary is an unstructured model that does not explicitly model the correlations between output variables in any way. Struct is a classical deep structured model using neural network potentials. We follow the inference and learning formulation of
(Chen et al., 2015b), where inference consists of a message passing algorithm derived using block coordinate descent on a relaxation of the inference problem. SPEN and NLStruct represent the formulations discussed in Sec. 2. Finally, GSPEN represents Graph Structured Prediction Energy Networks, described in Sec. 3. For GSPENs, the inner structured inference problems are solved using the same algorithm as for Struct. To compare the runtime of these approaches, Table 1gives the average epoch compute time (
i.e., time to compute the inference objective and update model weights) during training for our models for each task. In general, GSPEN training was more efficient with respect to time than NLStruct but, expectedly, more expensive than SPEN. Additional experimental details, including hyperparameter settings, are provided in Appendix A.2.4.1 Optical Character Recognition (OCR)
For the OCR experiments, we generate data by selecting a list of 50 common 5letter English words, such as ‘close,’ ‘other,’ and ‘world.’ To create each data point, we choose a word from this list and render each letter as a 28x28 pixel image by selecting a random image of the letter from the Chars74k dataset (de Campos et al., 2009), randomly shifting, scaling, rotating, and interpolating with a random background image patch. A different pool of backgrounds and letter images was used for the training, validation, and test splits of the data. The task is to identify the words given 5 ordered images. We create three versions of this dataset using different interpolation factors of , where each pixel in the final image is computed as . See Fig. 2 for a sample from each dataset. This process was deliberately designed to ensure that information about the structure of the problem (i.e., which words exist in the data) is a strong signal, while the signal provided by each individual letter image can be adjusted. The training, validation, and test set sizes for each dataset are 10,000, 2,000, and 2,000, respectively. During training we vary the training data to be either 200, 1k or 10k.
To study the inference algorithm, we train four different GSPEN models on the dataset containing 1000 training points and using . Each model uses either FrankWolfe or Mirror Descent and included/excluded the entropy term. To maintain tractability of inference, we fix a maximum iteration count for each model. We additionally investigate the effect of this maximum count on final performance. Additionally, we run this experiment by initializing from two different Struct models, one being trained using entropy during inference and one being trained without entropy. The results for this set of experiments are shown in Fig. 3. Most configurations perform similarly across the number of iterations, indicating these choices are sufficient for convergence. When initializing from the models trained without entropy, we observe that both FrankWolfe without entropy and Mirror Descent with entropy performed comparably. When initializing from a model trained with entropy, the use of mirror descent with entropy led to much better results.
The results for all values of using a train dataset size of 1000 are presented in Fig. 3, and results for all train dataset sizes with are presented in Fig. 3. We observe that, in all cases, GSPEN outperforms all baselines. The degree to which GSPEN outperforms other models depends most on the amount of train data: with a sufficiently large amount of data, SPEN and GSPEN perform comparably. However, when less data is provided, GSPEN performance does not drop as sharply as that of SPEN initially. It is also worth noting that GSPEN outperformed NLStruct by a large margin. The NLStruct model is less stable due to its saddlepoint formulation. Therefore it is much harder to obtain good performance with this model.
4.2 Image Tagging
Next, we evaluate on the MIRFLICKR25k dataset (Huiskes and Lew, 2008), which consists of 25,000 images taken from Flickr. Each image is assigned a subset of 24 possible tags. The train/val/test sets for these experiments consist of 10,000/5,000/10,000 images, respectively.
We compare to NLStruct and SPEN. We initialize the structured portion of our GSPEN model using the pretrained DeepStruct model described by Graber et al. (2018), which consists of unary potentials produced from an AlexNet architecture (Krizhevsky et al., 2012) and linear pairwise potentials of the form , i.e., containing one weight per pair in the graph per assignment of values to that pair. A fullyconnected pairwise graph was used. The function for our GSPEN model consists of a 2layer MLP with 130 hidden units. It takes as input a concatenation of the unary potentials generated by the AlexNet model and the current prediction. Additionally, we train a SPEN model with the same number of layers and hidden units. We used FrankWolfe without entropy for both SPEN and GSPEN inference.
The results are shown in Fig. 4. GSPEN obtains similar test performance to the NLStruct model, and both outperform SPEN. However, the NLStruct model was run for 100 iterations during inference without reaching ‘convergence’ (change of objective smaller than threshold), while the GSPEN model required an average of 69 iterations to converge at training time and 52 iterations to converge at test time. Our approach has the advantage of requiring fewer variables to maintain during inference and requiring fewer iterations of inference to converge. The final test losses for SPEN, NLStruct and GSPEN are 2.158, 2.037, and 2.029, respectively.
4.3 Multilabel Classification
We use the Bibtex and Bookmarks multilabel datasets (Katakis et al., 2008). They consist of binaryvalued input feature vectors, each of which is assigned some subset of 159/208 possible labels for Bibtex/Bookmarks, respectively. We train unary and SPEN models with architectures identical to (Belanger and McCallum, 2016) and (Gygli et al., 2017) but add dropout layers. In addition, we further regularize the unary model by flipping each bit of the input vectors with probability 0.01 when sampling minibatches. For Struct and GSPEN, we generate a graph by first finding the label variable that is active in most training data label vectors and add edges connecting every other variable to this most active one. Pairwise potentials are generated by passing the input vector through a 2layer MLP with 1k hidden units. The GSPEN model is trained by starting from the SPEN model, fixing its parameters, and training the pairwise potentials.
The results are in Table 2 alongside those taken from (Belanger and McCallum, 2016) and (Gygli et al., 2017). We found the Unary models to perform similarly to or better than previous best results. Both SPEN and Struct are able to improve upon these Unary results. GSPEN outperforms all configurations, suggesting that the contributions of the SPEN component and the Struct component to the score function are complementary.
4.4 Ner
We also assess suitability for Named Entity Recognition (NER) using the English portion of the CoNLL 2003 shared task (Tjong Kim Sang and De Meulder, 2003). To demonstrate the applicability of GSPEN for this task, we transformed two separate models, specifically the ones presented by Ma and Hovy (2016) and Akbik et al. (2018), into GSPENs by taking their respective score functions and adding a component that jointly scores an entire set of predictions. In each case, we first train six instances of the structured model using different random initializations and drop the model that performs the worst on validation data. We then train the GSPEN model, initializing the structured component from these pretrained models.
The final average performance is presented in Table 3, and individual trial information can be found in Table 4 in the appendix. When comparing to the model described by Ma and Hovy (2016), GSPEN improves the final test performance in four out of the five trials, and GSPEN has a higher overall average performance across both validation and test data. Compared to Akbik et al. (2018), on average GSPEN’s validation score was higher, but it performed slightly worse at test time. Overall, these results demonstrate that it is straightforward to augment a taskspecific structured model with an additional prediction scoring function which can lead to improved final task performance.
Bibtex Bookmarks Validation Test Validation Test SPEN (Belanger and McCallum, 2016) – 42.2 – 34.4 DVN (Gygli et al., 2017) – 44.7 – 37.1 Unary 43.3 44.1 38.4 37.4 Struct 45.8 46.1 39.7 38.9 SPEN 46.6 46.5 40.2 39.2 GSPEN 47.5 48.6 41.2 40.7  Avg. Val. Avg. Test Struct (Ma and Hovy, 2016) 94.88 0.18 91.37 0.04 + GSPEN 94.97 0.16 91.51 0.17 Struct (Akbik et al., 2018) 95.88 0.10 92.79 0.08 + GSPEN 95.96 0.08 92.69 0.17 
5 Related Work
A variety of techniques have been developed to model structure among output variables, originating from seminal works of (Lafferty et al., 2001; Taskar et al., 2003; Tsochantaridis et al., 2005). These works focus on extending linear classification, both probabilistic and nonprobabilistic, to model the correlation among output variables. Generally speaking, scores representing both predictions for individual output variables and for combinations of output variables are used. A plethora of techniques have been developed to solve inference for problems of this form, e.g., Schlesinger (1976); Werner (2007); Boykov et al. (1998, 2001); Wainwright and Jordan (2003); Globerson and Jaakkola (2006); Welling (2004); Sontag et al. (2012); Batra et al. (2011); Sontag et al. (2008); Sontag and Jaakkola (2007); Wainwright and Jordan (2008); Sontag and Jaakkola (2009); Murphy and Weiss (1999); Meshi et al. (2009); Globerson and Jaakkola (2007); Wainwright et al. (2005a, b, 2003); Heskes (2006); Hazan and Shashua (2010, 2008); Yanover et al. (2006); Meltzer et al. (2009); Weiss et al. (2007); Heskes (2002); Heskes et al. (2003); Yedidia et al. (2001); Ihler et al. (2004); Wiegerinck and Heskes (2003); Schwing et al. (2012, 2014, 2011b); Komodakis et al. (2010); Meshi et al. (2015); Meshi and Schwing (2017). As exact inference for general structures is NPhard (Shimony, 1994), early work focused on tractable exact inference. However, due to interest in modeling problems with intractable structure, a plethora of approaches have been studied for learning with approximate inference (Finley and Joachims, 2008; Kulesza and Pereira, 2008; Pletscher et al., 2010; Hazan and Urtasun, 2010; Meshi et al., 2010; Komodakis, 2011; Schwing et al., 2011a; Meshi et al., 2016). More recent work has also investigated the role of different types of prediction regularization, with Niculae et al. (2018) replacing the oftenused entropy regularization with an L2 norm and Blondel et al. (2018) casting both as special cases of a FenchelYoung loss framework.
To model both nonlinearity and structure, deep learning and structured prediction techniques were combined. Initially, local, pervariable score functions were learned with deep nets and correlations among output variables were learned in a separate second stage
(Alvarez et al., 2012; Chen et al., 2015a). Later work simplified this process, learning both local score functions and variable correlations jointly (Tompson et al., 2014; Zheng et al., 2015; Chen et al., 2015b; Schwing and Urtasun, 2015; Lin et al., 2015).Structured Prediction Energy Networks (SPENs), introduced by Belanger and McCallum (2016), take a different approach to modeling structure. Instead of explicitly specifying a structure apriori and enumerating scores for every assignment of labels to regions, SPENs learn a function which assigns a score to an input and a label. Inference uses gradientbased optimization to maximize the score w.r.t. the label. Belanger et al. (2017) extend this technique by unrolling inference in a manner inspired by Domke (2012). Both approaches involve iterative inference procedures, which are slower than feedforward prediction of deep nets. To improve inference speed, Tu and Gimpel (2018) learn a neural net to produce the same output as the gradientbased methods. Deep Value Networks (Gygli et al., 2017) follow the same approach of Belanger and McCallum (2016) but use a different objective that encourages the score to equal the task loss of the prediction. All these approaches do not permit to include known structure. The proposed approach enables this.
Our approach is most similar to our earlier work Graber et al. (2018), which combines explicitlyspecified structured potentials with a SPENlike score function. The score function of our earlier work is a special case of the one presented here. In fact, earlier we required a classical structured prediction model as an intermediate layer of the score function, while we don’t make this assumption any longer. Additionally, in our earlier work we had to solve inference via a computationally challenging saddlepoint objective.
6 Conclusions
The developed GSPEN model combines the strengths of several prior approaches to solving structured prediction problems. It allows machine learning practitioners to include inductive bias in the form of known structure into a model while implicitly capturing higherorder correlations among output variables. The model formulation described here is more general than previous attempts to combine explicit local and implicit global structure modeling while not requiring inference to solve a saddlepoint problem.
Acknowledgments
This work is supported in part by NSF under Grant No. 1718221 and MRI #1725729, UIUC, Samsung, 3M, Cisco Systems Inc. (Gift Award CG 1377144) and Adobe. We thank NVIDIA for providing GPUs used for this work and Cisco for access to the Arcetri cluster.
References
 Contextual string embeddings for sequence labeling. In Proc. COLING, Cited by: §A.1, §A.2, Table 4, §4.4, §4.4, Table 3.
 Semantic road segmentation via multiscale ensembles of learned features. In Proc. ECCV, Cited by: §5.
 Tighter Relaxations for MAPMRF Inference: A Local PrimalDual Gap Based Separation Algorithm. In Proc. AISTATS, Cited by: §5.
 Mirror descent and nonlinear projected subgradient methods for convex optimization. Oper. Res. Lett.. Cited by: §3.3, §3.3.
 Structured Prediction Energy Networks. In Proc. ICML, Cited by: §1, §2, §3.1, §3.3, §4.3, §4.3, Table 2, §5.
 Endtoend learning for structured prediction energy networks. In Proc. ICML, Cited by: §1, §3.1, §5.

Learning classifiers with fenchelyoung losses: generalized entropies, margins, and algorithms
. arXiv preprint arXiv:1805.09717. Cited by: §2, §3.1, §5.  Markov Random Fields with Efficient Approximations. In Proc. CVPR, Cited by: §5.
 Fast Approximate Energy Minimization via Graph Cuts. PAMI. Cited by: §5.
 A firstorder primaldual algorithm for convex problems with applications to imaging. JMIV. Cited by: §2.
 Semantic image segmentation with deep convolutional nets and fully connected crfs. In Proc. ICLR, Cited by: §5.
 Learning Deep Structured Models. In Proc. ICML, Cited by: §1, §4, §5.
 Character recognition in natural images. In Proc. VISAPP, Cited by: §4.1.
 Generic methods for optimizationbased modeling. In Proc. AISTATS, Cited by: §5.
 Training structural SVMs when exact inference is intractable. In Proc. ICML, Cited by: §5.
 An algorithm for quadratic programming. NRLQ. Cited by: §3.2.
 Approximate Inference Using Planar Graph Decomposition. In Proc. NIPS, Cited by: §5.
 Fixing MaxProduct: Convergent Message Passing Algorithms for MAP LPRelaxations. In Proc. NIPS, Cited by: §5.
 Deep structured prediction with nonlinear output transformations. In Proc. NeurIPS, Cited by: §A.2, §A.2, §1, §1, §2, §4.2, §5.
 Deep value networks learn to evaluate and iteratively refine structured outputs. In ICML, Cited by: §4.3, §4.3, Table 2, §5.
 Convergent MessagePassing Algorithms for Inference Over General Graphs with Convex Free Energies. In Proc. UAI, Cited by: §5.
 NormProduct Belief Propagation: PrimalDual MessagePassing for LPRelaxation and Approximate Inference. Trans. Information Theory. Cited by: §5.
 A PrimalDual MessagePassing Algorithm for Approximated Large Scale Structured Prediction. In Proc. NIPS, Cited by: §5.
 Approximate Inference and Constrained Optimization. In Proc. UAI, Cited by: §2, §5.
 Stable Fixed Points of Loopy Belief Propagation are Minima of the Bethe Free Energy. In Proc. NIPS, Cited by: §5.
 Convexity Arguments for Efficient Minimization of the Bethe and Kikuchi Free Energies. AI Research. Cited by: §5.
 The mir flickr retrieval evaluation. In Proc. ICMR, Cited by: §4.2.
 Message Errors in Belief Propagation. In Proc. NIPS, Cited by: §5.
 Revisiting frankwolfe: projectionfree sparse convex optimization. In Proc. ICML, Cited by: §3.2.
 Multilabel text classification for automated tag suggestion. ECML PKDD Discovery Challenge 2008. Cited by: §4.3.
 MRF Energy Minimization & Beyond via Dual Decomposition. PAMI. Cited by: §5.
 Efficient training for pairwise or higher order crfs via dual decomposition. In Proc. CVPR, Cited by: §5.
 Barrier frankwolfe for marginal inference. In Proc. NIPS, Cited by: §3.2.
 ImageNet Classification with Deep Convolutional Neural Networks. In Proc. NIPS, Cited by: §4.2.
 Structured learning with approximate inference. In Proc. NIPS, Cited by: §5.
 On the global linear convergence of frankwolfe optimization variants. In Proc. NIPS, Cited by: §3.2.
 Convergence rate of frankwolfe for nonconvex objectives. arXiv preprint arXiv:1607.00345. Cited by: §3.2.
 Conditional Random Fields: Probabilistic Models for segmenting and labeling sequence data. In Proc. ICML, Cited by: §5.
 Neural architectures for named entity recognition. In Proc. NAACLHLT, Cited by: §A.2.
 Deeply learning the messages in message passing inference. In Proc. NIPS, Cited by: §5.
 Endtoend sequence labeling via bidirectional lstmcnnscrf. In Proc. ACL, Cited by: §A.1, §A.2, Table 4, §4.4, §4.4, Table 3.
 Convergent Message Passing Algorithms: A Unifying View. In Proc. UAI, Cited by: §5.
 Convexifying the Bethe Free Energy. In Proc. UAI, Cited by: §5.
 Smooth and Strong: MAP Inference with Linear Convergence. In Proc. NIPS, Cited by: §5.
 Train and test tightness of LP relaxations in structured prediction. In Proc. ICML, Cited by: §5.
 Asynchronous Parallel Coordinate Minimization for MAP Inference. In Proc. NIPS, Cited by: §5.
 Learning Efficiently with Approximate Inference via Dual Losses. In Proc. ICML, Cited by: §5.
 Loopy Belief Propagation for Approximate Inference: An Empirical Study. In Proc. UAI, Cited by: §5.
 SparseMAP: differentiable sparse structured inference. In Proc. ICML, Cited by: §2, §5.
 Glove: global vectors for word representation. In Proc. EMNLP, Cited by: §A.2, §A.2.
 Entropy and Margin Maximization for Structured Output Learning. In Proc. ECML PKDD, Cited by: §5.
 Sintaksicheskiy analiz dvumernykh zritelnikh signalov v usloviyakh pomekh (Syntactic Analysis of TwoDimensional Visual Signals in Noisy Conditions). Kibernetika. Cited by: §5.
 Distributed Message Passing for Large Scale Graphical Models. In Proc. CVPR, Cited by: §5.
 Distributed Message Passing for Large Scale Graphical Models. In Proc. CVPR, Cited by: §5.
 Globally Convergent Dual MAP LP Relaxation Solvers Using FenchelYoung Margins. In Proc. NIPS, Cited by: §5.
 Globally Convergent Parallel MAP LP Relaxation Solver Using the FrankWolfe Algorithm. In Proc. ICML, Cited by: §5.
 Fully Connected Deep Structured Networks. arXiv preprint arXiv:1503.02351. Cited by: §1, §5.
 Finding MAPs for Belief Networks is NPhard. AI. Cited by: §2, §5.
 Efficiently Searching for Frustrated Cycles in MAP Inference. In Proc. UAI, Cited by: §5.
 New Outer Bounds on the Marginal Polytope. In Proc. NIPS, Cited by: §5.
 Tree Block Coordinate Descent for MAP in Graphical Models. In Proc. AISTATS, Cited by: §5.
 Tightening LP Relaxations for MAP Using Message Passing. In Proc. NIPS, Cited by: §5.
 MaxMargin Markov Networks. In Proc. NIPS, Cited by: §1, §2, §5.
 Introduction to the conll2003 shared task: languageindependent named entity recognition. In Proc. NAACLHLT, Cited by: §4.4.

Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation
. In Proc. NIPS, Cited by: §5.  Large Margin Methods for Structured and Interdependent Output Variables. JMLR. Cited by: §1, §2, §5.
 Learning approximate inference networks for structured prediction. In Proc. ICLR, Cited by: §5.
 TreeBased Reparameterization Framework for Analysis of SumProduct and Related Algorithms. Trans. Information Theory. Cited by: §5.
 A New Class of Upper Bounds on the Log Partition Function. Trans. Information Theory. Cited by: §5.
 MAP Estimation via Agreement on Trees: MessagePassing and Linear Programming. Trans. Information Theory. Cited by: §5.
 Variational Inference in Graphical Models: The View from the Marginal Polytope. In Proc. Conf. on Control, Communication and Computing, Cited by: §5.
 Graphical Models, Exponential Families and Variational Inference. FTML. Cited by: §2, §2, §5.
 MAP Estimation, Linear Programming and Belief Propagation with Convex Free Energies. In Proc. UAI, Cited by: §5.
 On the Choice of Regions for Generalized Belief Propagation. In Proc. UAI, Cited by: §5.
 A Linear Programming Approach to MaxSum Problem: A Review. PAMI. Cited by: §5.
 Fractional Belief Propagation. In Proc. NIPS, Cited by: §5.
 Linear Programming Relaxations and Belief Propagation – An Empirical Study. JMLR. Cited by: §5.
 Generalized Belief Propagation. In Proc. NIPS, Cited by: §5.

Conditional random fields as recurrent neural networks
. In Proc. ICCV, Cited by: §5.
Appendix A Appendix
In this Appendix, we first present additional experimental results for the NER before providing additional details on experiments, including model architectures and selection of hyperparameters.
a.1 Additional Experimental Results
Ma and Hovy (2016)  + GSPEN  
Trial  Val. F1  Test F1  Val. F1  Test F1 
Akbik et al. (2018)  + GSPEN  
Table 4 contains the results for all trials of the NER experiments. When comparing to Ma and Hovy (2016), we outperform their model on both validation and test data in four out of five trials. When comparing to Akbik et al. (2018), we outperform their model on validation data in four out of five trials, but only outperform their model on test data in one trial.
a.2 Additional Experimental Details
General Details:
Unless otherwise specified, all Struct models were trained by using the corresponding pretrained Unary model, fixing these parameters, and training pairwise potentials. All SPEN models were trained by using the pretrained Unary model, fixing these parameters, and training the
function. Early stopping based on task performance on validation was used to select the number of epochs for training. For SPEN, GSPEN, and Struct models, lossaugmented inference was used where the loss function equals the sum of the
 losses per output variable, i.e., where is the number of output variables.Ocr:
The Unary model is a single
layer multilayer perceptron (MLP) with ReLU activations, hidden layer sizes of
, and a dropout layer after the first linear layer with keep probability . Scores for each image were generated by independently passing them into this network. Both Struct and GSPEN use a graph with one pairwise region per pair of adjacent letters, for a total of pairs. Linear potentials are used, containing one entry per pair per set of assignments of values to each pair. The score function for both SPEN and GSPEN takes the form , where in the SPEN case contains only unary regions and in the GSPEN case consists of the graph used by Struct. Each represents the outputs of the same model as Unary/Struct for SPEN/GSPEN, respectively, and represents the vector . For every SPEN and GSPEN model trained, is a layer MLP with softplus activations, an output size of , and either , , or hidden units. These hidden sizes as well as the number of epochs of training for each model were determined based on task performance on the validation data. Messagepassing inference used in both Struct and GSPEN ran for iterations. GSPEN models were trained by using the pretrained Struct model, fixing these parameters, and training the function. The NLStruct model consisted of a layer MLP with hidden units, an output size of , and softplus activations. We use the same initialization described by Graber et al. (2018)for their word recognition experiments, where the first linear layer was initialized to the identity matrix and the second linear layer was initialized to a vector of all 1s. NLStruct models were initialized from the Struct models trained without entropy and used fixed potentials. The inference configuration described by
Graber et al. (2018) was used, where inference was run for iterations with averaging applied over the final iterations.All settings for the OCR experiments used a minibatch size of and used the Adam optimizer, with Unary, SPEN, and GSPEN using a learning rate of and Struct using a learning rate of . Gradients were clipped to a norm of before updates were applied. Inference in both SPEN and GSPEN were run for a maximum of iterations. Inference was terminated early for both models if the inference objective for all datapoints in the minibatch being processed changed by less than .
Three different versions of every model, initialized using different random seeds, were trained for these experiments. The plots represent the average of these trials, and the error represented is the standard deviation of these trials.
Tagging:
The SPEN and GSPEN models for the image tagging experiment used the same scoring function form as for the OCR experiments. The model in both cases is a layer MLP with softplus activations and hidden units. Both GSPEN and SPEN use the same score function as in the OCR experiments, with the exception that the function used for GSPEN is only a function of the beliefs and does not include the potentials as input. Both models were trained using gradient descent with a learning rate of , a momentum of , and a minibatch size of . Once again, only the component was trained for GSPEN, and the pairwise potentials were initialized to a Struct model trained using the settings described in Graber et al. (2018).
The messagepassing procedure used to solve the inner optimization problem for GSPEN was run for iterations per iteration of FrankWolfe. Inference for SPEN and GSPEN was run for iterations and was terminated early if the inference objective for all datapoints in the minibatch being processed changed by less than .
Multilabel Classification:
For the Bibtex dataset, percent of the training data was set aside to be used as validation data; this was not necessary for Bookmarks, which has a prespecified validation dataset. For prediction in both datasets and for all models, a threshold determining the boundary between positive/negative label predictions was tuned on the validation dataset.
For the Bibtex dataset, the Unary model consists of a layer MLP taking the binary feature vectors as input and returning a dimensional vector representing the potentials for label assignments ; the potentials for are fixed to . The Unary model uses ReLU activations, hidden unit sizes of , and dropout layers before the first and second linear layers with keep probability of . The Struct model consists of a layer MLP which also uses the feature vector as input, and it contains hidden units and ReLU activations. The SPEN model uses the same scoring function form as used in the previous experiments, except the function is only a function of the prediction vector and does not use the unary potentials as input. The model consists of a layer MLP which takes the vector as input. This model has hidden units, an output size of , and uses softplus activations. The GSPEN model was trained by starting from the SPEN model, fixing these parameters, and training a pairwise model with the same architecture as the Struct model.
For the bookmarks dataset, the models use the same architectures with slightly different configurations. the Unary model consists of a similar layer MLP, except dropout is only applied before the second linear layer. The Struct model uses the same architecture as the one trained on the Bibtex data. The model for SPEN/GSPEN uses hidden units.
For both datasets and for both SPEN and GSPEN, mirror descent was used for inference with an additional entropy term with a coefficient of ; for Struct, a coefficient of was used. Inference was run for iterations, with early termination as described previously using the same threshold. For Struct and GSPEN, message passing inference was run for iterations. The Unary model was trained using gradient descent with a learning rate of and a momentum of , while Struct, SPEN and GSPEN were trained using the Adam optimizer with a learning rate of .
Ner:
Both structured model baselines were trained using code provided by the authors of the respective papers. In both cases, hyperparameter choices for these structured models were chosen to be identical to the the choices made from their original works. For completeness, we will review these choices.
The structured model of Ma and Hovy (2016) first produces a vector for each word in the input sentence by concatenating two vectors: the first is a dimensional embedding for the word, which is initialized from pretrained GloVe embeddings (Pennington et al., 2014) and finetuned. The second is the output of a D convolutional deep net with filters of length taking as input dimensional character embeddings for each character in the word. These representations are then passed into a layer bidirectional LSTM with a hidden state size of , which is passed through a linear layer followed by an ELU activation to produce an intermediate representation. Unary/pairwise graphical model scores are finally obtained by passing this intermediate representation through two further linear layers. Predictions are made using the Viterbi algorithm. Dropout is applied to the embeddings before they are fed into the RNN (zeroing probability of , corresponding to two separate dropout layers with zeroing probability of being applied) and to the output hidden states of the RNN (zeroing probability of
). The GSPEN models in this setting were trained by initializing the structured component from the pretrained models and fixing them – that is, only the parameters in the MLP were trained during this step. Due to the fact that the input sentences are of varying size, we zeropad all inputs of the MLP to the maximum sequence length. Dropout with a zeroing probability of
was additionally applied to the inputs of the MLP. Inference was conducted using mirror descent with added entropy and convergence threshold of . For both the structured baseline and GSPEN, model parameters were trained for epochs using SGD with initial learning rate of , which was decayed every epoch using the formula . The structured baseline was trained with a minibatch size of , while the GSPEN model used a minibatch size of during training. A larger batch size was used for the GSPEN model to decrease the amount of time to complete one pass through the data.Akbik et al. (2018) use a concatenation of three different pretrained embeddings per token as input to the bidirectional LSTM. The first is generated by a bidirectional LSTM which takes characterlevel embeddings as input and is pretrained using a characterbased language modeling objective (see Akbik et al. (2018) for more details). The other two embeddings are GloVe word embeddings (Pennington et al., 2014), and tasktrained characterbased embeddings (as specified by Lample et al. (2016)). During training, these embeddings are finetuned by passing them through a linear layer whose parameters are learned. The embeddings are passed into a layer bidirectional LSTM with a hidden state size of . Unary scores are generated from the outputs of the LSTM by passing them through a linear layer; pairwise scores consist of a matrix of scores for every pair of labels, which are shared across sentence indices. In this setting, the GSPEN models were trained by initializing the structured component from the pretrained models and then finetuning all of the model parameters. Mirror descent with added entropy was used for GSPEN inference with a convergence threshold of . For both the structured baseline and GSPEN, model parameters were trained for a maximum of epochs using SGD with minibatch size of and initial learning rate of , which was decayed by when the training loss did not decrease past its current minimum for epochs. Training was terminated early if the learning rate fell below .
Comments
There are no comments yet.