Learning Energy-Based Approximate Inference Networks for Structured Applications in NLP

by   Lifu Tu, et al.

Structured prediction in natural language processing (NLP) has a long history. The complex models of structured application come at the difficulty of learning and inference. These difficulties lead researchers to focus more on models with simple structure components (e.g., local classifier). Deep representation learning has become increasingly popular in recent years. The structure components of their method, on the other hand, are usually relatively simple. We concentrate on complex structured models in this dissertation. We provide a learning framework for complicated structured models as well as an inference method with a better speed/accuracy/search error trade-off. The dissertation begins with a general introduction to energy-based models. In NLP and other applications, an energy function is comparable to the concept of a scoring function. In this dissertation, we discuss the concept of the energy function and structured models with different energy functions. Then, we propose a method in which we train a neural network to do argmax inference under a structured energy function, referring to the trained networks as "inference networks" or "energy-based inference networks". We then develop ways of jointly learning energy functions and inference networks using an adversarial learning framework. Despite the inference and learning difficulties of energy-based models, we present approaches in this thesis that enable energy-based models more easily to be applied in structured NLP applications.


Improving Joint Training of Inference Networks and Structured Prediction Energy Networks

Deep energy-based models are powerful, but pose challenges for learning ...

An Exploration of Arbitrary-Order Sequence Labeling via Energy-Based Inference Networks

Many tasks in natural language processing involve predicting structured ...

Learning Discrete Energy-based Models via Auxiliary-variable Local Exploration

Discrete structures play an important role in applications like program ...

Learning Discriminators as Energy Networks in Adversarial Learning

We propose a novel framework for structured prediction via adversarial l...

Scaling Matters in Deep Structured-Prediction Models

Deep structured-prediction energy-based models combine the expressive po...

Benchmarking Approximate Inference Methods for Neural Structured Prediction

Exact structured inference with neural network scoring functions is comp...

Graph Structured Prediction Energy Networks

For joint inference over multiple variables, a variety of structured pre...

Please sign up or login with your details

Forgot password? Click here to reset