Log In Sign Up

Fine-grained Generalization Analysis of Structured Output Prediction

by   Waleed Mustafa, et al.

In machine learning we often encounter structured output prediction problems (SOPPs), i.e. problems where the output space admits a rich internal structure. Application domains where SOPPs naturally occur include natural language processing, speech recognition, and computer vision. Typical SOPPs have an extremely large label set, which grows exponentially as a function of the size of the output. Existing generalization analysis implies generalization bounds with at least a square-root dependency on the cardinality d of the label set, which can be vacuous in practice. In this paper, we significantly improve the state of the art by developing novel high-probability bounds with a logarithmic dependency on d. Moreover, we leverage the lens of algorithmic stability to develop generalization bounds in expectation without any dependency on d. Our results therefore build a solid theoretical foundation for learning in large-scale SOPPs. Furthermore, we extend our results to learning with weakly dependent data.


page 1

page 2

page 3

page 4


Fine-grained Generalization Analysis of Vector-valued Learning

Many fundamental machine learning tasks can be formulated as a problem o...

High probability generalization bounds for uniformly stable algorithms with nearly optimal rate

Algorithmic stability is a classical approach to understanding and analy...

Joint Learning of Set Cardinality and State Distribution

We present a novel approach for learning to predict sets using deep lear...

Online Multiple Kernel Learning for Structured Prediction

Despite the recent progress towards efficient multiple kernel learning (...

Stability and Generalization for Markov Chain Stochastic Gradient Methods

Recently there is a large amount of work devoted to the study of Markov ...

Structured Prediction Cascades

Structured prediction tasks pose a fundamental trade-off between the nee...

Localized Structured Prediction

Key to structured prediction is exploiting the problem structure to simp...