DeepAI
Log In Sign Up

Fine-grained Generalization Analysis of Structured Output Prediction

05/31/2021
by   Waleed Mustafa, et al.
0

In machine learning we often encounter structured output prediction problems (SOPPs), i.e. problems where the output space admits a rich internal structure. Application domains where SOPPs naturally occur include natural language processing, speech recognition, and computer vision. Typical SOPPs have an extremely large label set, which grows exponentially as a function of the size of the output. Existing generalization analysis implies generalization bounds with at least a square-root dependency on the cardinality d of the label set, which can be vacuous in practice. In this paper, we significantly improve the state of the art by developing novel high-probability bounds with a logarithmic dependency on d. Moreover, we leverage the lens of algorithmic stability to develop generalization bounds in expectation without any dependency on d. Our results therefore build a solid theoretical foundation for learning in large-scale SOPPs. Furthermore, we extend our results to learning with weakly dependent data.

READ FULL TEXT

page 1

page 2

page 3

page 4

04/29/2021

Fine-grained Generalization Analysis of Vector-valued Learning

Many fundamental machine learning tasks can be formulated as a problem o...
02/27/2019

High probability generalization bounds for uniformly stable algorithms with nearly optimal rate

Algorithmic stability is a classical approach to understanding and analy...
09/13/2017

Joint Learning of Set Cardinality and State Distribution

We present a novel approach for learning to predict sets using deep lear...
10/13/2010

Online Multiple Kernel Learning for Structured Prediction

Despite the recent progress towards efficient multiple kernel learning (...
09/16/2022

Stability and Generalization for Markov Chain Stochastic Gradient Methods

Recently there is a large amount of work devoted to the study of Markov ...
08/06/2012

Structured Prediction Cascades

Structured prediction tasks pose a fundamental trade-off between the nee...
06/06/2018

Localized Structured Prediction

Key to structured prediction is exploiting the problem structure to simp...