DeepAI AI Chat
Log In Sign Up

Descriptive Dimensionality and Its Characterization of MDL-based Learning and Change Detection

by   Kenji Yamanishi, et al.

This paper introduces a new notion of dimensionality of probabilistic models from an information-theoretic view point. We call it the "descriptive dimension"(Ddim). We show that Ddim coincides with the number of independent parameters for the parametric class, and can further be extended to real-valued dimensionality when a number of models are mixed. The paper then derives the rate of convergence of the MDL (Minimum Description Length) learning algorithm which outputs a normalized maximum likelihood (NML) distribution with model of the shortest NML codelength. The paper proves that the rate is governed by Ddim. The paper also derives error probabilities of the MDL-based test for multiple model change detection. It proves that they are also governed by Ddim. Through the analysis, we demonstrate that Ddim is an intrinsic quantity which characterizes the performance of the MDL-based learning and change detection.


page 1

page 2

page 3

page 4


Score-Based Change Detection for Gradient-Based Learning Machines

The widespread use of machine learning algorithms calls for automatic ch...

A Probabilistic Spectral Analysis of Multivariate Real-Valued Nonstationary Signals

A class of multivariate spectral representations for real-valued nonstat...

Detecting Signs of Model Change with Continuous Model Selection Based on Descriptive Dimensionality

We address the issue of detecting changes of models that lie behind a da...

Word2vec Skip-gram Dimensionality Selection via Sequential Normalized Maximum Likelihood

In this paper, we propose a novel information criteria-based approach to...

Renormalized Normalized Maximum Likelihood and Three-Part Code Criteria For Learning Gaussian Networks

Score based learning (SBL) is a promising approach for learning Bayesian...

Detecting Hierarchical Changes in Latent Variable Models

This paper addresses the issue of detecting hierarchical changes in late...

IMS at SemEval-2020 Task 1: How low can you go? Dimensionality in Lexical Semantic Change Detection

We present the results of our system for SemEval-2020 Task 1 that exploi...