Estimation for High-Dimensional Multi-Layer Generalized Linear Model – Part I: The Exact MMSE Estimator

by   Haochuan Zhang, et al.

This two-part work considers the minimum means square error (MMSE) estimation problem for a high dimensional multi-layer generalized linear model (ML-GLM), which resembles a feed-forward fully connected deep learning network in that each of its layer mixes up the random input with a known weighting matrix and activates the results via non-linear functions, except that the activation here is stochastic and following some random distribution. Part I of the work focuses on the exact MMSE estimator, whose implementation is long known infeasible. For this exact estimator, an asymptotic analysis on the performance is carried out using a new replica method that is refined from certain aspects. A decoupling principle is then established, suggesting that, in terms of joint input-and-estimate distribution, the original estimation problem of multiple-input multiple-output is indeed identical to a simple single-input single-output one subjected to additive white Gaussian noise (AWGN) only. The variance of the AWGN is further shown to be determined by some coupled equations, whose dependency on the weighting and activation is given explicitly and analytically. Comparing to existing results, this paper is the first to offer a decoupling principle for the ML-GLM estimation problem. To further address the implementation issue of an exact solution, Part II proposes an approximate estimator, ML-GAMP, whose per-iteration complexity is as low as GAMP, while its asymptotic MSE (if converged) is as optimal as the exact MMSE estimator.



There are no comments yet.


page 1


Estimation for High-Dimensional Multi-Layer Generalized Linear Model – Part II: The ML-GAMP Estimator

This is Part II of a two-part work on the estimation for a multi-layer g...

Multi-Layer Bilinear Generalized Approximate Message Passing

In this paper, we extend the bilinear generalized approximate message pa...

Inference with Deep Generative Priors in High Dimensions

Deep generative priors offer powerful models for complex-structured data...

Limiting free energy of multi-layer generalized linear models

We compute the high-dimensional limit of the free energy associated with...

Asymptotics of MAP Inference in Deep Networks

Deep generative priors are a powerful tool for reconstruction problems w...

Replica Analysis for Generalized Linear Regression with IID Row Prior

Different from a typical independent identically distributed (IID) eleme...

Randomized residual-based error estimators for the Proper Generalized Decomposition approximation of parametrized problems

This paper introduces a novel error estimator for the Proper Generalized...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.