Mapping Energy Landscapes of Non-Convex Learning Problems

by   Maria Pavlovskaia, et al.

In many statistical learning problems, the target functions to be optimized are highly non-convex in various model spaces and thus are difficult to analyze. In this paper, we compute Energy Landscape Maps (ELMs) which characterize and visualize an energy function with a tree structure, in which each leaf node represents a local minimum and each non-leaf node represents the barrier between adjacent energy basins. The ELM also associates each node with the estimated probability mass and volume for the corresponding energy basin. We construct ELMs by adopting the generalized Wang-Landau algorithm and multi-domain sampler that simulates a Markov chain traversing the model space by dynamically reweighting the energy function. We construct ELMs in the model space for two classic statistical learning problems: i) clustering with Gaussian mixture models or Bernoulli templates; and ii) bi-clustering. We propose a way to measure the difficulties (or complexity) of these learning problems and study how various conditions affect the landscape complexity, such as separability of the clusters, the number of examples, and the level of supervision; and we also visualize the behaviors of different algorithms, such as K-mean, EM, two-step EM and Swendsen-Wang cuts, in the energy landscapes.


page 3

page 5

page 20

page 22

page 24

page 26


An Analysis of Constant Step Size SGD in the Non-convex Regime: Asymptotic Normality and Bias

Structured non-convex learning problems, for which critical points have ...

An Evolutionary Algorithm with Crossover and Mutation for Model-Based Clustering

The expectation-maximization (EM) algorithm is almost ubiquitous for par...

A fast and efficient Modal EM algorithm for Gaussian mixtures

In the modal approach to clustering, clusters are defined as the local m...

The Landscape of Non-convex Empirical Risk with Degenerate Population Risk

The landscape of empirical risk has been widely studied in a series of m...

Clustering of check-in sequences using the mixture Markov chain process

This work is devoted to the clustering of check-in sequences from a geos...

Minimum Message Length Clustering Using Gibbs Sampling

The K-Mean and EM algorithms are popular in clustering and mixture model...

Efficient Clustering for Stretched Mixtures: Landscape and Optimality

This paper considers a canonical clustering problem where one receives u...

Please sign up or login with your details

Forgot password? Click here to reset