Mining Associated Text and Images with Dual-Wing Harmoniums

07/04/2012
by   Eric P Xing, et al.
0

We propose a multi-wing harmonium model for mining multimedia data that extends and improves on earlier models based on two-layer random fields, which capture bidirectional dependencies between hidden topic aspects and observed inputs. This model can be viewed as an undirected counterpart of the two-layer directed models such as LDA for similar tasks, but bears significant difference in inference/learning cost tradeoffs, latent topic representations, and topic mixing mechanisms. In particular, our model facilitates efficient inference and robust topic mixing, and potentially provides high flexibilities in modeling the latent topic spaces. A contrastive divergence and a variational algorithm are derived for learning. We specialized our model to a dual-wing harmonium for captioned images, incorporating a multivariate Poisson for word-counts and a multivariate Gaussian for color histogram. We present empirical results on the applications of this model to classification, retrieval and image annotation on news video collections, and we report an extensive comparison with various extant models.

READ FULL TEXT
research
04/26/2020

Neural Topic Modeling with Bidirectional Adversarial Training

Recent years have witnessed a surge of interests of using neural topic m...
research
06/24/2015

Efficient Learning for Undirected Topic Models

Replicated Softmax model, a well-known undirected topic model, is powerf...
research
01/25/2023

Improving the Inference of Topic Models via Infinite Latent State Replications

In text mining, topic models are a type of probabilistic generative mode...
research
05/23/2013

A Supervised Neural Autoregressive Topic Model for Simultaneous Image Classification and Annotation

Topic modeling based on latent Dirichlet allocation (LDA) has been a fra...
research
09/13/2014

A Deep and Autoregressive Approach for Topic Modeling of Multimodal Data

Topic modeling based on latent Dirichlet allocation (LDA) has been a fra...
research
05/01/2017

Stochastic Divergence Minimization for Biterm Topic Model

As the emergence and the thriving development of social networks, a huge...
research
01/15/2013

Factorized Topic Models

In this paper we present a modification to a latent topic model, which m...

Please sign up or login with your details

Forgot password? Click here to reset