Variational Pseudolikelihood for Regularized Ising Inference

by   Charles K. Fisher, et al.

I propose a variational approach to maximum pseudolikelihood inference of the Ising model. The variational algorithm is more computationally efficient, and does a better job predicting out-of-sample correlations than L_2 regularized maximum pseudolikelihood inference as well as mean field and isolated spin pair approximations with pseudocount regularization. The key to the approach is a variational energy that regularizes the inference problem by shrinking the couplings towards zero, while still allowing some large couplings to explain strong correlations. The utility of the variational pseudolikelihood approach is illustrated by training an Ising model to represent the letters A-J using samples of letters from different computer fonts.



There are no comments yet.


page 2

page 4


Copula variational inference

We develop a general variational inference method that preserves depende...

Solving Statistical Mechanics using Variational Autoregressive Networks

We propose a general framework for solving statistical mechanics of syst...

Energy-Based Learning for Cooperative Games, with Applications to Feature/Data/Model Valuations

Valuation problems, such as attribution-based feature interpretation, da...

Variational Bayes algorithm and posterior consistency of Ising model parameter estimation

Ising models originated in statistical physics and are widely used in mo...

Mean field variational Bayesian inference for support vector machine classification

A mean field variational Bayes approach to support vector machines (SVMs...

Graph partition strategies for generalized mean field inference

An autonomous variational inference algorithm for arbitrary graphical mo...

A Factorized Variational Technique for Phase Unwrapping in Markov Random Fields

Some types of medical and topographic imaging device produce images in w...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.


  • Bialek et al. (2012) W. Bialek, A. Cavagna, I. Giardina, T. Mora, E. Silvestri, M. Viale,  and A. M. Walczak, Proceedings of the National Academy of Sciences 109, 4786 (2012).
  • Tkačik et al. (2013) G. Tkačik, O. Marre, T. Mora, D. Amodei, M. J. Berry II,  and W. Bialek, Journal of Statistical Mechanics: Theory and Experiment 2013, P03011 (2013).
  • Cocco and Monasson (2011a) S. Cocco and R. Monasson, BMC Neuroscience 12, P224 (2011a).
  • Roudi et al. (2009a) Y. Roudi, J. Tyrcha,  and J. Hertz, Physical Review E 79, 051915 (2009a).
  • Cocco et al. (2013a) S. Cocco, R. Monasson,  and M. Weigt, PLoS computational biology 9, e1003176 (2013a).
  • Ekeberg et al. (2013) M. Ekeberg, C. Lövkvist, Y. Lan, M. Weigt,  and E. Aurell, Physical Review E 87, 012707 (2013).
  • Weigt et al. (2009) M. Weigt, R. A. White, H. Szurmant, J. A. Hoch,  and T. Hwa, Proceedings of the National Academy of Sciences 106, 67 (2009).
  • Hopf et al. (2012) T. A. Hopf, L. J. Colwell, R. Sheridan, B. Rost, C. Sander,  and D. S. Marks, Cell 149, 1607 (2012).
  • Schug et al. (2009) A. Schug, M. Weigt, J. N. Onuchic, T. Hwa,  and H. Szurmant, Proceedings of the National Academy of Sciences 106, 22124 (2009).
  • Mora et al. (2010) T. Mora, A. M. Walczak, W. Bialek,  and C. G. Callan, Proceedings of the National Academy of Sciences 107, 5405 (2010).
  • Santolini et al. (2013) M. Santolini, T. Mora,  and V. Hakim, arXiv preprint arXiv:1302.4424  (2013).
  • Jaynes (1982) E. T. Jaynes, Proceedings of the IEEE 70, 939 (1982).
  • Pressé et al. (2013) S. Pressé, K. Ghosh, J. Lee,  and K. A. Dill, Reviews of Modern Physics 85, 1115 (2013).
  • Roudi et al. (2009b) Y. Roudi, E. Aurell,  and J. A. Hertz, Frontiers in computational neuroscience 3 (2009b).
  • Kappen and Rodriguez (1998) H. J. Kappen and F. Rodriguez, Neural Computation 10, 1137 (1998).
  • Tanaka (1998) T. Tanaka, Physical Review E 58, 2302 (1998).
  • Sessak and Monasson (2009) V. Sessak and R. Monasson, Journal of Physics A: Mathematical and Theoretical 42, 055001 (2009).
  • Aurell and Ekeberg (2012) E. Aurell and M. Ekeberg, Physical review letters 108, 090201 (2012).
  • Nguyen and Berg (2012a) H. C. Nguyen and J. Berg, Journal of Statistical Mechanics: Theory and Experiment 2012, P03004 (2012a).
  • Mastromatteo (2013) I. Mastromatteo, Journal of Statistical Physics 150, 658 (2013).
  • Huang (2013) H. Huang, The European Physical Journal B 86, 1 (2013).
  • Cocco and Monasson (2012) S. Cocco and R. Monasson, Journal of Statistical Physics 147, 252 (2012).
  • Nguyen and Berg (2012b) H. C. Nguyen and J. Berg, Physical review letters 109, 050602 (2012b).
  • Huang and Kabashima (2013) H. Huang and Y. Kabashima, Physical Review E 87, 062129 (2013).
  • Cocco and Monasson (2011b) S. Cocco and R. Monasson, Physical review letters 106, 090601 (2011b).
  • Huang (2010) H. Huang, Physical Review E 81, 036104 (2010).
  • Aurell et al. (2010) E. Aurell, C. Ollion,  and Y. Roudi, The European Physical Journal B-Condensed Matter and Complex Systems 77, 587 (2010).
  • Barton et al. (2014) J. Barton, S. Cocco, E. De Leonardis,  and R. Monasson, arXiv preprint arXiv:1405.0233  (2014).
  • Hopfield (1982) J. J. Hopfield, Proceedings of the national academy of sciences 79, 2554 (1982).
  • Cocco et al. (2013b) S. Cocco, R. Monasson,  and M. Weigt, in Journal of Physics: Conference Series, Vol. 473 (IOP Publishing, 2013) p. 012010.
  • Tang and Sutskever (2011) Y. Tang and I. Sutskever, Data normalization in the learning of restricted Boltzmann machines, Tech. Rep. (Technical Report UTML-TR-11-2, Department of Computer Science, University of Toronto, 2011).
  • Ciresan et al. (2012) D. Ciresan, U. Meier,  and J. Schmidhuber, in Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on (IEEE, 2012) pp. 3642–3649.
  • Salakhutdinov et al. (2007) R. Salakhutdinov, A. Mnih,  and G. Hinton, in Proceedings of the 24th international conference on Machine learning (ACM, 2007) pp. 791–798.