Information Limits for Recovering a Hidden Community

09/25/2015
by   Bruce Hajek, et al.
0

We study the problem of recovering a hidden community of cardinality K from an n × n symmetric data matrix A, where for distinct indices i,j, A_ij∼ P if i, j both belong to the community and A_ij∼ Q otherwise, for two known probability distributions P and Q depending on n. If P= Bern(p) and Q= Bern(q) with p>q, it reduces to the problem of finding a densely-connected K-subgraph planted in a large Erdös-Rényi graph; if P=N(μ,1) and Q=N(0,1) with μ>0, it corresponds to the problem of locating a K × K principal submatrix of elevated means in a large Gaussian random matrix. We focus on two types of asymptotic recovery guarantees as n →∞: (1) weak recovery: expected number of classification errors is o(K); (2) exact recovery: probability of classifying all indices correctly converges to one. Under mild assumptions on P and Q, and allowing the community size to scale sublinearly with n, we derive a set of sufficient conditions and a set of necessary conditions for recovery, which are asymptotically tight with sharp constants. The results hold in particular for the Gaussian case, and for the case of bounded log likelihood ratio, including the Bernoulli case whenever p/q and 1-p/1-q are bounded away from zero and infinity. An important algorithmic implication is that, whenever exact recovery is information theoretically possible, any algorithm that provides weak recovery when the community size is concentrated near K can be upgraded to achieve exact recovery in linear additional time by a simple voting procedure.

READ FULL TEXT

page 1

page 2

page 3

page 4

02/20/2016

Semidefinite Programs for Exact Recovery of a Hidden Community

We study a semidefinite programming (SDP) relaxation of the maximum like...
09/05/2018

Recovering a Single Community with Side Information

We study the effect of the quality and quantity of side information on t...
11/18/2019

Consistent recovery threshold of hidden nearest neighbor graphs

Motivated by applications such as discovering strong ties in social netw...
11/03/2021

On role extraction for digraphs via neighbourhood pattern similarity

We analyse the recovery of different roles in a network modelled by a di...
02/08/2021

Community Detection: Exact Recovery in Weighted Graphs

In community detection, the exact recovery of communities (clusters) has...
12/30/2021

Variable selection, monotone likelihood ratio and group sparsity

In the pivotal variable selection problem, we derive the exact non-asymp...
08/26/2019

The Overlap Gap Property in Principal Submatrix Recovery

We study support recovery for a k × k principal submatrix with elevated ...