A Linearly Convergent Douglas-Rachford Splitting Solver for Markovian Information-Theoretic Optimization Problems
share
In this work, we propose solving the Information bottleneck (IB) and Privacy Funnel (PF) problems with Douglas-Rachford Splitting methods (DRS). We study a general Markovian information-theoretic Lagrangian that formulates the IB and PF problems into a convex-weakly convex pair of functions in a unified framework. Exploiting the recent non-convex convergence analysis for splitting methods, we prove the linear convergence of the proposed algorithms using the Kurdyka-Łojasiewicz inequality. Moreover, our analysis is beyond IB and PF and applies to any convex-weakly convex pair objectives. Based on the results, we develop two types of IB solvers, with one improves the performance of convergence over existing solvers while the other is linearly convergent independent to the relevance-compression trade-off, and a class of PF solvers that can handle both random and deterministic mappings. Empirically, we evaluate the proposed DRS solvers for both the IB and PF problems with our gradient-descent-based implementation. For IB, the proposed solvers result in solutions that are comparable to those obtained through the Blahut-Arimoto-based benchmark and is convergent for a wider range of the penalty coefficient than existing solvers. For PF, our non-greedy solvers can explore the information plane better than the clustering-based greedy solvers.
READ FULL TEXT