A Parametric Information Bottleneck Algorithm for Gaussian Random Variables and Gaussian Mixtures

Abstract:
Recently, the information bottleneck method, a machine learning framework, was incorporated in several communication engineering related applications. However, most of these applications are limited to discrete relevant random variables. This is mostly due to the lack of appropriate deterministic information bottleneck algorithms suitable for continuous random variables. In this paper, we present a novel deterministic information bottleneck algorithm, which we call the parametric information bottleneck algorithm, suitable for continuous relevant variables with a Gaussian distribution. We show that our proposed algorithm operates close to the theoretically achievable Gaussian information bottleneck bound. In addition, our proposed algorithm allows to efficiently compress any continuous random variable whose distribution can be approximated by a Gaussian mixture distribution. Exemplarily, using the proposed parametric information bottleneck algorithm, we devise a relevant-information- preserving temperature sensor. Although the resolution of the sensor’s analog-to-digital converter is only 5 bit, the proposed information bottleneck algorithm finds quantization regions such that 99.9% relevant information is preserved.