Overview and Investigation of Algorithms for the Information Bottleneck Method

Abstract:
Lossy data compression has been studied under the celebrated Rate-Distortion theory which provides the compression rate in order to quantize a signal without exceeding a given distortion measure. Recently, with information bottleneck an alternative approach has been emerged in the field of machine learning. The fundamental idea is to include the original source into the problem setup when quantizing an observation variable and to use strictly information theoretic measures to design the quantizer. This paper yields an insight to this framework, discusses corresponding algorithms and their performance, and provides a new algorithmic approach of low complexity.