Fynn Gerding

Information theory has proven to be a valuable tool for the analysis of deep learning: The information bottleneck theory of deep learning investigates learning dynamics; compression has been suggested as an explanation of good generalization performance; and information-theoretic bounds relate information stored in a network's weights to the generalization gap. In this project we extend the information-theoretic analysis of deep learning towards redundancy, which so far has mainly been discussed in the lottery ticket hypothesis. Building on partial information decomposition, a family of methods that decomposes target information contained in a neural representation into unique, synergistic, and redundant parts, we will study redundancy in neural networks. Specifically, we will try to understand how redundancy changes during training, and relate the redundancy in trained neural networks to properties such as generalization performance and pruning robustness. As an important side results, the project will develop computationally efficient estimators of the redundancy of high-dimensional representations.