Regularization lies at the core of successful training state-of-the-art deep neural networks. It allows to control overfitting and allows to obtain good generalization even with massively overparametrized models. Regularization influences the training process both implicitly - through the properties of optimizers - and explicitly - by using a regularized loss function, dropout, batch normalization, etc. The goal of this Project is to shed more light on the foundations of regularization techniques employed in deep learning and to formally ground empirical results using the insights from the regularization theory.