Generative modeling is a fundamental yet complex task in machine and deep learning. This project aims to investigate and broaden our understanding of the theoretical foundations of (deep) generative models. Our interest is twofold. First, we are interested in theoretical properties of the optimization and training dynamics, e.g., provable convergence, and how such properties transfer from theoretical abstractions to practical implementations. Following a complementary viewpoint, we study alternative characterizations of optimality in generative models, e.g., how learning objectives (at optimality) could be expressed via entropies reflecting fundamental model specifications. Such reformulations could serve as theoretically grounded guidelines to (re-)design (parts of) the model. Both lines of research include to rethink and improve upon existing techniques as well as to design novel approaches in order to support successful and meaningful generative learning.