Reinforcement Learning and Imitation Learning achieved impressive empirical results in recent years. Still, little is known about the theoretical properties of commonly used algorithms. This PhD project aims to use insights from the field of game theory and optimization to develop algorithms with mathematical guarantees. The hope is to bring contributions on the practical RL side as well, proposing theory grounded algorithms easily applicable and implementable.