Despite the recent success of reinforcement learning (RL), modern RL methods face numerous challenges that prohibit their wide usage in the real world. Reality is dynamic, open-ended, continually changing, and multi-agent by nature. Therefore, RL algorithms need to be robust to variation in their environments and other agents within them, and have the capability to transfer and adapt to unseen situations during deployment. The goal of this PhD project is to develop novel methods and paradigms for multi-agent RL in open-ended domains. We will strive to design dynamic learning regimes to train agents that are generally capable across a large space of tasks and co-players by using recent techniques from multi-agent learning, autocurricula, and unsupervised environment design.