Causal Representations for Reinforcement Learning

Yashas Annadani (Ph.D. Student)

Systematic generalisation beyond the independent and identically distributed (IID) setting is crucial for learning agents interacting with the environment in the real-world. Improvements with regards to generalisation has been seen in supervised learning with large language and vision models and out-of-distribution generalisation has been shown with just massive amounts of data. However, in reinforcement learning such generalisation from large data is almost infeasible due to large state-spaces. Causality offers a way to deal with this problem as actions in reinforcement learning are inherently a form of intervention. To this end, the goal of this project is two fold: Firstly, we investigate how we can learn good causal models of data from reasonable set of assumptions as well as experimental design of performing interventions and acquiring interventional data. Secondly, we look at how these techniques can be useful in learning causal models of high-dimensional data where learning representations of data becomes imperative and a causal model is defined over this representation. This is useful in reinforcement learning where the agent usually has access to high-dimensional signals like images/video. In addition, causal models usually assume causal sufficiency, i.e. all the variables of interest are specified, which is hard to satisfy in many reinforcement learning settings. We also look at how this can be addressed while learning high-dimensional causal representations.

Primary Host:	Stefan Bauer (Helmholtz Center Munich & Technical University of Munich)
Exchange Host:	Bernhard Schölkopf (ELLIS Institute Tübingen & Max Planck Institute for Intelligent Systems)
PhD Duration:	01 October 2021 - 30 September 2025
Exchange Duration:	01 October 2021 - 31 March 2022 - Ongoing