In human-AI collaboration, learning a good model of the human is important for an autonomous learning system which aims to help its users in the most efficient way possible. Unfortunately, the data in human-AI interaction is scarce due to the online nature of the tasks. Additional difficulties arise from the bounded-rationality of human decision-making, which often deviates from what standard decision-theoretic methods consider optimal. The goal of my thesis is to develop methods which can learn more realistic user models from the interaction data, in order to help AI systems understand their users better. To this end, my work combines opponent modelling with machine learning in order to develop and learn theory-of-mind (ToM) based agent models of the human users from the interaction data, which take into account the inner-states of other agents such as their knowledge levels and perception abilities. The normative assumptions of these models have strong roots in cognitive science and behavioural economics. My main academic interests are the theory of multi-agent and reinforcement learning, and the applications of human-AI collaboration.