This project revolves around bridging the gap between the proficiency of ML algorithms in lab settings and their performance in real-world settings. Estimating the uncertainty of the prediction of a model is a central aspect of this, enabling models that are more interpretable and trustworthy. Furthermore, even with good estimates of this quantity, it remains an open question how human users can benefit most from such information. Good uncertainty estimates are also of great interest in autonomous decision-making, especially in Reinforcement Learning settings where outcomes of decisions are observed with delay. Uncertainty estimates can serve various purposes in such settings, ranging from exploration strategies of exploring situations with high uncertainty in outcomes to risk assessment of actions in high stakes applications.