Deploying Machine Learning systems in safety-critical real-world scenarios such as the medical domain poses particular challenges. Often, classes are semantically very similar and the data available for training is noisy and suffers from high class-imbalance. Further, modern Machine Learning systems, despite being tremendously successful at a plethora of tasks, are known to make overly confident predictions. This is, they are typically unable to communicate low confidence when confronted with data that is different from the data seen during training. Designing safe systems, capable of flagging inputs they do not know how to properly process, is therefore crucial for safety-critical areas. The goal of this thesis is thus to develop and investigate methods to this end, involving areas like out-of-distribution detection, long-tailed classification, uncertainty estimation, robustness and generalization.