Combining Probabilistic Inference and Deep Learning

James Urquhart Allingham (Ph.D. Student)

Deep learning is an extremely successful machine learning paradigm that has been the driving force behind the renewed excitement around machine learning over the last seven or so years since AlexNet won the ImageNet LSVRC competition. However, despite the successes of deep learning, it is not without its flaws. One of these flaws is that deep learning models are not good at knowing when they don’t know. That is, they are unable to robustly and reliably quantify the uncertainty in their predictions. Another flaw is that most deep learning models are discriminative rather than generative. As an illustration of the first flaw, consider the application of deep learning to self-driving cars, a safety-critical application for which deep learning is already being used in production (e.g. Tesla Autopilot). Self-driving car technology must be able to handle never before encountered traffic situations since it is impossible to collect training data involving all possible combinations of vehicles, pedestrians, weather conditions, traffic signs, and other objects/events. In these situations where the software cannot act safely and reliably, it makes sense for a backup scheme to take over. However, backup schemes can only be enabled if the software is aware that it is currently acting outside of its normal operating conditions. That is, the software must know when it does not know. As an illustration of the second flaw, consider the application of machine learning to skin cancer diagnosis. A discriminative model might struggle to make accurate predictions for photographs with low lighting, especially if there are few training images with similar lighting conditions. However, a generative model imbued with a causal understanding of the photography process will be able to better generalise to these settings. Furthermore, the ability of such a model to generate data is useful in its own right and can help us debug pathologies of our model. Probabilistic machine learning, in contrast to deep learning, does not suffer from these flaws. Uncertainty plays a fundamental part in the probabilistic framework, and there is an intimate link to generative modelling via Bayes’ rule. The probabilistic framework also offers other advantages such as principled model comparison, and natural handling of missing data, to name a few. Ideally, we would like to build models which leverage both the predictive power of deep learning and the advantages of the probabilistic framework. However, combining the probabilistic approach with deep learning provides many challenges and has become an active area of research. This project aims to solve some of these problems by building on the exciting recent work in this field and developing novel algorithms that combine deep learning with probabilistic inference. Thus far, the project has focused on probabilistic reasoning and uncertainty in neural networks – specifically, the development of novel algorithms for inference over neural network architectures and scalable yet expressive approximations for BNNs. Future work will also include deep generative models.

Primary Host:	José Miguel Hernández-Lobato (University of Cambridge)
Exchange Host:	Eric Nalisnick (University of Amsterdam)
PhD Duration:	01 October 2019 - 30 April 2023
Exchange Duration:	01 February 2022 - 31 July 2022 - Ongoing