Neural networks are a flexible class of models that have soared in popularity due to their scalability to large amounts of data and computation. The fact that these models are able to represent a very broad range of functions makes performing Bayesian inference in them challenging. Most existing approaches focus on probabilistic reasoning over neural network weights. Unfortunately, these are often only able to characterise local regions of weight space, corresponding to small subsets of similar functions. My research focuses on performing probabilistic inference in neural networks while considering spaces different from weight space, such as architecture or function space.