Sets help us organize our understanding of the real world: complex images can be simplified as sets of objects, arbitrary 3D shapes can be described by sets of coordinates, and graphs (a tuple of two sets) can model all relational structures between objects. My research focuses on inferring sets with deep learning approaches, for which I develop methods that scale efficiently and are generally useful for set prediction applications. Example applications include object localization and identification in videos, anomaly detection to identify outlier groups, and 3D point-cloud reconstruction.