Perception—and, in a more direct sense, dynamic scene understanding—is fundamental to the success of autonomous robots. Adequately capturing and representing environmental complexity and dynamics in a reliable and robust manner is critical to subsequent tasks such as (local) navigation and planning, especially in environments where other autonomous agents and humans are involved. While this field spans a multitude of tasks, many of which share common aspects, they are generally solved independently. This poses the question whether there are benefits to be gained by more closely coupling related tasks and solving them jointly. In particular, we believe that sharing related information between tasks and extending it with complementary information formed through related tasks could lead to improved robustness and performance on all tasks involved. In this PhD project, we therefore explore how such holistic ways for dynamic scene understanding could be formulated and, further, work towards holistic scene representations, incorporating dynamics and motion.