Holistic 3D scene understanding plays a critical role in self-driving cars. It involves several sub-tasks, such as geometric layout estimation, object detection, recognition, and tracking. While each sub-task may be solved independently, it would be beneficial to utilize the complementary nature of different sub-task and model the 3D scene in a holistic way. Therefore, this project will focus on research problems to achieve this goal, such as representing the 3D scene compactly and finding a unified representation that can model scene geometry, sematic, and dynamics, and effectively and efficiently infer such 3D scene representation sensor inputs.