The main aim of this project is to build a holistic model for 3D scene reconstruction. Despite 3D scene reconstruction having been a popular sub-field in computer vision and computer graphics, the majority of previous work focuses only on geometric reconstruction. However, human-level understanding of a dynamic scene requires a holistic approach that considers not only the geometry but also the temporal and semantic properties. During my Ph.D., I will address these problems by developing models that can jointly understand and reconstruct dynamic scenes in an efficient and robust manner.