Novel View Synthesis for learning-based Perception, Planning and Control

Christina Ourania Tze (Ph.D. Student)

This PhD project will leverage recent advances in novel view synthesis, in particular approaches related to neural radiance fields to investigate the feasibility of automatically constructing simulation environments and using them to train deep learning models for various downstream tasks, ranging from low-level vision tasks like stereo and optical flow, over mid level-vision tasks like semantic segmentation and localization to high-level sensori-motor tasks like closed-loop navigation. In particular, the project will focus on outdoor scenes and tasks related to self-driving such as safe navigation between two points on a road network as well as reducing the number of infractions incurred. The PhD project will leverage the KITTI-360 dataset and the CARLA simulator to train imitation policies in the reconstructed worlds and deploy the trained models. The PhD project will develop new novel view synthesis approaches that incorporate constraints (eg, via regularization) such that not the visual fidelity of the rendered images (which is the usual interest in novel view synthesis and measured in PSNR) but the downstream task performance (eg, the driving score or semantic segmentation performance) is improved. It will also investigate questions on how common artifacts (like floaters in the 3D density field) effect robustness and performance of the trained models and systematically investigate the hyperparameters of the resulting model, including data augmentation and camera viewpoint selection strategies.

Primary Advisor:	Andreas Geiger (University of Tübingen & Max Planck Institute for Intelligent Systems)
Industry Advisor:	Dzmitry Tsishkou (Huawei Technologies)
PhD Duration:	15 September 2023 - 14 September 2026