Learning-based end-to-end autonomous driving approaches have achieved considerable progress in recent years and the mainstream follows the imitation learning pattern, where the agent is encouraged to clone the expert's behavior. The policy learning in existing methods merely replicates the expert’s behavior rather than selecting actions that contribute positively to driving tasks and enhance safety. Along with decreased interpretability, the dependency on pre-trained experts employing privileged information and the absence of capacity for online fine-tuning via exploration further constrain the potential performance improvements and practical applications of imitation learning methods.

To address these issues, this project aims to develop a novel reinforcement learning approach for interpretable autonomous driving.

(1) Novel Frameworks: We will formulate the environment dynamics model and predict future states conditioned by selected actions. Then we can explicitly evaluate the consequences of actions based on the predictions and choose the most suitable action.

(2) Multi-modal Fusion: The prediction can be substantially enhanced by incorporating multi-view inputs and LiDAR data. Suitable auxiliary tasks such as scene reconstruction and depth estimation could be introduced to acquire fused features containing rich semantic information.

(3) Domain Adaptation: The capability of online learning introduces the potential for domain adaptation. Our model can learn in unseen sceneries from online interaction and benefits from learned knowledge, such as training on sunny days and testing on rainy nights.