Many NLP tasks require incorporating not only textual data but also other modalities, such as images, speech, or software code. In this project, we aim to build systems that can jointly reason over these different modalities with diverse applications, for example in dialog systems. Furthermore, the information that grounds such systems changes over time. For example, facts become outdated, new information emerges, and users provide feedback about the model and their preferences. Hence, we plan to utilize continual learning methods that allow for integration this information into the model while retaining already learned concepts that have remained valid. We hope that this project can contribute to meaningful real-world NLP applications like education and human assistance.