Continual Multimodal Leaming

Nico Daheim (Ph.D. Student)

Many NLP tasks require incorporating not only textual data but also other modalities, such as images, speech, or software code. In this project, we aim to build systems that can jointly reason over these different modalities with diverse applications, for example in dialog systems. Furthermore, the information that grounds such systems changes over time. For example, facts become outdated, new information emerges, and users provide feedback about the model and their preferences. Hence, we plan to utilize continual learning methods that allow for integration this information into the model while retaining already learned concepts that have remained valid. We hope that this project can contribute to meaningful real-world NLP applications like education and human assistance.

Primary Host:	Iryna Gurevych (Technical University of Darmstadt)
Exchange Host:	()
PhD Duration:	01 September 2022 - 31 August 2026
Exchange Duration:	01 September 2024 - 31 August 2025 - Ongoing