The human learning experience is multimodal. We utilize information from both language and vision, along with other contextual clues, to understand, reason, and communicate in social settings. In this project, we aim to utilize textual and visual information to solve different NLP tasks, such as classification, generation, as well as cross-modality tasks such as understanding of memes, multimodal and multilingual question answering. We hope to create systems that are easily transferable to many different applications and gain a better understanding of human communication.