Video streams consist of multiple modalities, e.g., RGB frames, optical flow, audio. Their natural correspondence provides rich semantic information to achieve effective multi-modal perception and learning. In this project, our aim is to understand the video content by analyzing multiple modalities and decide which modality to trust in different scenarios, especially under harsh vision conditions. To this end, various cross-modal interaction modules will be developed to fit specific tasks, such as action recognition, repetition counting, activity localization.