In recent years, representation learning has made significant strides, enabling self-supervised learning methods to achieve comparable performance to supervised approaches. However, these models typically require extensive training datasets and substantial computational resources to achieve their optimal performance. Since not everyone has access to such abundant resources, there is a pressing need to develop more efficient methods. During my Ph.D., my focus is on enhancing the efficiency of self-supervised learning methods by either devising novel approaches to extract more information from limited data or incorporating new training modalities and designing improved architectures.