Transfer learning is key to adapting foundation models to new domains; however, its theoretical foundation remains underdeveloped. In my Ph.D. project with Carlo and Massi, our goal is to better understand the conditions under which transfer learning effectively occurs. By doing so, we hope to elucidate key design principles that can be applied to create improved transfer learning algorithms for foundation models. A pivotal aspect of our investigation is the transfer of knowledge across different domains, such as from natural language to tabular data. We aim to determine under which circumstances this is both possible and advantageous.