Lisa Petry

In recent years, deep-learning research has been marked by the rapid scaling of models to increasingly large sizes, particularly as seen in the Transformer architecture. While this growth has enabled significant breakthroughs in various applications, it has also introduced challenges related to computational cost, energy consumption, and the accessibility of necessary infrastructure. Transformers are now applied across fields from language to biology, yet their resource demands limit broader experimentation and deployment. This PhD aims to explore more efficient formulations of the Transformer by revisiting its underlying principles and drawing inspiration from state-space models and probabilistic machine learning. The central hypothesis is that these perspectives may lead to Transformer variants that strike a better balance between efficiency, interpretability, and performance.