Neural networks are powerful models with big impacts on the economy, politics and society. Yet, we lack a general theory of how these networks learn structure in their representation space from input data. A scientific understanding of learning in neural networks is a foundation for interpretability and safety. Given the high energy consumption of state-of-the-art models, it also provides entry points for making training more energy-efficient. This PhD asks (i) how data, architecture and optimization algorithms together form structured representations, and (ii) whether these representations align with efficient principles from biological learning. I hypothesize that, rather than following a single universal law, learning is governed by a small set of dynamical laws acting on measurable quantities, each valid in a characterizable regime (e.g., lazy vs. rich learning, edge of stability) (Simon et al., 2026). To test this hypothesis, I focus on the geometry of representations and of the loss landscape, which I can track through training with analytical (exact learning dynamics, mean-field theory) and descriptive (effective rank, weight spectra) tools. Given their relevance, I analyze nonlinear and transformer-based networks, as well as linear networks because of their simplicity and solvability. The goal is a better theoretical understanding of the speed and order of learning, the contribution of individual layers, and how initialization and parameterization (NTK vs. μP) determine the training regime. For (ii), I build on efficient principles evident in the brain, such as local competition, sparse coding, functional modularity. By comparing solutions and emergent structures in training with those found in biological networks, I aim to identify limitations of our current training approaches, and to inspire optimization algorithm design.