In my research, I focus on enforcing desirable properties to the solution of learning algorithms, such as incorporating human beliefs, natural constraints, and causal structures. This translates to faster, more accurate, and more flexible models, which directly relates to real-world impact. I tackle this challenge on three sides: (i) I work on efficient constrained optimization algorithms that guarantee structural properties, provably converge and scale. (ii) I apply our novel theoretical insights to the problem of approximate inference with the goal of making it structured, efficient and accurate. (iii) I work towards learning algorithms that can spontaneously discover the natural structure present in the data, infer causal relations, and are useful for arbitrary downstream tasks.