Partial differential equations (PDEs) are used in physics, engineering and many other scientific disciplines. While their numerical solutions have been a longstanding challenge, deep learning methods offer an appealing meshfree approach. On the other hand, PDEs are a language in which we can express inductive biases very well. Therefore, the combination of deep learning methods and PDEs has a huge potential in both directions: deep learning methods can be used to improve PDE solvers with data, and PDEs can be used to form a backbone prior for deep learning methods enabling theoretically grounded frameworks to neural network architecture design. The latter might even be generalized to get classes of networks that depend only on intrinsic geometries. In this research program, we want to elaborate on both: physics inspired neural network solutions to differential equations as well as a theoretically grounded generalization of neural network design for geometric deep learning and inverse models where only the forwards models are known.