This dissertation project aims at providing a robust, scalable inference technique for parametric models of time series. In particular, it focuses on Gaussian process based collocation methods, investigating the weaknesses of existing ideas and developing new algorithms for parameter inference in systems of ODEs and SDEs. This goal requires to adapt and extend current machine learning methods like Gaussian processes with derivative information as well as developing new theoretical models for their application. At the moment four first author publications are either under review or already published, including work on the theoretical foundation (FGPGM, published at AISTATS 2019; ODIN, published at AAAI 2020), extensions to SDEs (Ares and Mars, published at ICML 2019) and scaling (SLEIPNIR, currently under review at ICML).