The cycling of carbon between the land, ocean and atmosphere is of major importance in climate change studies. The spatio-temporal distribution of land-atmosphere fluxes remains uncertain. Accurate measurements are only available at certain stations with eddy-covariance towers. Additional data such as meteorological observations, satellite images and atmospheric concentrations of carbon dioxide may reduce uncertainties. The integration of these data streams into a single multi-modal statistical model is challenging. In this thesis, the integration of multi-modal data and process understanding into a deep learning model is studied. Such models are studied theoretically and applied to improving ecosystem flux estimates.