Deep learning has quickly become the state-of-the-art approach for extracting knowledge from visual data and it is rapidly solving some of the most complex problems in Computer Vision, such as image classification, object detection, and visual-semantic understanding with supervised learning. As Deep Learning gets better at visual and semantic tasks, and new self-attentive operators and architectures emerge to tackle visual understanding and generative problems, the need for trustworthy algorithms increases. The purpose of my PhD is the design and analysis of novel and data-intensive algorithms for visual data understanding, visual generation, and for the integration of vision, semantics, and language - with a focus on the design of novel operators and on the trustworthiness of the results. It will also ensure that developed techniques are in line with the emerging European legal framework on AI.