Depth cameras and LiDAR sensors are important tools for agents that need to perceive the world and interact with it, thus 3D data managing and learning algorithms are quickly becoming essential. Deep learning has achieved a significant leap for 2D computer vision, and for 3D applications is currently demonstrating great potential. However, the task of fully understanding the 3D real world still remains far-fetched. How to extract information from unstructured and unordered point clouds and how to avoid their extremely expensive manual annotation are just two examples of the difficult and new challenges that need attention. This PhD project investigates 3D scenarios with a particular focus on the introduction of open-domain and open-set methods for 3D vision applications. We propose to elaborate algorithms able to adapt and transfer knowledge from synthetic to real 3D data distributions while also supporting perception (recognition, reconstruction, novelty detection) and interaction (object manipulation) tasks.