Shengbo Gao

Embodied agents such as robots, assistive systems and simulated characters need world models they can act in rather than only look at. Current 3D and 4D generative models produce convincing images and videos, but the scenes behind them are often not usable for interaction. Geometry, motion and the physical relations between objects are not guaranteed to hold once an agent starts moving things around.

This project will study how to build and learn 3D and 4D world models that support interaction, and how to measure whether they actually do. The main questions are the following. First, how to represent scenes and objects so that their structure and physical behaviour are explicit and can be checked instead of only rendered. Second, how to ground such models in real observations, including images, video, measurements and existing 3D data, while keeping track of what is known and what remains uncertain. Third, how to use these worlds to train and evaluate embodied agents in simulation, and how the experience of the agent can in turn improve the world model. The approach is general. Application areas of interest include everyday object manipulation, assistive and healthcare scenarios, and human centred environments.