Foundation models have been gaining momentum in robotics research recently. However, grounding the behavior for better predictability especially on physical robot has been a major challenge. In this proposal, research directions are proposed towards data scarcity, domain gap, physic grounding, and task and motion planning. The objective is to incorporate multi-modal sensory data for planning and manipulation; utilizing the foundation model for fast adaptation in new tasks and environments.