Deep learning shapes the world with given data. It works well with quality and abundant data while providing biased perspectives with scarce, poor data. This challenge has inspired me to study how to acquire language proficiency from limited data and interpret and control the model behaviors for robust and transparent usage. The interpretable and controllable language models would allow the learning procedure to de-bias the model quality with minimum human supervision and less computational cost. Within my Ph.D., we are interested in applying statistical concepts or linguistic approaches to address these issues for natural language processing.