As the importance of NLP systems in various domains like healthcare and finance continues to grow, the reliability and trustworthiness of NLP systems is paramount. This project aims at developing robust language models by investigating and creating techniques resilient to data leakage and adversarial attacks. An essential aspect of building trustworthy systems, particularly in domains like healthcare, involves generating synthetic data that emulates the characteristics of private data while safeguarding sensitive information, thereby enabling the secure development of workflows without immediate privacy concerns. This project will also involve an investigation into the trade-offs between quality, effort and ethics for different privacy and federated learning techniques. Additionally, I intend to investigate the intersection of trustworthy NLP systems with casuality and fairness, seeking to understand how these principles can be integrated into the development of safe NLP models.