As machine learning touches upon all areas of our daily lives, it is increasingly deployed to make or support consequential decisions about individuals. Such applications raise concerns about privacy violations, the fairness of algorithms, as well as the long-term impact automated decisions might have on individuals and society as a whole. We address these concerns by building fair, privacy-preserving machine learning models and analyze their impact within the social context.