It is possible to align large language models with human preferences and feedback data while satisfying strong mathematical guarantees of differential privacy.