Proxy-RLHF decouples generation and alignment processes in Large Language Models, achieving human value alignment with minimal computational cost.