Hierarchical Preference Optimization for Hierarchical Reinforcement Learning: Mitigating Non-Stationarity and Infeasible Subgoal Generation

Core Concepts

Hierarchical Preference Optimization (HPO) leverages preference-based learning and a novel bi-level optimization framework to address non-stationarity and infeasible subgoal generation in hierarchical reinforcement learning, leading to improved performance in complex robotic tasks.

Abstract

Bibliographic Information:

Singh, U., Chakraborty, S., Suttle, W. A., Sadler, B. M., Sahu, A. K., Shah, M., ... & Bedi, A. S. (2024). HIERARCHICAL PREFERENCE OPTIMIZATION: LEARNING TO ACHIEVE GOALS VIA FEASIBLE SUB-GOALS PREDICTION. arXiv preprint arXiv:2411.00361.

Research Objective:

This paper introduces Hierarchical Preference Optimization (HPO), a novel approach to mitigate the challenges of non-stationarity and infeasible subgoal generation in hierarchical reinforcement learning (HRL) for complex robotic control tasks.

Methodology:

HPO leverages a bi-level optimization framework to address the nested structure of HRL, ensuring the higher-level policy generates feasible subgoals for the lower-level policy. It utilizes a token-level Direct Preference Optimization (DPO) objective, eliminating the need for pre-trained reference policies. The lower-level policy is trained using traditional reinforcement learning, while the higher-level policy learns from preferences, mitigating non-stationarity. The authors employ a primitive-in-the-loop approach to autonomously generate preferences using sparse environment rewards, reducing the reliance on human feedback. The effectiveness of HPO is evaluated on four challenging robotic tasks: maze navigation, pick and place, push, and a Franka kitchen environment.

Key Findings:

• HPO significantly outperforms several hierarchical and non-hierarchical baselines in terms of success rate across all four robotic tasks.
• Ablation studies demonstrate the importance of both non-stationarity mitigation and feasible subgoal generation for HPO's superior performance.
• HPO effectively mitigates non-stationarity by generating achievable subgoals that induce optimal lower-level goal-reaching behavior.
• The primitive regularization in HPO ensures the generation of feasible subgoals, leading to more stable and efficient learning.

Main Conclusions:

HPO presents a promising solution for addressing key limitations in HRL, demonstrating significant performance improvements in complex robotic control tasks. The proposed bi-level optimization framework and preference-based learning approach effectively tackle non-stationarity and infeasible subgoal generation, paving the way for more efficient and robust HRL algorithms.

Significance:

This research significantly contributes to the field of hierarchical reinforcement learning by addressing critical limitations that hinder its practical application in complex robotic tasks. The proposed HPO method offers a novel and effective solution for improving the stability and efficiency of HRL, potentially enabling the development of more sophisticated and autonomous robotic systems.

Limitations and Future Research:

While HPO demonstrates promising results, future research could explore its application to a wider range of robotic tasks and environments with varying complexity. Additionally, investigating the impact of different preference elicitation methods and exploring alternative reward shaping techniques could further enhance HPO's performance and generalizability.

Stats

HPO shows an improvement of up to 35% over the baselines in most of the task environments.

Quotes

"In this work, we propose Hierarchical Preference Optimization (HPO), a hierarchical approach that optimizes the higher level policy using token-level DPO objective (Rafailov et al., 2024a), and the lower level primitive policy using RL."
"Since HPO learns higher level policy from preference data, it avoids dependence on changing lower level primitive and thus mitigates non-stationarity (C1)."

Key Insights Distilled From

Hierarchical Preference Optimization: Learning to achieve goals via feasible subgoals prediction

by Utsav Singh,... at arxiv.org 11-04-2024

https://arxiv.org/pdf/2411.00361.pdf