Core Concepts
This paper introduces a novel policy gradient algorithm for infinite horizon average reward constrained Markov Decision Processes (CMDPs) with general parameterization, achieving sublinear regret and constraint violation bounds.
Abstract
This paper explores the realm of infinite horizon average reward Constrained Markov Decision Processes (CMDP) with a focus on regret and constraint violation analysis. The proposed primal-dual based policy gradient algorithm manages constraints effectively while ensuring low regret towards achieving an optimal policy. The study bridges the gap in regret and constraint violation analysis for average reward CMDPs with general parametrization, providing significant advancements in the field of reinforcement learning.
The framework of Reinforcement Learning (RL) is discussed, emphasizing the significance of infinite horizon average reward setup for real-world applications. The paper addresses challenges in solving CMDP problems using model-based and model-free approaches, highlighting the importance of general parameterization to accommodate large state spaces efficiently. The proposed algorithm achieves sub-linear regret and constraint violation bounds, improving upon existing state-of-the-art results.
Key algorithms and their performance metrics are summarized in a table, showcasing the advancements made by the proposed primal-dual policy gradient algorithm. The convergence analysis, global convergence results, and detailed formulation provide insights into the effectiveness of the algorithm in handling complex reinforcement learning problems.
Quotes
"We propose a PG-based algorithm with general parameterized policies for the average reward CMDP."
"Our work improves the state-of-the-art regret guarantee in model-free tabular setup."