Core Concepts
Controlgym is a library of 36 industrial control settings and 10 infinite-dimensional partial differential equation (PDE)-based control problems, designed to serve as a comprehensive benchmark for evaluating the performance and scalability of reinforcement learning (RL) algorithms in large-scale control systems.
Abstract
The paper introduces controlgym, a Python library that provides a diverse set of control environments for benchmarking reinforcement learning (RL) algorithms. The environments span a wide range of applications, including industrial control settings from sectors like aerospace, cyber-physical systems, ground and underwater vehicles, and power systems, as well as large-scale control problems governed by partial differential equations (PDEs) in fluid dynamics and physics.
The key highlights of controlgym are:
Linear control environments: The library includes 36 linear control environments from various industries, which can be used to validate theoretical developments in RL for linear optimal control, robust control, dynamic games, estimation, and filtering.
PDE control environments: The library provides 10 PDE-based control environments, where the state dimensionality can be extended to infinity while preserving the intrinsic dynamics. This feature is crucial for assessing the scalability of RL algorithms.
Customizable dynamics: For the linear PDE environments, the authors provide explicit state-space models, allowing users to tune the open-loop system dynamics by adjusting the physical parameters of the PDEs.
Gym-compliant: All environments in controlgym are integrated within the OpenAI Gym/Gymnasium (Gym) framework, enabling the direct application of standard RL algorithms like stable-baselines3.
The authors demonstrate the usage of controlgym by providing examples of applying model-based controllers and model-free RL algorithms, such as the linear-quadratic-Gaussian (LQG) controller and the proximal policy optimization (PPO) algorithm, to the control environments.