Core Concepts
An unsupervised algorithm is proposed to discretize a continuous robot motion space and generate "action prototypes", each producing different effects in the environment. The algorithm automatically builds a representation of the effects and groups motions into action prototypes, where motions more likely to produce an effect are represented more than those that lead to negligible changes.
Abstract
The paper presents an unsupervised algorithm to learn discrete action prototypes for robots based on the effects they produce in the environment. The approach consists of three main stages:
Motion sampling: The robot samples random motions from its continuous motion space and stores the resulting (motion, effect) tuples.
Effect region clustering: The collected effects are clustered using k-means to find distinct effect classes. The number of clusters is determined based on the silhouette score.
Action prototype generation: For each effect class, the algorithm generates a small number of representative action prototypes using the Robust Growing Neural Gas (RGNG) algorithm. The number of prototypes per class is determined based on the variability of the effects.
The algorithm is evaluated on a simulated stair-climbing reinforcement learning task called "Up The Stairs". Compared to uniform and random discretization methods, the effect-driven discretization approach shows faster convergence and higher maximum reward when used with a Deep Q-Network agent. The performance is still lower than a Soft Actor-Critic agent operating in the continuous action space, but the effect-based discretization achieves this with an 85 times smaller network size.
The paper discusses the advantages of the effect-centric approach, such as the ability to discover meaningful action prototypes in an unsupervised manner. It also highlights the challenges in dealing with continuous effect spaces and the need for feature selection to maintain a good balance between performance and generalization.
Stats
The robot's state is represented by s = {x, y, z, qx, qy, qz, qw, dobs}, where x, y, z are the position, qx, qy, qz, qw is the orientation quaternion, and dobs is the distance to the next obstacle on the x-axis.
The robot's motion is represented by m = {α, μ}, where α = (0, α, 0) is the direction vector of the force applied at the robot's center of mass, and μ is its amplitude.
The reward function for the "Up The Stairs" environment is:
rt = (
1 if (sz
t+1 - sz
t) > 0
-(sz
t+1 - sz
t)/0.3 if (sz
t+1 - sz
t) <= 0.
)
Quotes
"Learning actions that are relevant to decision-making and can be executed effectively is a key problem in autonomous robotics."
"Current state-of-the-art action representations in robotics lack proper effect-driven learning of the robot's actions."
"We propose an unsupervised algorithm to discretize a continuous motion space and generate 'action prototypes', each producing different effects in the environment."