Accurate State Estimation of Legged Robots using Hybrid Kalman Filtering and Transformer-based Vision

Core Concepts

A hybrid approach combining Kalman filtering, optimization, and learning-based methods is proposed to accurately estimate the state of a legged robot's trunk by integrating proprioceptive and exteroceptive information.

Abstract

The paper presents a state estimation framework called OptiState that combines a model-based Kalman filter with a learning-based Gated Recurrent Unit (GRU) network to estimate the state of a legged robot's trunk.
The Kalman filter uses joint encoder, IMU, and ground reaction force measurements to provide an initial state estimate. The GRU then refines this estimate by considering the Kalman filter output, latent space representation of depth images from a Vision Transformer (ViT) autoencoder, and other sensor data over a receding time horizon.
The key aspects of the approach are:

The Kalman filter leverages a single-rigid body model and reuses ground reaction forces from a Model Predictive Control (MPC) optimization to propagate the state.
The GRU learns to correct any nonlinearities or errors in the Kalman filter's estimate, while also providing an uncertainty measure for the prediction.
The ViT autoencoder extracts semantic information from depth images to aid the GRU's state estimation.
The proposed OptiState framework is evaluated on a quadruped robot traversing various terrains, including slippery, inclined, and rough surfaces. It demonstrates a 65% improvement in Root Mean Squared Error compared to a state-of-the-art Visual-Inertial Odometry (VIO) SLAM baseline.

Stats

The robot's trunk state includes orientation (roll, pitch, yaw), position (x, y, z), angular velocity (roll, pitch, yaw), and linear velocity (x, y, z).
The Kalman filter uses joint encoder positions and velocities, IMU orientation and angular velocity, and ground reaction forces from an MPC controller as inputs.
The GRU uses the Kalman filter's state estimate, latent space of depth images, odometry data, IMU accelerations, and ground reaction forces as inputs.

Quotes

"By integrating Kalman filtering, optimization, and learning-based modalities, we propose a hybrid solution that combines proprioception and exteroceptive information for estimating the state of the robot's trunk."
"The estimation is further refined through Gated Recurrent Units, which also considers semantic insights and robot height from a Vision Transformer autoencoder applied on depth images."
"This framework not only furnishes accurate robot state estimates, including uncertainty evaluations, but can minimize the nonlinear errors that arise from sensor measurements and model simplifications through learning."

Key Insights Distilled From

OptiState: State Estimation of Legged Robots using Gated Networks with Transformer-based Vision and Kalman Filtering

by Alexander Sc... at arxiv.org 04-30-2024

https://arxiv.org/pdf/2401.16719.pdf

Deeper Inquiries

How could the proposed OptiState framework be extended to handle more complex robot dynamics, such as multi-body systems or flexible appendages

The OptiState framework can be extended to handle more complex robot dynamics by incorporating advanced modeling techniques and sensor fusion strategies. For multi-body systems, the framework can be adapted to include additional state variables and dynamics equations to account for the interactions between multiple bodies. This would involve expanding the state estimation model to include the states of each body, their relative positions, and the forces acting between them. By integrating multi-body dynamics principles and constraints into the Kalman filter and GRU components, the framework can effectively estimate the states of complex robotic systems with interconnected bodies.
Moreover, for robots with flexible appendages, such as robotic arms or manipulators, the OptiState framework can be enhanced to include flexible body dynamics models. This would require incorporating flexible body dynamics equations, such as modal analysis or finite element methods, into the state estimation process. By integrating sensor measurements related to the flexibility and deformation of the appendages, such as strain sensors or tactile sensors, the framework can adapt to the varying dynamics of flexible structures. Additionally, machine learning algorithms can be trained on data from these sensors to improve the estimation of flexible body states and uncertainties.

What other types of exteroceptive sensors, beyond depth cameras, could be integrated into the learning-based component to further improve state estimation performance

To further enhance the state estimation performance of the OptiState framework, various exteroceptive sensors beyond depth cameras can be integrated into the learning-based component. These sensors can provide complementary information about the robot's environment and improve the robustness of the state estimation system. Some examples of exteroceptive sensors that can be integrated include:

Lidar Sensors: Lidar sensors can provide detailed 3D point cloud data of the robot's surroundings, enabling accurate mapping and localization in complex environments. By incorporating lidar data into the learning-based component, the framework can improve its perception capabilities and enhance state estimation accuracy.

RGB Cameras: RGB cameras can capture color images of the environment, which can be used for object detection, scene understanding, and visual odometry. By processing RGB images with convolutional neural networks (CNNs) or other computer vision algorithms, the framework can extract valuable features for state estimation and improve its performance in visually challenging scenarios.

Inertial Measurement Units (IMUs): IMUs can provide information about the robot's acceleration, angular velocity, and orientation. By fusing IMU data with other sensor modalities, such as joint encoders and vision data, the framework can enhance its estimation of dynamic states and improve its resilience to external disturbances.

By integrating a diverse set of exteroceptive sensors and leveraging their complementary information, the OptiState framework can achieve more robust and accurate state estimation for legged robots in various operating conditions.

How could the uncertainty estimates provided by the GRU be leveraged in a stochastic control or planning framework for legged robots

The uncertainty estimates provided by the GRU in the OptiState framework can be leveraged in a stochastic control or planning framework for legged robots to improve decision-making and adaptability in uncertain environments. Here are some ways in which the uncertainty estimates can be utilized:

Risk-Aware Planning: By incorporating the uncertainty estimates into the cost function of a stochastic planner, the robot can make decisions that minimize risk and maximize performance. The uncertainty information can guide the robot to choose actions that are more reliable and robust in the face of unpredictable factors.

Adaptive Control: The uncertainty estimates can be used to adjust the control policies of the robot in real-time based on the confidence level of the state estimates. For instance, when the uncertainty is high, the control strategy can be modified to prioritize safety and stability, while in low uncertainty scenarios, the robot can focus on optimizing performance.

Exploration-Exploitation Trade-off: The uncertainty estimates can help the robot balance exploration and exploitation in unknown environments. By considering the uncertainty of state predictions, the robot can decide whether to explore new areas to reduce uncertainty or exploit known information to achieve its objectives efficiently.

Fault Detection and Recovery: High uncertainty levels in state estimates can indicate potential faults or anomalies in the system. The robot can use this information to trigger fault detection mechanisms and initiate recovery strategies to maintain operational integrity.

Overall, leveraging the uncertainty estimates from the GRU in a stochastic control or planning framework enables legged robots to make informed decisions, adapt to changing conditions, and operate effectively in uncertain and dynamic environments.