Enhancing Soft Robotic Perception through Multi-Modal Sensing and Generative Models

This paper introduces a perception model that harmonizes data from diverse modalities, including touch, vision, and proprioception, to build a compact yet comprehensive state representation for soft robots. The model employs a generative model to efficiently compress the fused information and predict the next observation, enabling perceptually-aware soft robots to interact with unstructured environments.