Unified Multimodal Decoding of Brain Signals for Improved Understanding of Visual Concepts and Spatial Relationships

UMBRAE, a unified multimodal decoding method, aligns brain signals with image features to recover both semantic and spatial information, enabling a range of downstream tasks such as brain captioning, grounding, and retrieval.