核心概念
Psychometry is an omnifit model that can efficiently capture both the inter-subject commonalities and individual specificities in fMRI data to reconstruct high-quality and realistic images.
摘要
The article presents Psychometry, an omnifit model for reconstructing images from functional Magnetic Resonance Imaging (fMRI) data obtained from different subjects.
Key highlights:
- Existing fMRI-to-Image methods train separate models for each individual subject, ignoring commonalities between the data.
- Psychometry is equipped with an Omni Mixture-of-Experts (Omni MoE) module that allows all experts to collectively identify the inter-subject commonalities while each expert handles the individual differences through subject-specific parameters.
- Psychometry also employs a retrieval-enhanced inference strategy called Ecphory, which retrieves the most relevant CLIP image or text embedding from pre-stored training data to enhance the learned fMRI representation.
- The enhanced fMRI representations serve as reliable conditional signals to guide a pre-trained diffusion model in reconstructing high-quality and realistic images.
- Psychometry significantly reduces the model size, training time, and computational resources required compared to existing methods that train separate models for each subject.
- Experiments show that Psychometry outperforms state-of-the-art methods in both low-level and high-level evaluation metrics for image reconstruction from fMRI data.
統計資料
"Reconstructing the viewed images from human brain activity bridges human and computer vision through the Brain-Computer Interface."
"The inherent variability in brain functioning across individuals adds complexity to interpreting brain activity."
"State-of-the-art fMRI-to-Image methods suffer obvious performance degradation when utilizing data from all the subjects to train a unified model."
引述
"The inherent variability in brain functioning across individuals adds complexity to interpreting brain activity."
"Psychometry enjoys a few attractive qualities: First, it significantly reduces the model size, training time, and computational resources required. This is achieved by the creation of an omnifit model that can handle fMRI data of different subjects, eliminating the need for separately training tailored models on subject-specific data."