Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
We fine-tuned visual brain encoding models using individual perceptual similarity judgments to improve representational alignment with human perception, moving beyond prediction of fMRI responses. By adapting a generic similarity model to individual participants and using these personalized judgments to guide encoder fine-tuning, we achieved superior alignment with human similarity structures compared to baseline models or those fine-tuned with population-level judgments. This work demonstrates that personalized perceptual similarity is crucial for creating brain encoding models that truly reflect human visual experience.
Personalized perceptual similarity, not just fMRI data, unlocks visual encoding models that accurately mirror human judgments and their intersubject variability.
Encoding models that predict brain activation responses to images have traditionally been trained to maximize accuracy when predicting observed brain data. However, their internal representations may not reflect individual human perception. In this work, we investigate whether we can fine-tune brain encoding models to better align their representations with individuals'similarity judgments. Using similarity judgments of 100 images from 8 participants in the Natural Scenes Dataset (NSD), we adapt a generic human similarity judgment model (DreamSim) separately for each participant to obtain personalized perceptual similarity models. These models are then used to obtain subject-specific similarity structure across images in their encoding model training sets. We fine-tune each participant's baseline encoder in two conditions, aligning its representations with either generic or personalized similarity judgments, with the same initialization and objective. We find that baseline brain encoders reflect within-subject perceptual similarity more strongly than measured fMRI responses and preserve intersubject correlations present in measured fMRI, but fail to reflect those of human perceptual similarity judgments. Fine-tuning with generic, population-level perceptual similarity information results in better alignment with human judgments than baseline encoders, but these models still fail to capture intersubject correlations of human judgments. However, fine-tuning with personalized perceptual similarity information results in models that are the most aligned with human perceptual similarity judgments and also reflect intersubject correlations of human judgments. Our framework provides an approach for bootstrapping human similarity judgment data into more individualized, human-aligned brain encoding models of vision.