This paper deals with the study of reconstructing visual information from brain activity. Studies on decoding images using generative models using fMRI have been conducted, but it has been difficult to accurately restore highly complex visual stimuli. This is due to the density and diversity of elements in the stimulus, the elaborate spatial structure, and the multifaceted semantic information. To solve this problem, this paper proposes the HAVIR model, which includes two adapters. The AutoKL adapter transforms fMRI voxels into a latent diffusion dictionary that captures the topological structure, and the CLIP adapter transforms voxels into CLIP text and image embeddings that contain semantic information. These complementary representations are fused by Versatile Diffusion to generate the final reconstructed image. To extract the most important semantic information in complex scenarios, the CLIP adapter is trained using text captions describing the visual stimulus and the semantic images synthesized with the captions. The experimental results show that HAVIR effectively reconstructs the structural features and semantic information of visual stimuli even in complex scenarios, and outperforms existing models.
