Perception-to-Image: Reconstructing Natural Images from the Brain Activity of Visual Perception
- PDF / 7,639,214 Bytes
- 10 Pages / 593.972 x 792 pts Page_size
- 92 Downloads / 200 Views
Annals of Biomedical Engineering (Ó 2020) https://doi.org/10.1007/s10439-020-02502-3
Original Article
Perception-to-Image: Reconstructing Natural Images from the Brain Activity of Visual Perception WEI HUANG,1 HONGMEI YAN,1 CHONG WANG,1 JIYI LI,1 ZHENTAO ZUO,2,3 JIANG ZHANG,4 ZHAN SHEN,1 and HUAFU CHEN1 1
MOE Key Lab for Neuroinformation, University of Electronic Science and Technology of China, Chengdu 610054, People’s Republic of China; 2State Key Laboratory of Brain and Cognitive Science, Beijing MR Center for Brain Research, Institute of Biophysics, Chinese Academy of Sciences, Beijing 100101, China; 3University of Chinese Academy of Sciences, Beijing 100049, China; and 4Department of Medical Information Engineering, Sichuan University, Chengdu 610065, China (Received 28 October 2019; accepted 30 March 2020) Associate Editor Xiaoxiang Zheng oversaw the review of this article.
Abstract—The reappearance of human visual perception is a challenging topic in the field of brain decoding. Due to the complexity of visual stimuli and the constraints of fMRI data collection, the present decoding methods can only reconstruct the basic outline or provide similar figures/features of the perceived natural stimuli. To achieve a high-quality and highresolution reconstruction of natural images from brain activity, this paper presents an end-to-end perception reconstruction model called the similarity-conditions generative adversarial network (SC-GAN), where visually perceptible images are reconstructed based on human visual cortex responses. The SCGAN extracts the high-level semantic features of natural images and corresponding visual cortical responses and then introduces the semantic features as conditions of generative adversarial networks (GANs) to realize the perceptual reconstruction of visual images. The experimental results show that the semantic features extracted from SC-GAN play a key role in the reconstruction of natural images. The similarity between the presented and reconstructed images obtained by the SC-GAN is significantly higher than that obtained by a condition generative adversarial network (C-GAN). The model we proposed offers a potential perspective for decoding the brain activity of complex natural stimuli. Keywords—Visual decoding, Reconstruction, SC-GAN, Deep learning.
Address correspondence to Hongmei Yan and Huafu Chen, MOE Key Lab for Neuroinformation, University of Electronic Science and Technology of China, Chengdu 610054, People’s Republic of China. Electronic mails: [email protected], chenhf@uestc. edu.cn. Zhentao Zuo, State Key Laboratory of Brain and Cognitive Science, Beijing MR Center for Brain Research, Institute of Biophysics, Chinese Academy of Sciences, Beijing 100101, China. Electronic mail: [email protected]
INTRODUCTION Visualizing the perception of the human brain is a challenging goal in neuroscience, and brain decoding methods using machine learning based on fMRI activities have made the visualization of perceptual content possible.17 The reconstruction of natural image pe
Data Loading...
