Deep Natural Image Reconstruction from Human Brain Activity Based on Conditional Progressively Growing Generative Advers
- PDF / 5,790,028 Bytes
- 11 Pages / 595.276 x 790.866 pts Page_size
- 49 Downloads / 206 Views
www.neurosci.cn www.springer.com/12264
ORIGINAL ARTICLE
Deep Natural Image Reconstruction from Human Brain Activity Based on Conditional Progressively Growing Generative Adversarial Networks Wei Huang1 • Hongmei Yan1 • Chong Wang1 • Xiaoqing Yang1 • Jiyi Li1 Zhentao Zuo2 • Jiang Zhang3 • Huafu Chen1
•
Received: 27 February 2020 / Accepted: 16 June 2020 Ó Shanghai Institutes for Biological Sciences, CAS 2020
Abstract Brain decoding based on functional magnetic resonance imaging has recently enabled the identification of visual perception and mental states. However, due to the limitations of sample size and the lack of an effective reconstruction model, accurate reconstruction of natural images is still a major challenge. The current, rapid development of deep learning models provides the possibility of overcoming these obstacles. Here, we propose a deep learning-based framework that includes a latent feature extractor, a latent feature decoder, and a natural image generator, to achieve the accurate reconstruction of natural images from brain activity. The latent feature extractor is used to extract the latent features of natural images. The latent feature decoder predicts the latent features of natural images based on the response signals from the higher visual cortex. The natural image generator Electronic supplementary material The online version of this article (https://doi.org/10.1007/s12264-020-00613-4) contains supplementary material, which is available to authorized users. & Hongmei Yan [email protected] & Zhentao Zuo [email protected] & Huafu Chen [email protected] 1
The MOE Key Lab for Neuroinformation, University of Electronic Science and Technology of China, Chengdu 610054, China
2
State Key Laboratory of Brain and Cognitive Science, Beijing MR Center for Brain Research, Institute of Biophysics, Chinese Academy of Sciences, Beijing 100101, China
3
Department of Medical Information Engineering, Sichuan University, Chengdu 610065, China
is applied to generate reconstructed images from the predicted latent features of natural images and the response signals from the visual cortex. Quantitative and qualitative evaluations were conducted with test images. The results showed that the reconstructed image achieved comparable, accurate reproduction of the presented image in both highlevel semantic category information and low-level pixel information. The framework we propose shows promise for decoding the brain activity. Keywords Brain decoding fMRI Deep learning
Introduction Brain-reading based on brain activity has made notable achievements in the past decade. Functional magnetic imaging (fMRI) studies have shown that visual features such as orientation, spatial frequency [1], motion direction [2, 3], object category [4–11], perceptual imagination [12], dreams [13], and even memory [14] can be decoded from fMRI activity patterns by classificationbased machine-learning methods, which learn the linear or nonlinear mapping between a brain activity pattern and a stimulus category from a training datas
Data Loading...
