Unveiling functions of the visual cortex using task-specific deep neural networks

  • Abstract: The human visual cortex enables visual perception through a cascade of hierarchical computations in cortical regions with distinct functionalities. Here, we introduce an AI-driven approach to discover the functional mapping of the visual cortex. We related human brain responses to scene images measured with functional MRI (fMRI) systematically to a diverse set of deep neural networks (DNNs) optimized to perform different scene perception tasks. We found a structured mapping between DNN tasks and brain regions along the ventral and dorsal visual streams. Low-level visual tasks mapped onto early brain regions, 3-dimensional scene perception tasks mapped onto the dorsal stream, and semantic tasks mapped onto the ventral stream. This mapping was of high fidelity, with more than 60% of the explainable variance in nine key regions being explained. Together, our results provide a novel functional mapping of the human visual cortex and demonstrate the power of the computational approach. Author Summary: Human visual perception is a complex cognitive feat known to be mediated by distinct cortical regions of the brain. However, the exact function of these regions remains unknown, and thus it remains unclear how those regions together orchestrate visual perception. Here, we apply an AI-driven brain mapping approach to reveal visual brain function. This approach integrates multiple artificial deep neural networks trained on a diverse set of functions with functional recordings of the whole human brain. Our results reveal a systematic tiling of visual cortex by mapping regions to particular functions of the deep networks. Together this constitutes a comprehensive account of the functions of the distinct cortical regions of the brain that mediate human visual perception.
Author:Kshitij DwivediORCiDGND, Michael F. BonnerORCiD, Radoslaw Martin CichyORCiDGND, Gemma Roig NogueraORCiDGND
Parent Title (English):PLoS Computational Biology
Publisher:Public Library of Science
Place of publication:San Francisco, Calif.
Document Type:Article
Date of Publication (online):2021/08/13
Date of first Publication:2021/08/13
Publishing Institution:Universit├Ątsbibliothek Johann Christian Senckenberg
Release Date:2022/03/07
Tag:Functional magnetic resonance imaging; Linear regression analysis; Neural networks; Permutation; Semantics; Sensory perception; Vision; Visual cortex
Issue:8, art. e1009267
Page Number:22
First Page:1
Last Page:22
G.R. thanks the support of the Alfons and Gertrud Kassel Foundation. R.M.C. is supported by DFG grants (CI241/1-1, CI241/3-1) and the ERC Starting Grant (ERC-2018- StG 803370). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Institutes:Informatik und Mathematik
Dewey Decimal Classification:0 Informatik, Informationswissenschaft, allgemeine Werke / 00 Informatik, Wissen, Systeme / 004 Datenverarbeitung; Informatik
6 Technik, Medizin, angewandte Wissenschaften / 61 Medizin und Gesundheit / 610 Medizin und Gesundheit
Open-Access-Publikationsfonds:Informatik und Mathematik
Licence (German):License LogoCreative Commons - Namensnennung 4.0