Some pitfalls of measuring representational similarity using Representational Similarity Analysis

  • A core challenge in cognitive and brain sciences is to assess whether different biological systems represent the world in a similar manner. Representational Similarity Analysis (RSA) is an innovative approach that addresses this problem by looking for a second-order isomorphisim in neural activation patterns. This innovation makes it easy to compare latent representations across individuals, species and computational models, and accounts for its popularity across disciplines ranging from artificial intelligence to computational neuroscience. Despite these successes, using RSA has led to difficult-to-reconcile and contradictory findings, particularly when comparing primate visual representations with deep neural networks (DNNs): even though DNNs have been shown to learn and behave in vastly different ways to humans, comparisons based on RSA have shown striking similarities in some studies. Here, we demonstrate some pitfalls of using RSA and explain how contradictory findings can arise due to false inferences about representational similarity based on RSA-scores. In a series of studies that capture increasingly plausible training and testing scenarios, we compare neural representations in computational models, primate cortex and human cortex. These studies reveal two problematic phenomena that are ubiquitous in current research: a “mimic effect”, where confounds in stimuli can lead to high RSA-scores between provably dissimilar systems, and a “modulation effect”, where RSA-scores become dependent on stimuli used for testing. Since our results bear on a number of influential findings, such as comparisons made between human visual representations and those of primates and DNNs, we provide recommendations to avoid these pitfalls and sketch a way forward to a more solid science of representation in cognitive systems.

Download full text files

Export metadata

Additional Services

Share in Twitter Search Google Scholar
Author:Marin DujmovićORCiD, Jeffrey S. BowersORCiDGND, Federico AdolfiORCiD, Gaurav MalhotraORCiD
Parent Title (English):bioRxiv
Document Type:Preprint
Date of Publication (online):2022/11/18
Date of first Publication:2022/11/18
Publishing Institution:Universitätsbibliothek Johann Christian Senckenberg
Release Date:2023/09/01
Issue:2022.04.05.487135 Version 3
Edition:Version 3
Page Number:45
Dewey Decimal Classification:1 Philosophie und Psychologie / 15 Psychologie / 150 Psychologie
Licence (German):License LogoCreative Commons - CC BY-NC - Namensnennung - Nicht kommerziell 4.0 International