TY  - INPR
A1  - Dujmović, Marin
A1  - Bowers, Jeffrey S.
A1  - Adolfi, Federico
A1  - Malhotra, Gaurav
T1  - Some pitfalls of measuring representational similarity using Representational Similarity Analysis
T2  - bioRxiv
N2  - A core challenge in cognitive and brain sciences is to assess whether different biological systems represent the world in a similar manner. Representational Similarity Analysis (RSA) is an innovative approach that addresses this problem by looking for a second-order isomorphisim in neural activation patterns. This innovation makes it easy to compare latent representations across individuals, species and computational models, and accounts for its popularity across disciplines ranging from artificial intelligence to computational neuroscience. Despite these successes, using RSA has led to difficult-to-reconcile and contradictory findings, particularly when comparing primate visual representations with deep neural networks (DNNs): even though DNNs have been shown to learn and behave in vastly different ways to humans, comparisons based on RSA have shown striking similarities in some studies. Here, we demonstrate some pitfalls of using RSA and explain how contradictory findings can arise due to false inferences about representational similarity based on RSA-scores. In a series of studies that capture increasingly plausible training and testing scenarios, we compare neural representations in computational models, primate cortex and human cortex. These studies reveal two problematic phenomena that are ubiquitous in current research: a “mimic effect”, where confounds in stimuli can lead to high RSA-scores between provably dissimilar systems, and a “modulation effect”, where RSA-scores become dependent on stimuli used for testing. Since our results bear on a number of influential findings, such as comparisons made between human visual representations and those of primates and DNNs, we provide recommendations to avoid these pitfalls and sketch a way forward to a more solid science of representation in cognitive systems.
Y1  - 2022
UR  - http://publikationen.ub.uni-frankfurt.de/frontdoor/index/index/docId/73062
UR  - https://nbn-resolving.org/urn:nbn:de:hebis:30:3-730624
UR  - https://www.biorxiv.org/content/10.1101/2022.04.05.487135v3
IS  - 2022.04.05.487135 Version 3
ER  -