Skip to content

Neural networks

Deep neural networks (DNNs) trained on images predict neural responses along the primate ventral visual stream better than earlier models (Yamins et al., 2014; the Brain-Score benchmarks compare many of them). We show a network the same images as our participants, record what its layers do, and compare those activations with fMRI or EEG data. This section gets you started with that workflow in Python and PyTorch, using a small set of pixel-art images that runs on any laptop.

Top: the toy kit, 24 pixel-art sprites in three categories, each in four versions. Bottom: the workflow, in which the same images go to a network and to a participant, and layer activations are compared with brain patterns by RSA, decoding and encoding

Before you start

You need basic Python and a conda installation. If either is new to you, start with Coding practices.

All pages use the same small dataset, so you can run every code block as you read. It has 24 pixel-art sprites in three categories, each category with its own scene and each sprite in four variants, and synthetic "brain activations" for two regions of interest.

Download the toy kit What is inside

What do you want to do?

Pick the path that matches your project. Every path starts with Set up and pick a model.

  • Use a pretrained model

    The path most projects take: download a public network (AlexNet, ResNet, CORnet, ...), show it your stimuli and record its layers, without training anything.

    Set up → Extract → Compare

  • Train or fine-tune a model

    When public models do not know your images or task (chess boards, pixel art, new categories), adapt one first.

    Set up → Fine-tune → Extract → Compare

  • Test a hypothesis about the layers

    No brain data yet: ask which layers follow a visual property and which follow category, with model RDMs and decoding.

    Set up → Extract → Compare with a model

  • Compare a model with brain data

    You already have layer activations and brain patterns for the same images, and want RSA, decoding or an encoding model. The page also shows how to read betas from an SPM GLM.

    Set up → Compare

Do I need a GPU?

No. Everything in this section runs on a laptop CPU. For real projects with large models or many images, use a GPU machine or the VSC cluster (see HPC). If the environment gives you trouble on a GPU machine, Set up shows how to keep training in a separate one.

I work in MATLAB. Can I still use this?

The network side needs Python: the models, their weights and the tools to read their layers are all Python packages. Once you have saved the activations (as on Extract activations), you can also save them as a .mat file with scipy.io.savemat and run RSA or decoding in MATLAB with the tools you know, for example CoSMoMVPA.