Skip to content

Neural networks

Deep neural networks (DNNs) trained on images predict neural responses along the primate ventral visual stream better than earlier models (Yamins et al., 2014), and the Brain-Score benchmarks compare many of them. We show a network the same images as our participants, record what its layers do, and compare those activations with human data recorded for the same images. These can be fMRI, EEG or MEG, intracranial recordings, or behaviour such as similarity judgements and categorisation. This section gets you started with that workflow in Python and PyTorch, using a small set of pixel-art images that runs on any laptop.

Before you start

You need basic Python and a conda installation. If either is new to you, start with Coding practices. If neural networks are new to you, look at From an image to an RDM first.

All pages use the same small dataset, so you can run every code block as you read. It has 24 pixel-art sprites in three categories, each shown on three scenes in two positions, and simulated brain data from ten participants for two regions of interest.

Download the toy kit What is inside

What do you want to do?

Pick the path that matches your project. Every path starts with Set up and pick a model.

  • Use a pretrained model

    Most projects take this path. Download a public network (AlexNet, ResNet, CORnet, ...), show it your stimuli and record its layers, without training anything.

    Set up → Extract → Compare

  • Train or fine-tune a model

    When public models do not know your images or task (chess boards, pixel art, new categories), adapt one first.

    Set up → Fine-tune → Extract → Compare

  • Test a hypothesis about the layers

    If you have no human data yet, ask which layers follow the scene, the position or the category of the images, with model RDMs and decoding.

    Set up → Extract → Compare with a model

  • Compare a model with human data

    You already have layer activations and human data for the same images (brain patterns or behaviour), and want RSA, decoding or an encoding model. The page also shows how to read betas from an SPM GLM.

    Set up → Compare

Do I need a GPU?

No. Everything in this section runs on a laptop CPU. For real projects with large models or many images, use a GPU machine or the VSC cluster (see HPC). If the environment gives you trouble on a GPU machine, Set up shows how to keep training in a separate one.

I work in MATLAB. Can I still use this?

The network side needs Python, because the models, their weights and the tools to read their layers are all Python packages. Once you have saved the activations (as on Extract activations), you can also save them as a .mat file with scipy.io.savemat and run RSA or decoding in MATLAB with the tools you know, for example CoSMoMVPA.