Set up and pick a model¶

On this page

You need: conda (Miniforge or Anaconda) and about 7 GB of free disk space. A GPU helps but is not required for the toy examples.
You get: a working Python environment, the pixel-art toy kit, and a pretrained network that runs on it.
Next: train or fine-tune if the model needs to learn your images, or go straight to extract activations.

1. Create the environment¶

We work in Python with PyTorch, plus three packages built for this kind of analysis: thingsvision to extract activations, rsatoolbox for RSA and himalaya for encoding models. Keep one conda environment per project. Save this file as environment.yml in your project folder:

environment.yml

name: dnn
channels:
  - conda-forge
dependencies:
  - python=3.11
  - pip
  - pip:
      - thingsvision  # (1)!
      - git+https://github.com/openai/CLIP.git  # (2)!
      - rsatoolbox[imaging]  # (3)!
      - himalaya

thingsvision brings PyTorch, torchvision, scikit-learn, pandas and matplotlib with it. It pins PyTorch 2.0.1 and also installs TensorFlow, so the environment takes about 7 GB.
Needed by thingsvision for all its source="custom" models, such as CORnet and CLIP, as its installation instructions say.
The imaging extra adds the readers for SPM and NIfTI files, used to load your own betas from an SPM GLM.

Then create and activate it:

conda env create -f environment.yml
conda activate dnn

The first install downloads everything and takes a while.

Laptop (CPU)Machine with an NVIDIA GPUVSC cluster

Nothing else to do. Everything in this section runs on a CPU.

PyTorch uses the GPU when it finds one, and the check below tells you whether it does. If the environment gives you trouble on your machine, for example when PyTorch does not find the GPU, keep training apart. Use this environment for extraction and analysis, and a second one with the latest PyTorch for the training page, whose code runs there unchanged:

conda create -n dnn-train python=3.11
conda activate dnn-train
pip install torch torchvision pandas scikit-learn matplotlib

For large models or datasets, run on the GPU nodes of the VSC cluster. The HPC page explains how to log in, where to store data and how to submit jobs with Slurm. Install the same environment there. If it does not work with the cluster's GPUs, split it as in the previous tab.

Check that everything imports and whether PyTorch sees a GPU:

from importlib.metadata import version

import torch

# The installed version of each package (include these in your methods section)
for package in ["thingsvision", "rsatoolbox", "himalaya", "torch"]:
    print(f"{package:>12} {version(package)}")
# False is fine: everything here also runs on a CPU
print("GPU available:", torch.cuda.is_available())

Keep the environment clean

Install packages in the project environment only, never with pip install --user. Packages in ~/.local can silently override the ones in your environment. Setting export PYTHONNOUSERSITE=1 in your shell makes Python ignore them.
Once the analysis works, save the exact versions with conda env export > environment.lock.yml and commit that file with your code.
Importing thingsvision also loads TensorFlow, which prints a few lines of warnings. export TF_CPP_MIN_LOG_LEVEL=3 silences them.
See Coding practices for editors, Git and general Python set-up.

2. Get the toy kit¶

All pages in this section use the same small dataset. It has 24 pixel-art sprites in three categories (critters, food and spooky things), and every sprite is shown on three scenes (a meadow, a room and a night sky) in two positions (centred, and shifted 6 pixels down and to the right). That makes 144 images of 48 × 48 pixels. Every sprite appears in every scene and position, so neither the scene nor the position tells you the category. A layer or a brain region that separates the categories has to do it from the sprites themselves.

Download the toy kit

Unzip it next to your scripts. You get this layout:

dnn-toy-kit/
├── manifest.csv          # one row per image
├── stimuli/              # 144 PNG files, 48 x 48 pixels
└── brain/
    ├── sub-01.npz        # one file per participant: V1 and IT patterns
    ├── ...
    └── sub-10.npz

The brain data in the kit are simulated

These pages start after the brain analysis, with one activity pattern per image and run from each region of interest (see the fMRI analysis workflow; Compare with human data shows how to read them from an SPM GLM). The kit's patterns are simulated with PcmPy for ten participants with six runs each, in two regions of interest called V1 and IT.

Every file lists the images in the same order, the order of manifest.csv. Your own data should follow the same structure, so that the code on these pages works on it unchanged:

The manifest has one row per image. The category, scene and position columns are the three properties we test on the next pages. The sprite column names the object, so that all six versions of a sprite stay together when you split the images into a training and a test set. In each participant's file, the arrays V1 and IT hold the patterns as runs × images × voxels, and image_id stores the image order. The two ROIs have different numbers of voxels. Your own ROIs can have any number of runs and voxels, as long as the image axis follows the manifest.

Load the manifest:

from pathlib import Path

import pandas as pd

KIT = Path("dnn-toy-kit")  # the folder you unzipped, next to this script
# One row per image, in the order every array uses
manifest = pd.read_csv(KIT / "manifest.csv")
print(manifest[["image_id", "sprite", "category", "scene", "position"]].head())
# Sprites per category in every scene and position: 8 everywhere
print(pd.crosstab(manifest["category"], [manifest["scene"], manifest["position"]]))

Output

                image_id   sprite category   scene position
0      cat_meadow_centre      cat  critter  meadow   centre
1     duck_meadow_centre     duck  critter  meadow   centre
2     frog_meadow_centre     frog  critter  meadow   centre
3  penguin_meadow_centre  penguin  critter  meadow   centre
4      dog_meadow_centre      dog  critter  meadow   centre
scene    meadow        night         room
position centre shift centre shift centre shift
category
critter       8     8      8     8      8     8
food          8     8      8     8      8     8
spooky        8     8      8     8      8     8

Plot the sprites

import matplotlib.pyplot as plt
from PIL import Image

# One image per sprite: the meadow, centred
one_version = manifest[(manifest["scene"] == "meadow") & (manifest["position"] == "centre")]
fig, axes = plt.subplots(3, 8, figsize=(9, 4))  # one row per category
for ax, (_, row) in zip(axes.flat, one_version.iterrows()):
    ax.imshow(Image.open(KIT / row["file"]), interpolation="nearest")  # (1)!
    ax.set_title(row["sprite"], fontsize=9)
    ax.set(xticks=[], yticks=[])  # no ticks, but keep the thin frame around each image
plt.show()

# The six versions of one sprite: three scenes, two positions
cat = manifest[manifest["sprite"] == "cat"]
fig, axes = plt.subplots(1, 6, figsize=(9, 1.9))
for ax, (_, row) in zip(axes, cat.iterrows()):
    ax.imshow(Image.open(KIT / row["file"]), interpolation="nearest")
    ax.set_title(f"{row['scene']}, {row['position']}", fontsize=9)
    ax.set(xticks=[], yticks=[])
plt.show()

interpolation="nearest" draws each pixel as a sharp square. The default smooths the pixel art into a blur.

From an image to an RDM¶

If neural networks are new to you, start here. The animation shows what the rest of this section does with every image, and how to read the model cards below.

Each filter of AlexNet's first layer slides over the image and gives one map of responses, and the ReLU sets the negative responses to zero. The 64 maps of 55 × 55 are laid end to end into one vector of 193,600 numbers per image, and the RDM holds 1 − Pearson r between the vectors of every pair of images. The bars are only a preview of each vector, while the RDM uses all its values. Each pair of sprites in the animation shares a scene, the critters on the meadow, the food in the room and the spooky sprites at night, so the three dark blocks of the RDM can come from the scene as much as from the category. The kit crosses the two, and Compare with a model shows how to tell them apart. The animation uses AlexNet, whose first layer is the simplest to show. The rest of this section uses ResNet-50, which works the same way with more layers.

3. Pick a model¶

You rarely need to train a network yourself. Many trained models are public, and studies often compare several of them with the brain. Each card below says when a model is a good choice and shows what it does with our cat sprite. You see its first layer (four filters and their responses, or a view of the whole layer for models without shared filters) and the RDMs of its first and last layers over all 144 images, with the critter, food and spooky blocks marked.

Pick by your question, not by accuracy

A model is a hypothesis about the brain. Choose models that differ in the way your question asks about, such as architecture and size (Maniquet et al., 2025), training data and objective (Duyck et al., 2024) or topography (Cortinovis et al., 2025), and compare several rather than one.

The models at a glance (each name links to its card):

Model	Kind	Pick it when	Loads with
ResNet-50	classic	you want a deep, strong ImageNet baseline	thingsvision
AlexNet	classic	you want the shared reference model	thingsvision
CORnet-S	brain-inspired	layers should map onto V1, V2, V4 and IT	thingsvision
VOneNet	brain-inspired	you want a fixed model of V1 at the front	its own package
HMAX	brain-inspired	you want a model without trained weights	its own repository
CLIP	language	your question is about meaning	thingsvision
TDANN	topographic	your question is about cortical maps	its own repository
TopoNets	topographic	you want topography in a standard network	its own package
All-TNN	topographic	you want topography without convolutions	its own repository (TensorFlow)

Classic image classifiers¶

ResNet-50¶

He et al., 2016 Docs

Pick it when you want a deep, strong ImageNet baseline. Skip connections are what make a network this deep trainable, and many brain-inspired models (VOneNet, CLIP's RN50, TopoNets) are built on ResNets, so it is the natural control for them.

Architecture: CNN, 50 layers with skip connections
Trained on: ImageNet, 1000 labels
In thingsvision: resnet50, source torchvision
License: BSD-3 code; weights under ImageNet terms

Load it

from thingsvision import get_extractor

extractor = get_extractor(
    model_name="resnet50",
    source="torchvision",
    device="cpu",
    pretrained=True,
    model_parameters={"weights": "IMAGENET1K_V2"},  # (1)!
)

torchvision has two sets of ResNet-50 weights: IMAGENET1K_V1 (the original training recipe) and IMAGENET1K_V2 (a newer recipe, more accurate). They are different models, so report which one you used.

AlexNet¶

Krizhevsky et al., 2012 Docs

Pick it when you want the shared reference point: five convolutional and three fully connected layers, reported in many DNN-brain studies. Small and fast.

Architecture: CNN, 8 layers: 5 convolutional, 3 fully connected
Trained on: ImageNet, 1000 labels
In thingsvision: alexnet, source torchvision
License: BSD-3 code; weights under ImageNet terms

Load it

from thingsvision import get_extractor

extractor = get_extractor(
    model_name="alexnet",
    source="torchvision",  # the model comes from torchvision's model zoo
    device="cpu",
    pretrained=True,
    model_parameters={"weights": "IMAGENET1K_V1"},  # (1)!
)

Name the weights explicitly and report them, because torchvision may change its default weights in a later release.

Brain-inspired models¶

CORnet-S¶

Kubilius et al., 2019 Code

Pick it when you want layers that map onto brain areas by design: four areas named V1, V2, V4 and IT, with recurrence inside V2, V4 and IT (V1 is feed-forward). Its architecture was chosen for how well it predicts ventral-stream data (Brain-Score), not only for accuracy.

Architecture: Recurrent CNN with areas V1, V2, V4 and IT
Trained on: ImageNet, 1000 labels
In thingsvision: cornet-s, source custom
License: GPL-3.0

Load it

from thingsvision import get_extractor

extractor = get_extractor(
    model_name="cornet-s",
    source="custom",  # (1)!
    device="cpu",
    pretrained=True,
)

thingsvision ships CORnet itself, so no extra package is needed. The first download is 408 MB. Areas such as IT.output run several times per image (the recurrent time steps), so report which step you used.

VOneNet¶

Dapello et al., 2020 Code

Pick it when you want a model of primate V1 in front of a standard CNN: a fixed bank of Gabor filters with simple and complex cells and neuronal noise, followed by a trained ResNet-50. It was introduced to study robustness to image perturbations.

Architecture: Fixed model of V1, then a ResNet-50
Trained on: ImageNet, 1000 labels (the ResNet part)
In thingsvision: wrap it with get_extractor_from_model
License: GPL-3.0

Load it

# pip install git+https://github.com/dicarlolab/vonenet.git
import torch
import vonenet
from thingsvision import get_extractor_from_model
from torchvision import transforms

model = vonenet.get_model(model_arch="resnet50", pretrained=True, map_location="cpu").module  # (1)!
preprocess = transforms.Compose([
    transforms.Resize(224, interpolation=transforms.InterpolationMode.NEAREST),
    transforms.ToTensor(),
    # VOneNet's own normalisation
    transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5]),
])
extractor = get_extractor_from_model(model=model, device="cpu", backend="pt", preprocess=preprocess)
torch.manual_seed(0)  # (2)!

get_model returns the network wrapped for multiple GPUs, and .module unwraps it so that layer names do not start with module..
VOneNet adds random noise to its V1 responses on every pass, so two runs give slightly different activations. Fix the seed before extracting.

HMAX¶

Riesenhuber & Poggio, 1999 Serre et al., 2007 Code

Pick it when you want a classic model of the ventral stream with no weights trained by gradient descent: fixed Gabor filters and a stored set of patches, in alternating simple (S) and complex (C) layers, with a MAX operation that makes responses tolerant to position and size. A useful baseline against trained networks.

Architecture: Simple and complex cell layers, S1 to C2
Trained on: nothing: fixed Gabor filters and stored patches
In thingsvision: no; call the model directly
License: no license file

Load it

# git clone https://github.com/wmvanvliet/pytorch_hmax  (not on PyPI)
import sys

sys.path.append("pytorch_hmax")  # the folder you cloned
import hmax

model = hmax.HMAX("pytorch_hmax/universal_patch_set.mat")  # (1)!

The patch file ships with the repository. HMAX takes grayscale images with values from 0 to 255, and its output is the top (C2) layer, with 8 scales × 400 patches per image. It is not a thingsvision model, so call it directly, as the repository's example shows.

Trained with language¶

CLIP¶

Radford et al., 2021 Code

Pick it when your question is about meaning: CLIP learned image features by matching images to their captions, not to a fixed list of ImageNet labels. Compare it with an ImageNet model of the same architecture to see what language supervision adds.

Architecture: Image encoder: ViT-B/32 or ResNet-50
Trained on: 400 million image-text pairs
In thingsvision: clip, source custom
License: MIT; research use only, per the model card

CLIP's 768 patch filters look noisy one by one, so the strip shows four principal components of them, as the ViT paper does (Dosovitskiy et al., 2021, Fig. 7). Each map shows how strongly each of the 49 patches of the cat loads on that component. Loadings can be negative, so here white is the lowest loading, not 0.

Load it

from thingsvision import get_extractor

extractor = get_extractor(
    model_name="clip",
    source="custom",  # needs the CLIP package from environment.yml
    device="cpu",
    pretrained=True,
    model_parameters={"variant": "ViT-B/32"},  # (1)!
)

"RN50" gives the ResNet-50 version. The ViT version cuts each image into 32 × 32 patches, so its "first layer" is a 7 × 7 grid of patch responses rather than a feature map.

Topographic models¶

TDANN¶

Margalit et al., 2024 Code

Pick it when your question is about cortical maps: every unit has a position on a simulated cortical sheet, and the training balances a self-supervised task with spatial smoothness. It reproduces V1-like maps and category-selective patches in ventral temporal cortex.

Architecture: ResNet-18 whose units have positions on a cortical sheet
Trained on: self-supervised task (SimCLR) plus a spatial loss
In thingsvision: wrap it with get_extractor_from_model
License: no license file

On the left, each unit's preferred orientation, measured with TDANN's own grating images and tuning fits and smoothed over 1.5 mm as in the paper. Neighbouring units prefer similar orientations, and the colours meet at pinwheel-like points. On the right, each unit's response to the cat at its position on the sheet, as in the TDANN demo.

Load it

# git clone https://github.com/neuroailab/TDANN ; weights from https://osf.io/64qv3/
import sys

import torch
from thingsvision import get_extractor_from_model

# The repository's demo loader needs only torch and torchvision
sys.path.append("TDANN/demo")
from src.model import load_model_from_checkpoint

model = load_model_from_checkpoint("model_final_checkpoint_phase199.torch")  # (1)!
extractor = get_extractor_from_model(model=model, device="cpu", backend="pt")

The SimCLR checkpoint from OSF (88 MB). On a computer without a GPU, the loader's torch.load(path) fails with "Attempting to deserialize object on a CUDA device". Change it to torch.load(path, map_location="cpu", weights_only=True) in TDANN/demo/src/model.py (weights_only=True also stops the file from running code when it loads). For maps you also need each unit's position, from the same OSF project.

TopoNets¶

Deb et al., 2025 Code

Pick it when you want topography in a standard, high-performing network: a topographic loss makes neighbouring units respond alike without much loss in accuracy. It loads as an ordinary PyTorch model.

Architecture: ResNet-18/50 or ViT-B/32 with a topographic loss
Trained on: ImageNet, 1000 labels
In thingsvision: wrap it with get_extractor_from_model
License: no license file

The paper's vision maps show category selectivity (Fig. 5A). This view of the first topographic layer is ours. It shows each channel's response to the cat, placed on the 8 × 8 grid that TopoLoss uses for this layer. Neighbouring tiles tend to look alike.

Load it

# pip install git+https://github.com/toponets/toponets.git
import toponets  # (1)!
from thingsvision import get_extractor_from_model

model = toponets.resnet18(tau=10.0, checkpoint_path="resnet18_tau_10.0.pt")  # (2)!
extractor = get_extractor_from_model(model=model, device="cpu", backend="pt")

With the PyTorch 2.0.1 of our environment, add torch.amp.GradScaler = torch.cuda.amp.GradScaler (after import torch) before this import. Newer PyTorch versions do not need it.
tau sets the strength of the topographic loss. The checkpoint (45 MB) is downloaded from Hugging Face to checkpoint_path the first time, with wget.

All-TNN¶

Lu et al., 2025 Code Weights

Pick it when you want topography without convolutions: every unit on a simulated cortical sheet has its own filter, and smooth orientation and category maps emerge during training. All-TNNs also matched human spatial biases in object recognition better than control models.

Architecture: Locally connected layers on a 2-D sheet, no weight sharing
Trained on: ecoset, 565 categories
In thingsvision: no; TensorFlow 2.12, load it with the repository's helpers
License: MIT

On the left, each unit's preferred orientation, measured with gratings as in the repository's get_tuning_curves. Neighbouring units prefer similar orientations, as in V1. White dots are units that respond to no grating (0.7%). On the right, the summed response of the 64 units at each position in the visual field, which traces the cat and the edge of the grass.

Load it

# git clone https://github.com/KietzmannLab/All-TNN
# In a separate environment: pip install tensorflow==2.12.0 "numpy<=1.23.5" "h5py<=3.8.0" scipy scikit-learn pyyaml
# From https://osf.io/6m3g4/ download one model folder from shared_weights/ (862 MB)
import sys

sys.path.insert(0, "All-TNN")  # (1)!
from all_tnn.analysis.util.analysis_help_funcs import load_and_override_hparams
from all_tnn.analysis.util.setup_model_and_data import load_models

MODEL = "shared_weights/tnn_ecoset_l2_no_flip_seed1_drop0.0_learnable_False_1e-05_alpha10.0_constant_20_0.1Factors_adam0.05_L21e-06_ecoset_square256_proper_chunks/"
hparams = load_and_override_hparams(MODEL, batch_size=1)  # (2)!
activities_model, model = load_models(300, hparams, MODEL, pre_relu=False, post_norm=False, n_classes_override=565)

pip install -e . fails in the current repository (a missing _version.py), so put the clone on the path instead. The network is built from the code saved inside the model folder, which is why you need the whole folder, not only the checkpoint.
The input layer has a fixed batch size, so set it before the model is built. Images go in as 150 × 150 RGB, scaled to the range −1 to 1, and activities_model returns the response of each of the six sheets.

4. Run the model on the sprites¶

Before any analysis, check that the model runs on your images and that the preprocessing matches what it was trained on. The example network on these pages is ResNet-50. What does it see in our sprites?

from pathlib import Path

import torch
from PIL import Image
from torchvision import models, transforms

KIT = Path("dnn-toy-kit")
weights = models.ResNet50_Weights.IMAGENET1K_V2  # (1)!
model = models.resnet50(weights=weights).eval()  # (2)!

preprocess = transforms.Compose([
    transforms.Resize(224, interpolation=transforms.InterpolationMode.NEAREST),  # (3)!
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),  # (4)!
])

for sprite in ["cat", "banana", "ghost", "pizza"]:
    img = Image.open(KIT / f"stimuli/{sprite}_meadow_centre.png").convert("RGB")
    with torch.no_grad():  # (5)!
        # unsqueeze(0): a batch of one image; softmax: the 1000 outputs as probabilities
        probs = model(preprocess(img).unsqueeze(0)).softmax(dim=1)
    p, idx = probs.max(dim=1)  # the most likely ImageNet class and its probability
    print(f"{sprite:>7} -> {weights.meta['categories'][idx.item()]} ({p.item():.0%})")

For this quick check we use torchvision directly. The weights object knows the preprocessing the model was trained with (weights.transforms()) and the names of the 1000 ImageNet classes (weights.meta["categories"]).
.eval() makes batch normalisation use the statistics stored during training (and switches off dropout, in networks that have it). Always use it when you only run images through a model.
ImageNet models expect 224 × 224 pixels. Nearest-neighbour upsampling keeps the pixel art sharp. For photographs, use weights.transforms().
The mean and standard deviation of the ImageNet images. Every model has its own, so check the documentation of the weights you use.
No gradients are needed to run a model, and skipping them saves memory and time.

The output is:

    cat -> picket fence (3%)
 banana -> picket fence (5%)
  ghost -> envelope (3%)
  pizza -> picket fence (12%)

Why does ResNet-50 get these wrong?

ResNet-50 learned from photographs. Cartoons of 48 by 48 pixels are far from anything it has seen, so its 1000 ImageNet classes do not fit. This is common with experimental stimuli (line drawings, chess boards, scrambled images). You can still record its layers and compare them with the brain, because the early layers respond to edges and colours whatever the image. If you need the network to know your categories, train or fine-tune it.

Where next?¶

Train or fine-tune

Teach ResNet-50 the sprites' categories, by fine-tuning or from scratch, and test it on sprites it has not seen.
Extract activations

Record what every layer does with each image and save it for analysis.