For decades, neuroscientists have studied the human brain by asking people to lie very still inside a brain scanner and perform carefully controlled tasks. Those experiments have revealed a remarkable amount about how the brain works. But they also leave out something fundamental.
Real life is not a series of isolated tasks. We are constantly perceiving the world, making predictions, deciding what to do, acting on our surroundings and adjusting our behaviour when those actions produce unexpected results.
Now, Western University neuroscientists Jörn Diedrichsen and Jody Culham are helping lead an international effort to study the brain in a way that more closely mimics this complexity – by putting people inside an fMRI scanner and having them play video games.
The work is part of the Digital Brain Project, a large international research initiative supported by social media giant Meta that aims to create a massive, openly shared dataset of human brain activity. The project brings together researchers from institutions around the world, including Western, to explore how the brain behaves across a much broader range of cognitive states than conventional brain imaging typically captures.
Diedrichsen has spent years developing ways to map brain activity across many different tasks, including the Raynor Cerebellum Project, rather than relying primarily on resting-state scans or one carefully isolated experiment at a time. His lab has developed a battery of more than 60 tasks that can be performed during an fMRI scan, allowing researchers to push one participant’s brain into many different states.
“Think of brain activity as an enormous landscape. At any given moment, your brain is somewhere in that landscape, and different experiences, decisions and actions move you to different places,” said Diedrichsen, a computer science professor.
Resting-state fMRI – or functional magnetic resonance imaging, which measures brain activity through changes in blood flow – allows researchers to observe the brain’s spontaneous fluctuations, but gives them little control over where in that landscape the brain goes. Diedrichsen’s multi-task approach deliberately samples a much wider range of brain activity.
Diedrichsen and Culham saw a way to bring their strategies together – his work on sampling a broad range of brain activity with her lab’s work on how the brain responds when people actively interact with changing environments. Video games offered a way to put both ideas into the same project.
When someone is playing a game like Pac-Man, they are constantly making predictions. A ghost is approaching, so they change direction. They move the controller, Pac-Man responds, the environment changes and the player immediately updates their plan,” said Culham, Canada Research Chair in Immersive Neuroscience and psychology professor.
It is a continuous loop of prediction, action and feedback, which is fundamentally different from asking someone to look at an image and press a button to indicate what they see.
In previous work, Culham’s lab has compared people playing Pac-Man with people watching the game and with people moving a controller that has no effect on the game. The sensory experience and physical movements can be similar, but the player’s relationship with the environment is completely different.
This experience of making predictions, taking action and getting feedback is called a ‘closed loop’ between the person and the environment. And that loop may be one of the most important things missing from conventional approaches to studying the brain.
Western University neuroscientists Jörn Diedrichsen and Jody Culham, who are working together on the Digital Brain Project. (Christopher Kindratsky/Western Communications)
