Jeff Renaud

For decades, neuroscientists have studied the human brain by asking people to lie very still inside a brain scanner and perform carefully controlled tasks. Those experiments have revealed a remarkable amount about how the brain works. But they also leave out something fundamental. Real life is not a series of isolated tasks. We are constantly perceiving the world, making predictions, deciding what to do, acting on our surroundings and adjusting our behaviour when those actions produce unexpected results. Now, Western University neuroscientists Jörn Diedrichsen and Jody Culham are helping lead an international effort to study the brain in a way that more closely mimics this complexity – by putting people inside an fMRI scanner and having them play video games. The work is part of the Digital Brain Project, a large international research initiative supported by social media giant Meta that aims to create a massive, openly shared dataset of human brain activity. The project brings together researchers from institutions around the world, including Western, to explore how the brain behaves across a much broader range of cognitive states than conventional brain imaging typically captures. Diedrichsen has spent years developing ways to map brain activity across many different tasks, including the Raynor Cerebellum Project, rather than relying primarily on resting-state scans or one carefully isolated experiment at a time. His lab has developed a battery of more than 60 tasks that can be performed during an fMRI scan, allowing researchers to push one participant’s brain into many different states. “Think of brain activity as an enormous landscape. At any given moment, your brain is somewhere in that landscape, and different experiences, decisions and actions move you to different places,” said Diedrichsen, a computer science professor. Resting-state fMRI – or functional magnetic resonance imaging, which measures brain activity through changes in blood flow – allows researchers to observe the brain’s spontaneous fluctuations, but gives them little control over where in that landscape the brain goes. Diedrichsen’s multi-task approach deliberately samples a much wider range of brain activity. Diedrichsen and Culham saw a way to bring their strategies together – his work on sampling a broad range of brain activity with her lab’s work on how the brain responds when people actively interact with changing environments. Video games offered a way to put both ideas into the same project. When someone is playing a game like Pac-Man, they are constantly making predictions. A ghost is approaching, so they change direction. They move the controller, Pac-Man responds, the environment changes and the player immediately updates their plan,” said Culham, Canada Research Chair in Immersive Neuroscience and psychology professor. It is a continuous loop of prediction, action and feedback, which is fundamentally different from asking someone to look at an image and press a button to indicate what they see. In previous work, Culham’s lab has compared people playing Pac-Man with people watching the game and with people moving a controller that has no effect on the game. The sensory experience and physical movements can be similar, but the player’s relationship with the environment is completely different. This experience of making predictions, taking action and getting feedback is called a ‘closed loop’ between the person and the environment. And that loop may be one of the most important things missing from conventional approaches to studying the brain.  

Exploring the brain’s landscape

Rather than simply asking participants to perform a long list of conventional laboratory tasks, Diedrichsen and Culham are combining carefully designed cognitive tasks with immersive video games, which will be built around rich 3D environments and first-person avatars A core set of tasks will remain as anchors for data comparison – including activities like kart-racing, action and crafter video games, digital puzzles and go-kart racing, while new tasks and game elements will be introduced throughout the Digital Brain Project. “By gamifying conventional tasks, our project can provide a bridge between strategies to understand well-established cognitive functions and strategies to provide maximal entertainment and engagement,” said Culham, who like Diedrichsen, is a faculty member in the Western Institute for Neuroscience. “That gives us a way to explore a much broader range of brain states while also capturing the brain as it predicts, acts and adapts.”  

A brain scanner built for the experiment

The new study is beginning as Western upgrades one of the key pieces of equipment needed to conduct it. Western is investing in an upgraded 7-Tesla MRI scanner, a powerful system housed at the Centre for Functional and Metabolic Mapping capable of producing markedly higher-quality images than conventional MRI systems. For the Digital Brain Project, the researchers are also developing a new 32-channel head coil designed to improve data collected from deep brain structures, including the cerebellum, thalamus and midbrain. That matters particularly to Diedrichsen, whose research focuses on the cerebellum and its connections with the cerebral cortex. https://www.youtube.com/watch?v=LzKKjlUWCQY “We’ve traditionally thought of the cerebellum as being about movement and coordination, but it’s increasingly clear that it’s also involved in prediction and cognition. That makes it a really interesting part of the brain to study when you’re looking at how people interact with a changing environment,” said Diedrichsen. The upgraded scanner will also provide a much larger visual display, making it possible to create more immersive environments for participants lying inside the scanner. The researchers are working toward 3D displays that could make the participant feel as though they are inside the game world, despite being physically confined to the MRI.  

From individual to digital brain

One of the project’s larger ambitions is to move beyond the idea of the “average” human brain. Individual brains can differ considerably, particularly in regions involved in complex actions and cognition, and Diedrichsen wants to collect rich data from individual participants across many different activities. “If you take a picture of a forest and put all the trees on top of each other, you get a very blurry picture of what a tree actually looks like, brown on the bottom, green on top,” said Diedrichsen. “The same thing happens when we average brains together. We want to understand the individual brain in much more detail.” That kind of deep individual mapping could eventually help researchers understand why brains are organized differently from one person to another and how that may influence behaviour. For now, however, the goal is to create a large, high-quality dataset researchers around the world can use, with the project’s data openly shared for non-commercial research. “AI is only as good as the data you give it. If you don’t have good data, you’re starting with garbage. What we’re trying to do is build the kind of rich, high-quality dataset that allows people to ask new questions and apply new methods we haven’t even thought of yet,” said Diedrichsen. AI will also become part of the experiment itself, with researchers developing artificial systems that can learn cognitive tasks and play simple games. By examining how those systems learn to navigate, predict and make decisions, scientists can explore possible solutions to problems faced by biological brains, using them as what Diedrichsen calls an “intuition pump” for thinking about how complex systems work. The researchers are not suggesting resting-state fMRI or conventional task-based experiments should disappear. Instead, they see the video-game approach as opening another part of the brain’s functional landscape – one that has been difficult to reach with experiments built around isolated questions and tightly controlled responses. “We’re trying closely study the brain as it actually works,” said Culham. “Video games give us a way to bring some of that into the scanner and start asking what the brain is doing when it’s engaged in the world.” Learn more about how Western is optimizing health for all.