Protected: News

A new AI tool can guess what you’re looking at just by analyzing your brain scans—and re-create that image with remarkable precision. It can go the other way, too, and predict a person’s brain activity based on what they’re looking at. 

In the image above, for example, the left-hand image of each pair is what the user actually saw—and its right-hand counterpart is what the model reconstructed from the brain scan.  

Michal Irani, who developed the tool with her colleagues at the Weizmann Institute of Science in Rehovot, Israel, hopes her “mind-reading” tool will ultimately reveal more about how the brain works, and could perhaps be used to help locked-in people communicate, or allow scientists to re-create the content of dreams. 

Judy Illes, a neuroethicist and professor of neurology at the University of British Columbia in Canada, who was not involved in the research, describes the work as “magnificent.” “The idea [of using this approach] to help people with neurologic conditions … therapeutically is tremendously exciting,” she says.

But other scientists warn that a similar approach could be used to reveal people’s inner thoughts and mental imagery, potentially without their consent. “The results seem very impressive,” says Tommy Sprague, a neuroscientist at the University of California, Santa Barbara. “But if there’s a way to surreptitiously extract information about what you’re thinking about, then …150 years of sci-fi can come true anytime, and that’s worrisome in a lot of ways.”

Peeking into the brain

Neuroscientists have been working for years on ways to use functional magnetic resonance imaging (fMRI) to reconstruct what people see and what’s going on in their minds. The first attempts produced images that were blurry and hard to make sense of. Advances in technology—both in the fMRI scans themselves and in the tools used to make sense of the results—have led to improvements over the years.

Irani and her colleagues started by analyzing publicly available brain-scan data. Other researchers had already collected scans from volunteers who were shown hundreds of images while they lay in fMRI scanners.

fMRI uses a giant magnet to track the flow of oxygenated blood through the brain. Brain areas that “light up” on the scans are thought to be those that are particularly active at any given moment. The results are not especially specific—in typical fMRI scanners, each highlighted “voxel” of activity covers around three cubic millimeters, containing around 16,000 neurons.

But Irani and her colleagues used newer datasets collected using scanners with a higher resolution—each voxel covered around one cubic millimeter of neurons, she says. Those datasets showed what the brain activity of volunteers looked like when they viewed various images.

Other teams have done this too, and several other tools have been used to re-create images from brain-scan data. But they’re not good enough, says Irani. Say a person saw a banana. These models can generate an image of a banana, but it would look different, she says. “It wouldn’t have the same structure, the same position.”

A better decoder

The researchers wanted to more closely re-create the images that had been seen. The first step was to train an AI model on already available data from eight people who each had been shown around 9,000 images while in a high-resolution fMRI scanner.

Crucially, their “brain decoder” has two branches—one to predict the structure of an image (where the colors are, for instance) and a second to predict its content (for example, a bunch of bananas on a plate). The predictions allow a diffusion model, a type of AI best known for creating video and images by gradually cleaning up a noisy mess of pixels, to produce a much more accurate representation of what the person saw.

But to improve the models they needed more data—far more than was actually available.  

To get around this problem, Irani and her colleagues trained another model—an encoder that can predict brain activity from an image. The team then used the encoder and decoder together to improve both tools.

It works like this: Start with a new image of, say, a leopard. Then use the encoder to predict what someone’s fMRI brain scan would look like when the person saw that picture. The decoder is then used to reconstruct the image. At first, the reconstruction probably won’t look much like a leopard, says Irani. But repeatedly training the models this way eventually leads to dramatic improvements.

This approach also allows the team to train their models on as many images as they want, even images that have never been shown to a person in an fMRI scanner. Irani says that around 70% of the training data is from images that were not originally paired with fMRI scans.

By combining data from multiple studies, they were also able to identify brain regions that seem to share functions across all individuals. One region seemed to respond to images of food, for example, while another responded to images of sports. Irani, a computer scientist, says she is now working with neuroscientists “to see if we can actually use these tools that we’ve developed to really find out new things about the brain.”

The resulting “universal brain encoder” can work on a scan from a new person with minimal calibration. In other attempts, a tool has typically required about 40 hours of fMRI data on anyone new before it can be used to predict what that person is seeing. Irani’s decoder only needs one hour of data, she says. The finding was presented at the Cognitive Computational Neuroscience conference in New York last month.

That could make it valuable for neuroscientists studying the brain, says Sprague. “None of us can afford 40 hours of imaging for a new subject,” he says. “It’s something like $600 to $1,000 an hour.” Tools like this one could speed up research, he says.

State of the art

The encoder and decoder aren’t perfect. “Of course we have failures,” says Irani. Over a Zoom call, she pointed out an image of a cake that her tool reconstructed as a pile of three sandwiches, and another of a dog in a bathtub that was reconstructed as a similarly colored goat in a bathtub.

But they represent the state of the art. In a comparison test, the tool was found to be much better than previously described ones. “All in all, really we outperformed the others by a significant margin,” Irani says. “Mind reading” is a “cute, jazzy name” for what they’re doing, she adds.

Irani is now planning to move beyond images to video and audio. She wants to be able to reconstruct what people are thinking about or imagining, and the contents of their dreams. “That’s something we don’t have yet,” she says. “But we’re striving to achieve it.”

Such a tool might enable people who are “locked in” and completely paralyzed to communicate using their brain activity alone, she says. It could also help scientists unpick some enduring mysteries surrounding the inner workings of our minds, such as what PTSD flashbacks look like.

Advances like this inevitably raise questions about mental privacy. What if some bad actor could reconstruct someone’s thoughts or memories?

“If you’d asked me that 10 years ago, I’d have laughed a lot,” says Sprague. Getting a willing person to lie still in a scanner and actively engage with a research question is hard enough; imagine making someone do it involuntarily. But Irani and other scientists are working on similar approaches to decode brain activity from EEG—electrical brain activity measures collected via a cap of electrodes or even through headphones. 

And as models improve, it will become even easier to analyze the brain activity collected this way. “We have to be a little more serious about the ethical considerations,” says Sprague. He thinks Irani’s approach would probably “work quite well” in predicting images that a person is thinking about but not looking at.

The move to EEG would be a “game changer,” says Marcello Ienca, a neuroscientist and philosopher at the Technical University of Munich, Germany. Once an EEG device has been calibrated to a user’s own brain, it could be relatively easy for companies to extract additional information from that person’s brain—potentially without consent. Ienca can also imagine some courts allowing mental image reconstructions as legal evidence.

“I have no doubt that this is, you know, well-intentioned research, but I think it’s also pretty obvious that it could be co-opted for … ethically and societally problematic commercial uses,” he says.

Irani acknowledges the potential for misuse with the use of EEG. But she’s not concerned for now. “I’m trying to think only of good things,” she says.