The Future of the Mind (12 page)

Gallant’s MRI machine is so powerful it can identify two to three hundred
distinct regions of the brain and, on average, can take snapshots that have one hundred dots per region of the brain. (One goal for future generations of MRI technology is to provide an even sharper resolution by increasing the number of dots per region of the brain.)

At first, this 3-D collection of colored dots looks like gibberish. But after years of research, Dr. Gallant and his colleagues have developed a mathematical formula that begins to find relationships between certain features of a picture (edges, textures, intensity, etc.) and the MRI voxels. For example, if you look at a boundary, you’ll notice it’s a region separating lighter and darker areas, and hence the edge generates a certain pattern of voxels. By having subject after subject view such a large library of movie clips, this mathematical formula is refined, allowing the computer to analyze how all sorts of images are converted into MRI voxels. Eventually the scientists were able to ascertain a direct correlation between certain MRI patterns of voxels and features within each picture.

At this point, the subject is then shown another movie trailer. The computer analyzes the voxels generated during this viewing and re-creates a rough approximation of the original image. (The computer selects images from one hundred movie clips that most closely resemble the one that the subject just saw and then merges images to create a close approximation.) In this way, the computer is able to create a fuzzy video of the visual imagery going through your mind. Dr. Gallant’s mathematical formula is so versatile that it can take a collection of MRI voxels and convert it into a picture, or it can do the reverse, taking a picture and then converting it to MRI voxels.

I had a chance to view the video created by Dr. Gallant’s group, and it was very impressive. Watching it was like viewing a movie with faces, animals, street scenes, and buildings through dark glasses. Although you could not see the details within each face or animal, you could clearly identify the kind of object you were seeing.

Not only can this program decode what you are looking at, it can also decode imaginary images circulating in your head. Let’s say you are asked to think of the
Mona Lisa
. We know from MRI scans that even though you’re not viewing the painting with your eyes, the visual cortex of your brain will light up. Dr. Gallant’s program then scans your brain while you are thinking of the
Mona Lisa
and flips through its data files of pictures, trying to find the closest match. In one experiment I saw, the computer selected a picture of the actress Salma Hayek as the closest approximation to the
Mona Lisa
. Of
course, the average person can easily recognize hundreds of faces, but the fact that the computer analyzed an image within a person’s brain and then picked out this picture from millions of random pictures at its disposal is still impressive.

The goal of this whole process is to create an accurate dictionary that allows you to rapidly match an object in the real world with the MRI pattern in your brain. In general, a detailed match is very difficult and will take years, but some categories are actually easy to read just by flipping through some photographs. Dr. Stanislas Dehaene of the Collège de France in Paris was examining MRI scans of the parietal lobe, where numbers are recognized, when one of his postdocs casually mentioned that just by quickly scanning the MRI pattern, he could tell what number the subject was looking at. In fact, certain numbers created distinctive patterns on the MRI scan. He notes, “
If you take 200 voxels in this area, and look at which of them are active and which are inactive, you can construct a machine-learning device that decodes which number is being held in memory.”

This leaves open the question of when we might be able to have picture-quality videos of our thoughts. Unfortunately, information is lost when a person is visualizing an image. Brain scans corroborate this. When you compare the MRI scan of the brain as it is looking at a flower to an MRI scan as the brain is thinking about a flower, you immediately see that the second image has far fewer dots than the first. So although this technology will vastly improve in the coming years, it will never be perfect. (I once read a short story in which a man meets a genie who offers to create anything that the person can imagine. The man immediately asks for a luxury car, a jet plane, and a million dollars. At first, the man is ecstatic. But when he looks at these items in detail, he sees that the car and the plane have no engines, and the image on the cash is all blurred. Everything is useless. This is because our memories are only approximations of the real thing.)

But given the rapidity with which scientists are beginning to decode the MRI patterns in the brain, will we soon be able to actually read words and thoughts circulating in the mind?

READING THE MIND

In fact, in a building next to Gallant’s laboratory,
Dr. Brian Pasley and his colleagues are literally reading thoughts—at least in principle. One of the
postdocs there, Dr. Sara Szczepanski, explained to me how they are able to identify words inside the mind.

The scientists used what is called ECOG (electrocorticogram) technology, which is a vast improvement over the jumble of signals that EEG scans produce. ECOG scans are unprecedented in accuracy and resolution, since signals are directly recorded from the brain and do not pass through the skull. The flipside is that one has to remove a portion of the skull to place a mesh, containing sixty-four electrodes in an eight-by-eight grid, directly on top of the exposed brain.

Luckily they were able to get permission to conduct experiments with ECOG scans on epileptic patients, who were suffering from debilitating seizures. The ECOG mesh was placed on the patients’ brains while open-brain surgery was being performed by doctors at the nearby University of California at San Francisco.

As the patients hear various words, signals from their brains pass through the electrodes and are then recorded. Eventually a dictionary is formed, matching the word with the signals emanating from the electrodes in the brain. Later, when a word is uttered, one can see the same electrical pattern. This correspondence also means that if one is thinking of a certain word, the computer can pick up the characteristic signals and identify it.

With this technology, it might be possible to have a conversation that takes place entirely telepathically. Also, stroke victims who are totally paralyzed may be able to “talk” through a voice synthesizer that recognizes the brain patterns of individual words.

Not surprisingly, BMI (brain-machine interface) has become a hot field, with groups around the country making significant breakthroughs.
Similar results were obtained by scientists at the University of Utah in 2011. They placed grids, each containing sixteen electrodes, over the facial motor cortex (which controls movements of the mouth, lips, tongue, and face) and Wernicke’s area, which processes information about language.

The person was then asked to say ten common words, such as “yes” and “no,” “hot” and “cold,” “hungry” and “thirsty,” “hello” and “good-bye,” and “more” and “less.” Using a computer to record the brain signals when these words were uttered, the scientists were able to create a rough one-to-one correspondence between spoken words and computer signals from the brain. Later, when the patient voiced certain words, they were able to correctly
identify each one with an accuracy ranging from 76 percent to 90 percent. The next step is to use grids with 121 electrodes to get better resolution.

In the future, this procedure may prove useful for individuals suffering from strokes or paralyzing illnesses such as Lou Gehrig’s disease, who would be able to speak using the brain-to-computer technique.

TYPING WITH THE MIND

At the Mayo Clinic in Minnesota, Dr. Jerry Shih has hooked up epileptic patients via ECOG sensors so they can learn how to type with the mind. The calibration of this device is simple. The patient is first shown a series of letters and is told to focus mentally on each symbol. A computer records the signals emanating from the brain as it scans each letter. As with the other experiments, once this one-to-one dictionary is created, it is then a simple matter for the person to merely think of the letter and for the letter to be typed on a screen, using only the power of the mind.

Dr. Shih, the leader of this project, says that the accuracy of his machine is nearly 100 percent. Dr. Shih believes that he can next create a machine to record images, not just words, that patients conceive in their minds.
This could have applications for artists and architects, but the big drawback of ECOG technology, as we have mentioned, is that it requires opening up patients’ brains.

Meanwhile, EEG typewriters, because they are noninvasive, are entering the marketplace. They are not as accurate or precise as ECOG typewriters, but they have the advantage that they can be sold over the counter. Guger Technologies, based in Austria, recently demonstrated an EEG typewriter at a trade show.
According to their officials, it takes only ten minutes or so for people to learn how to use this machine, and they can then type at the rate of five to ten words per minute.

TELEPATHIC DICTATION AND MUSIC

The next step might be to transmit entire conversations, which could rapidly speed up telepathic transmission. The problem, however, is that it would require making a one-to-one map between thousands of words and their EEG, MRI, or ECOG signals. But if one can, for example, identify the brain
signals of several hundred select words, then one might be able to rapidly transmit words found in a common conversation. This means that one would think of the words in entire sentences and paragraphs of a conversation and a computer would print them out.

This could be extremely useful for journalists, writers, novelists, and poets, who could simply think and have a computer take dictation. The computer would also become a mental secretary. You would mentally give instructions to the robo-secretary about a dinner, plane trip, or vacation, and it would fill in all the details about the reservations.

Not only dictation but also music may one day be transcribed in this way. Musicians would simply hum a few melodies in their head and a computer would print them out, in musical notation. To do this, you would ask someone to mentally hum a series of notes, which would generate certain electrical signals for each one. A dictionary would again be created in this way, so that when you think of a musical note, the computer would print it out in musical notation.

In science fiction, telepaths often communicate across language barriers, since thoughts are considered to be universal. However, this might not be true. Emotions and feelings may well be nonverbal and universal, so that one could telepathically send them to anyone, but rational thinking is so closely tied to language that it is very unlikely that complex thoughts could be sent across language barriers. Words will still be sent telepathically in their original language.

TELEPATHY HELMETS

In science fiction, we also often encounter telepathy helmets. Put them on, and—presto!—you can read other people’s minds. The U.S. Army, in fact, has expressed interest in this technology. In a firefight, with explosions going off and bullets whizzing overhead, a telepathy helmet could be a lifesaver, since it can be difficult to communicate orders amid the sound and fury of the battlefield. (I can personally testify to this. Years ago, during the Vietnam War, I served in the U.S. Infantry at Fort Benning, outside Atlanta, Georgia. During machine-gun training, the sound of hand grenades and rounds of bullets going off on the battlefield next to my ear was deafening; it was so intense I could not hear anything else. Later, there was a loud ringing in my
ear that lasted for three full days.) With a telepathy helmet, a soldier could mentally communicate with his platoon amid all the thunder and noise.

Recently, the army gave a $6.3 million grant to Dr. Gerwin Schalk at Albany Medical College, but it knows that a fully functional telepathy helmet is still years away. Dr. Schalk experiments with ECOG technology, which, as we have seen, requires placing a mesh of electrodes directly on top of the exposed brain. With this method, his computers have been able to recognize vowels and thirty-six individual words inside the thinking brain. In some of his experiments, he approached 100 percent accuracy. But at present, this is still impractical for the U.S. Army, since it requires removing part of the skull in the clean, sterile environment of a hospital. And even then, recognizing vowels and a handful of words is a far cry from sending urgent messages to headquarters in a firefight. But his ECOG experiments have demonstrated that it is possible to communicate mentally on the battlefield.

Another method is
being explored by Dr. David Poeppel of New York University. Instead of opening up the skulls of his subjects, he employs MEG technology, using tiny bursts of magnetic energy rather than electrodes to create electrical charges in the brain. Besides being noninvasive, the advantage of MEG technology is that it can precisely measure fleeting neural activity, in contrast to the slower MRI scans. In his experiments, Poeppel has been able to successfully record electrical activity in the auditory cortex when people think silently of a certain word. But the drawback is that this recording still requires the use of large, table-size machines to generate a magnetic pulse.

Obviously, one wants a method that is noninvasive, portable, and accurate. Dr. Poeppel hopes his work with MEG technology will complement the work being done using EEG sensors. But true telepathy helmets are still many years away, because MEG and EEG scans lack accuracy.

MRI IN A CELL PHONE

At present, we are hindered by the relatively crude nature of the existing instruments. But, as time goes by, more and more sophisticated instruments will probe deeper into the mind. The next big breakthrough may be MRI machines that are handheld.