The President’s Distinguished Lecture Series: Using Machine Learning to Study How Brains Represent Language Meaning

Photo Courtesy CMU.edu

On Wednesday, Jan. 31 at 4 p.m., Dr. Tom M. Mitchell gave a lecture titled “Using Machine Learning to Study How Brains Represent Language Meaning” as part of The President’s Distinguished Lecture Series. He is the E. Fredkin University Professor at Carnegie Mellon University, where he founded the first-ever machine learning department. He received his B.S. in Electrical Engineering from MIT in 1973 and his Ph.D. from Stanford University in 1979.

Dr. Mitchell addressed a daring yet fundamental question: “How does neural activity encode word meanings?” He noted that our knowledge of the brain is not as limited by the resolution of current brain images as it is by the models we use to explain them. Dr. Mitchell and other researchers at Carnegie Mellon University greatly advanced these models using artificial intelligence. They found that a machine learning system, given a fMRI brain image, could be trained to consistently determine the word which an individual had been shown when the brain image was captured. Not only could the system distinguish between brain scans from the same volunteer, but it could also apply its training to assess the brain images of other people, even across languages in bilingual individuals. According to Dr. Mitchell, this shows that “we all have essentially the same neural code,” and, furthermore, that “there is a distinct structure to the neural code.” “Even though we all have very different backgrounds and life experiences, somehow we have a shared physical, spatial pattern of neural activity that encodes these different words,” explained Dr. Mitchell.

He presented the “predictive model theory.” By retrieving statistics on various words through an n-gram analysis and then creating a vector sum of a word’s “contributing features,” the AI system was able to construct and predict fMRI responses to words it had never even encountered. To Dr. Mitchell, this proves that “it’s not like you have a hash code” assigning different meanings to words. Instead, the neural code is “built out of more primitive components.” But the information which can be communicated in fMRI images is mostly limited to two-dimensional space due to their poor time resolution. Dr. Mitchell and other researchers used MEG neuroimaging to address this, which takes a reading every millisecond.

Using machine learning together with MEG, they found a specific order in which the brain asks itself identifying questions about the stimulus. First are perceptual features (“How long is this word?), followed by a question of whether it is animate (“Is it hairy?”), and then further questions (“Can you pick it up?” “Does it have corners?” etc.) until the brain is able to fully deduce a meaning after 400 milliseconds. According to Dr. Mitchell, the most reliably decodable questions regard the size, manipulability, animacy, and whether it could be used as shelter.

Dr. Mitchell pointed to the work of his colleague, Leila Wehbe, who used both MEG and fMRI neuroimaging to during storytelling, yielding fascinating results about the brain’s treatment of context. She found that context predictably affected the neural responses to words and that an AI system could be used to predict coming words as the human brain likely does. Dr. Mitchell argued that the “context information is most vibrant in the brain,” and that the “current word is secondary.”

Dr. Mitchell concluded that his research is evidence that the brain functions like a computer. The predictive model theory is, to him, of particular importance, as he believes that comparing AI predictions to images of human brains will be crucial in deciding the merit of using computers to study brain function. He calls himself an optimist, seeing the potential which new access to data brings to the world of machine learning; “Intelligence without perception of the world around it is a very restricted playground. There is so much data available now. It explodes the opportunity for AI research … into a whole new universe of things.”

Following his lecture, Dr. Tom Mitchell was awarded the President’s Medal from Stevens Institute of Technology, and an academic scholarship was established in his name.