eye think lab

figurative language
The comprehension of a picture is more than the sum of its pixels, and the comprehension of a sentence is more than the sum of its words. When people listen to descriptions of objects, events, or scenes in their visual field, they must mesh verbal and visual information on the fly. How do they do this, especially when aspects of the verbal information are subtle or less than obvious? In this project, we are using the lens of implicit spatial language to view the integration of language and vision. Unlike explicit spatial language (e.g., X is above Y) or explicit referential language (e.g., Pick up the cube), implicit spatial language arises indirectly through implication, association, or metaphor.

Figurative language can be one form of implicit spatial language. Even though figurative language is pervasive in all cultures and all settings (Gibbs, 1994), eye movement research has focused on literal language. We discovered that figurative language can evoke mental representations distinct from those of equivalent literal sentences, and these representations immediately interact with visual processing.

In the first experiment of this project, we investigated how a scene would be perceived when it was described by forms of literal and figurative language that are reported to have equivalent meaning. We reasoned that any differences in eye movement patterns would tell us about both the distinct mental representations that are evoked by figurative language, and the scope of the integration between visual and verbal processing. We chose to examine fictive motion, a pervasive form of figurative language in English and other languages. The sentence The road runs along the coast is figurative because it contains a motion verb but describes no motion (Talmy, 2000). On the surface, fictive motion (FM) descriptions are equivalent to literal spatial descriptions (non-FM) such as The road is next to the coast. Evidence from reading times, temporal judgments, and drawing studies suggests that FM descriptions engage motion representations (Matlock, 2004; Matlock, Ramscar, & Boroditsky, 2005). Given this, how would comprehending FM descriptions interact with visual processing?

Matlock and Richardson (2004) presented participants with simple drawings of paths (linear objects) such as roads, rivers and pipelines and tracked their gaze. The same scene was shown to participants as they heard either an FM or non-FM description of the path, counter balanced between participants. The FM and non-FM sentences were of equivalent length, and were judged by an independent set of participants to have equivalent meaning. Watch eye movement recordings of a subject looking at a scene and hearing a non-fictive motion description, and a fictive motion description.

fictive motion results
As the figure here shows, FM descriptions caused participants to have a longer gaze duration within the region of the path. One could argue, of course, that FM descriptions are simply more interesting forms of speech, and caused participants to be generally more interested in the pictures in front of them. On the contrary, later work found evidence that FM sentences specifically evoke representations of motion.

Reading time studies (Matlock 2004) found that participants were quicker to process fictive motion target sentences after reading about terrains that were easy to traverse (e.g., The valley was flat and smooth) versus terrains that were not (e.g., The valley was bumpy and uneven). Critically, there was no difference for comparable literal target sentences without fictive motion (e.g., The road is in the valley). Following this logic, we presented participants with a picture and descriptions of easy or difficult terrains and then FM sentences or non-FM sentences (Richardson & Matlock, 2007). Terrain information modulated looking behavior with FM sentences, but not non-FM sentences. Specifically, difficult terrain information and FM sentences produced longer gaze durations within the region of the path, and more saccades between points along the path.

In these experiments, all we manipulated was the presence of figurative language, a change that did not alter the literal meaning or truth conditions of the sentence. Nevertheless this change appeared to alter visual processing. We argue that eye movements were affected because fictive motion language evokes a dynamic mental simulation which interacts with the ways in which the visual system interprets and inspects the world. Our findings, which have consequences for both the linguistic accounts of figurative language and the scope of top-down influences in visual perception, help illuminate the ways in which verbal and visual processes are intertwined.