How do we know where things are?
2. Eye movements, attention pointers and visual constancy

One of the challenges of vision is to keep track of the objects in front of us each time we make an eye movement. We have focused on tying this remarkable ability to the functions of the attention system. Each eye movement shifts the entire image on the retina and brings the incoming patterns to a completely new set of receptive fields. Following Bob Wurtz, among others, we have proposed that the remapping of “attention pointers” is a plausible solution to the problem of visual constancy (Rolfs et al. Nature Neuroscience, 2011): the post-saccadic locations of a few attended items are predicted from their current locations and the expected eye movement vector (Cavanagh et al, TiCS 2010).

The problem is that the attention and saccade maps are principally in retinotopic coordinates (or more accurately, oculocentric) and so every time the eyes move, the activity on these maps no longer lines up with the new target positions. According to numerous physiological studies of cells in saccade centers, the oculomotor system shifts or “remaps” the activations that code target locations to their predicted postsaccadic location, often even before the saccade happens (Duhamel et al., 1992, Gottlieb et al., 1998; Sommer & Wurtz, 2006; review Wurtz, 2008). Conveniently, this shifting response updates the saccade map to keep track of the locations the targets will have after the saccade. These results show that, even though saccade maps are principally retinotopic, they represent at times, not just the current retinal position, but also the predicted next position.

Here is movie with an example of the updating of expected location to compensate for the saccade. Synchronize your left and right eye movements to follow the changing colors of the fixations so that you are always fixating, say, the red dot. When you are synchronized, notice what is happening with the up and down motion of the black dot — without looking at it. You may find that it appears to move, as it really does, mostly up and down, even though its motion on the retina is almost horizontal. Its location has been corrected for the movements of the eyes — the motion does not appear perfectly vertical though, there is a slight overcompensation for the eye movement for most observers. See more at Szinte and Cavanagh (2011).

The dual location representation just before the saccade is critical. In the Figure below, a saccade is being prepared to the man's face on the right. Just before the saccade, an image is flashed in the upper right quadrant but that new, salient stimulus will be in the upper left after the saccade and the predictive remapping therefore activates the expected location — moves the attention pointer for the truck to the upper left at the time of the saccade or even a bit before. The predicted position, like the current location, is represented on the retinotopic coordinates of the saccade map, but it is the position where the target will be next. If the target-related activity on the saccade map generates attentional benefits (Awh et al, 2006), then shifting the saccade target to the predicted next location should also shift attention benefits.

With Martin Rolfs, Donatis Jonikaitis, and Heiner Deubel (Nature Neuroscience, 2011), we found exactly this: attentional benefits at the remapped target location (as well as its current retinal location) prior to the saccade.

Amelia Hunt and I took this further with masking experiments where a brief target and then a mask are presented immediately before a saccade (Hunt & Cavanagh, 2011). If the mask is at the same location as the target, a condition that normally produces severe masking, the upcoming saccade reduces masking significantly suggesting that the locus of attention for the target has moved away, to the predicted next location, before picking up the mask. In contrast, if the mask is presented at the target’s remapped location, a location with no overlap of with the target, masking is now seen even though none at all occurs without the saccade. In this case, remapping moves attention to a position where the target is expected, but picks up the mask instead.

With Tomas Knapen, Jasha Swisher, andFrank Tong, we studied the remapping activity with fMRI the remapped activity (Knapen et al, 2016). We showed evidence that the response at the remapped location was more than just a simple peak of activity, but instead takes on a shape that corresponds to the target’s shape. We used a saccade target that was a flickering, precessing wedge shape that disappeared every time the subject directed a saccade to it. We found that activation was remapped to the fovea and the activation reflected the wedge shape of the target itself. We are intrigued by this result because a shaped activation on the saccade map would, through downward projections (Awh et al., 2006), confer attentional benefits throughout the target area. This would be a mechanism to implement the characteristics of object-based attention (Duncan, 1984).

Awh, E., Armstrong, K.M., and Moore, T. (2006). Visual and oculomotor selection: links, causes and implications for spatial attention. Trends Cogn. Sci. 10, 124–130.
Duhamel, J.R., Colby, C. L., & Goldberg, M. E. (1992) The updating of the representation of visual space in parietal cortex by intended eye movements. Science, 255, 90-92
Duncan, J. (1984) Selective attention and the organization of visual information. J Exp Psychol Gen 113, 501-17
Gottlieb, J.P., Kusunoki, M., and Goldberg, M.E. (1998) The representation of visual salience in monkey parietal cortex. Nature, 391, 481-484
Sommer, M.A., and Wurtz, R.H. (2006). Influence of the thalamus on spatial visual processing in frontal cortex. Nature, 444, 374-377
Wurtz, R.H. (2008) Neuronal mechanisms of visual stability. Vision Res, 48, 2070-2089.