Chapter 9 Two brains or one?
A human brain has two halves, a left and a right, that are anatomically connected, but there are fewer cross-hemisphere connections than there are within-hemisphere connections. Much of sensory and perceptual processing runs rather independently in the two halves of the cortex, but more cognitive functions such as declarative memory benefit from a tight integration. This integration is extensive enough that the comparison of our two hemispheres to our two hands or our two legs is misleading.
Our conscious experience, too, is quite unified. We experience no discontinuity when the movement of our eyes, or of an object, cause an object to shift from one hemifield, where it is processed predominantly by one hemisphere, to the other hemifield. Communication between the two hemispheres happens rapidly and continuously. Contrary to the claims of those who prey on well-meaning schools and parents, there is no good evidence that exercises designed to insure both hemispheres process stimuli have any benefit for learning.
In “split-brain” patients, many of the connections between the hemispheres have been lost. Such patients can still perform tasks such as visual search in both hemifields, suggesting that both hemispheres have the mechanisms needed to do the task. When split-brain patients are asked to search for a target among many distractor objects, spreading the load by distributing the distractors across the two hemifields can yield a large benefit, suggesting that the two hemispheres in these patients carry out their searches independently (Steven J. Luck et al. 1994). For intact individuals, no such advantage is seen, suggesting that in a normal brain, the processes that evaluate each stimulus for whether it is the target are integrated across the hemispheres into a single attentional focus (S. J. Luck et al. 1989).
Although the two hemispheres work as one during many tasks, each hemisphere does specialize in certain functions. The left hemisphere has greater competence in language functions such as reading, while the right hemisphere is better at recognizing faces. One behavioral consequence is that response times for a face recognition task are slightly faster when the stimulus is presented wholly in the left hemifield (to the right hemisphere) than when it is presented wholly in the right hemifield, whereas the opposite is found for word reading (Rizzolatti, Umiltà, and Berlucchi 1971). With extended time to process a stimulus, however, such behavioral asymmetries can disappear, because eventually the information from one hemisphere gets communicated to the other.
From the performance of most perceptual and attentional tasks, then, in typically-developing humans there is little overt sign that the brain is divided into two halves. Multiple object tracking, however, is a major exception to this. The pattern of performance found indicates that the limited resource that determines how many objects one can keep track of resides largely with processing that operates independently in the two hemispheres.
9.1 The extraordinary hemifield independence of object tracking
In 2005, George Alvarez & Patrick Cavanagh reported a stunning MOT finding. They used objects that resembled spinning pinwheels, and they designated individual bars of the different pinwheels as targets. Performance in a two-target condition was contrasted with a one-target condition (G. A. Alvarez and Cavanagh 2005). When the second target was positioned in the same hemifield as the first target, accuracy in the two-target condition was much worse than in the one-target condition (89% vs. 63%). Remarkably, however, when the second target belonged to a pinwheel located in the opposite hemifield, there was very little performance decrement - accuracy was 93% in the one-target condition, and 90% correct in the two-target condition. This suggests that the processes that limit successful tracking in this task are specific to each hemifield.
It was already known that sensory processing and quite a lot of perceptual processing occurs independently in each hemisphere. What is interesting here is that a higher-level, limited-capacity process would be hemisphere-independent. Such capacities were traditionally thought to be among the processes that are tightly integrated across the two hemispheres, forming a single resource “pool”, not two independent limits. We will get back to this point, but first we’ll examine more extensively the evidence for hemispheric independence of object tracking.
9.2 Quantitative estimates of independence
The hemispheric independence of a task, such as MOT, can be quantified. Imagine that adding a second stimulus to a hemifield reduces performance by 20 percentage points, but adding that stimulus to the other hemifield reduces performance by only 5 percentage points. One can quantify the hemispheric independence, then, as (20-5) / 20 = 75% hemifield independence. Ideally, however, one would not use raw accuracy but instead would correct for the accuracy one can achieve by guessing. When applying such a calculation to the G. A. Alvarez and Cavanagh (2005) results, the estimated level of independence is very high: 88% independence in one of their experiments, and 92% in the other.
G. A. Alvarez and Cavanagh (2005) themselves, like others who have studies this issue, did not do these calculations. G. A. Alvarez and Cavanagh (2005) calculated expected performance if the hemifields are in fact completely independent, and reported that performance was not statistically significantly worse than that figure, and suggested on that basis that there is complete hemifield independence. That, however, is the fallacy of concluding a null hypothesis is true when the evidence does not reject it at a p<.05 level (Aczel et al. 2018). This is not uncommon in the scientific literature - setting the null hypothesis to the desired conclusion (complete independence), and then affirming this conclusion on the basis of not finding much evidence against it. Nevertheless, the data of G. A. Alvarez and Cavanagh (2005) do suggest (with unknown confidence, because the uncertainty was not quantified) a hemispheric independence level of approximately 90%. In a study with similar methods, Hudson, Howe, and Little (2012) found 65% independence (this is my calculation).
Some of the follow-up studies in this area have not included enough conditions to quantify the degree of independence, or confounded distribution of the targets to two hemifields with greater distance among them, such that any benefit might have been due to less spatial crowding interference, a phenomenon discussed in Chapter 5. Alex O. Holcombe and Chen (2012) and Chen, Howe, and Holcombe (2013), however, also found evidence for a high degree of independence, using a slightly different approach based on speed thresholds. The findings were compatible with 90-100% hemifield independence or a bit less. Won Mok Shim et al. (2010) and Störmer, Alvarez, and Cavanagh (2014) also found evidence for a substantial bilateral advantage compared to having two targets in the same hemifield.
The findings of hemispheric independence have not replicated in all circumstances (e.g., W. M. Shim, Alvarez, and Jiang 2008) , but the balance of published evidence strongly suggests that at least in some circumstances, tracking does occur mostly independently in the two hemispheres. I say “mostly independently” rather than suggesting complete independence because each individual study has too much statistical uncertainty to rule out a figure such as 75% independence, even for the point estimates I’ve reported above that suggest a higher degree of independence.
W. M. Shim, Alvarez, and Jiang (2008) suggested that the reason they did not find evidence for hemifield independence is that they used only two targets; according to them, the original G. A. Alvarez and Cavanagh (2005) report of hemifield independence used four targets. This is unlikely to be the reason for the discrepancy, however, because in their E1 and E2 G. A. Alvarez and Cavanagh (2005) did find evidence for hemifield independence using just two targets, as did Alex O. Holcombe and Chen (2012) and Störmer, Alvarez, and Cavanagh (2014). And the W. M. Shim, Alvarez, and Jiang (2008) data may have been afflicted by a ceiling effect, as accuracy was over 85% correct in all conditions in their experiment.
A limitation of deriving hemispheric independence from accuracy is that it depends on the assumption that if a person can only track one target, in a condition where the person is also trying to track a second target the person will succeed just as often in tracking one of the two. My introspective experience, however, indicates that in some circumstances, trying to track both targets causes one to fail at both, and thus one is better off only trying to track one. A particular threshold amount of resource is needed to track a target, and so if neither target is allocated that much resource, tracking will fail for both. Evidence for this was provided by Chen, Howe, and Holcombe (2013); in the terminology of the Norman and Bobrow (1975) framework introduced in Chapter 4, this would be described by saying that the resource function that relates attentional resource proportion to accuracy falls below a straight line. One implication is that quantitative estimates of hemispheric independence will be overestimates, particularly in circumstances where the participants do not realize they may be better off focusing their efforts on tracking fewer targets than the number they have been told to track .
Carlson, Alvarez, and Cavanagh (2007) found evidence not only for hemifield independence but also quadrant-level independence, which they attributed to the partial anatomical separation of the retinotopic quadrant representations in areas V2 and V3. Using different stimuli, W. M. Shim, Alvarez, and Jiang (2008) and Alex O. Holcombe, Chen, and Howe (2014) did not, however, find evidence for quadrantic independence. More work on this topic is needed.
9.3 Some tracking resources are NOT hemifield-specific
One attentional process that is not hemifield-specific is feature attention, for example attention to color. When a participant is told to look for a red target, they are able to use feature attention to enhance all red objects, no matter where they are in the visual field (Alex L. White and Carrasco 2011). The decision to look for red originates with cognitive processes and remains hemifield-unified rather than hemifield-specific at the level of visual cortex (Saenz, Buracas, and Boynton 2002). Indeed, people seem to be unable to confine the enhancement of red objects to one hemifield (Lo and Holcombe 2014). In real-world tracking where objects are at least somewhat heterogeneous and thus targets often have a different average color and other features than distractors, feature attention will facilitate tracking, and this facilitation is not hemifield-specific.
A previous Chapter (6) introduced the idea of C=1 cognitive processes that can support tracking of a single target but perhaps not multiple targets. Such processing likely is not hemisphere-specific, being aligned with “central executive” processes that integrate processing in both hemispheres.
Chen, Howe, and Holcombe (2013) found evidence for both hemifield-specific tracking processes and also processes not specific to a hemifield, operating in the same MOT task. Two targets were used, and on some trials they moved at different speeds. When a slow-moving target was paired (presented in the same trial) with a speedier target, accuracy was lower for the slow-moving target than if it was paired with a target that was slower. This suggests that participants allocate more tracking resources to the faster of two targets, presumably because slower targets do not require much resource to track well. This trade-off was most pronounced when the two targets were in the same hemifield, but seemed to occur to some degree even when the two targets were in different hemifields, implicating a cross-hemifield resource that plays a small role. This cross-hemifield resource may be a C=1 process. Furthermore, as discussed in the next section, perturbing one parietal lobe can affect performance in both hemifields, which suggests that each hemisphere can in some circumstances mediate tracking in either hemifield.
9.4 The underlying mechanisms
The evidence reviewed above for hemifield independence suggests that hemisphere-specific processes determine how many targets one can track. This raises the question of what sort of processes those are, and how they interact with the cognitive processes that are more integrated across the hemispheres.
Steve Franconeri and colleagues have championed the idea that the hemisphere independence stems from spatial interference processes; they suggested that spatial interference occurs largely within a hemisphere (Steven L. Franconeri 2013). The idea is that when when an object is tracked, the neurons representing that target in retinotopic cortical areas activate inhibitory connections to nearby neurons, suppressing the responses to neighboring objects. To explain the findings of hemifield specificity, an important detail of the account is that the inhibitory neural connections do not extend from one hemisphere’s retinotopic map to another (Steven L. Franconeri, Alvarez, and Cavanagh 2013b). This is plausible because in classic crowding tasks, spatial interference does show a discontinuity across the left- and right-visual field boundary (T. Liu et al. 2009). However, Alex O. Holcombe, Chen, and Howe (2014) found evidence against spatial interference extending any further than the classic crowding range of half the eccentricity of an object (for which an object placed six degrees of visual angle from where the point the eyes are looking at would be interfered with only by other objects closer to it than three degrees of visual angle (Bouma 1970)). Because in most studies of hemifield independence, the stimuli are not close to the vertical midline, any modulation of the crowding range by the vertical midline would not yield apparent evidence of hemifield independence, contrary to the hypothesis of Steven L. Franconeri, Alvarez, and Cavanagh (2013b). The more viable theory of hemifield independence, then, is that of two neural resources that span each hemifield.
Consistent with a putative pool of attentional resources, a number of studies have found that the activity of some parietal and frontal areas of cortex increase steadily with the number of targets in MOT (Culham, Cavanagh, and Kanwisher 2001; Piers D. Howe et al. 2009; Jovicich et al. 2001; Alnaes et al. 2014; Nummenmaa et al. 2017). Unfortunately, these studies provided little information about whether these activations are specific to target load within a hemifield, so we cannot be sure whether the brain activation measured reflects the hemifield-specific resource or a more global resource. The only imaging study I am aware of that investigated the issue is Won Mok Shim et al. (2010), who did find an activation difference when the objects designated as targets were in opposite hemifields compared to when they were in the same hemifield. The activation difference was found for the superior parietal lobule and transverse parieto-occipital area, suggesting that they may be part of the hemifield-specific resource. The difference was not found for the anterior intraparietal sulcus, which could mean its activation reflects a global resource .
Störmer, Alvarez, and Cavanagh (2014) used electroencephalography (EEG) to investigate the hemifield-specific resource. They found that SSVEP activation for targets was higher than that for distractors, especially when the two targets were positioned in different (left and right) hemifields. In contrast, an event-related potential (ERP) component known as the P3 thought to reflect more cognitive identification and decision processes was similar in the two conditions. This is consistent with the theory that tracking depends on hemisphere-specific attentive processing followed by some involvement of higher-order processes that are not hemisphere-specific.
Battelli et al. (2009) found they could disrupt MOT performance in a hemifield by stimulating the contralateral intraparietal sulcus (IPS) via repetitive transcranial magnetic stimulation. Importantly, this only occurred when the moving targets were present in both hemifields. When the targets were all in the left or all in the right hemifield, TMS to the left or to the right IPS had no effect on tracking accuracy, a result that replicated in a second experiment. This phenomenon is reminiscent of the inter-hemifield competition evident in “extinction”, a symptom seen in parietal neglect patients.
In extinction, responding to stimuli in the hemifield contralateral to parietal injury is only impaired if there are also stimuli presented to the ipsilateral hemifield. This, together with the analogous finding from tracking and TMS, inspired Battelli to propose two things. The first is that the IPS in each hemisphere can mediate the tracking of targets in either visual hemifield. The second is that under normal conditions, inter-hemisphere inhibition reduces the ipsilateral processing of each IPS, causing tracking capacity to be effectively hemifield-specific.
Evidence from patients and MOT also suggests a complicated relationship between the hemispheres. Battelli et al. (2001) found that in patients with damage to their right parietal lobe, MOT performance in the left visual field only was impaired, as expected — the right parietal lobe does not normally mediate tracking in the right visual field, so losing it did not hurt right visual field tracking performance. For an apparent motion task, however, the right parietal patients had impairments in both hemifields. The suggested involvement of the right parietal lobe, but not the left parietal lobe, in judging apparent motion and the temporal order of stimuli in both hemifields, which was further supported by both an additional study with patients and a TMS study (Agosta et al. 2017).
In summary, while there is evidence that parietal cortex is involved in field-wide processing for some tasks, it also likely mediates the hemifield independence evident in some circumstances. Both aspects may be in operation when a target travels from one hemifield to another. Using ERP, Drew et al. (2014) found evidence that when a target crosses the vertical midline, say from the left to the right hemifield, the left hemisphere becomes involved shortly before the target reaches the right hemifield, and the right hemisphere remains involved for a short time after the crossing. Because this was modulated by predictability of the motion, it did not appear to be entirely mediated by the well-known overlap of the two hemispheres’ receptive fields at the midline. This phenomenon may reflect the normally-inhibited ipsilateral representation of the visual field by parietal cortices highlighted by Battelli et al. (2009), although this remains uncertain as the origin of the ERP signals was not clear.
Consistent with tracking being mediated largely by the contralateral hemisphere, both Strong and Alvarez (2020) and Minami, Shinkai, and Nakauchi (2019) found evidence for a tracking performance cost when a target in MOT crossed the vertical midline. In a memory paradigm, too, Jun Saiki (2019) found evidence that when two objects moved between hemifields, memory for their features was more disrupted than when they moved from one to another quadrant within the same hemifield. Similarly, Strong and Alvarez (2020) found no cost when targets moved between quadrants while remaining within a hemifield, an important finding given that other work raised the prospect of quadrant-specific resources (Carlson, Alvarez, and Cavanagh 2007).
In summary, areas of parietal cortex likely subserve the hemifield-specific tracking resource that determines MOT capacity, but also may provide a resource that is not hemifield-specific. As we will see in the next Section (10), the hemifield-wide process may be responsible for feature updating and binding.
9.5 What else are hemifield-specific resources used for?
Multiple object tracking may involve the same spatial selection process as that used to select stimuli in popular attentional tasks, which use static stimuli almost exclusively. Is spatial selection even with static stimuli, then, hemifield-specific? Decades ago, performance for static stimuli tasks had been compared when two stimuli are presented in the same hemifield to when the stimuli are presented in different hemifields. For example, Dimond and Beaumont (1971) found that reporting two briefly-presented digits is associated with higher accuracy when the digits are presented in different hemifields than in the same hemifield. However, that study and others of that era did not include a single-stimulus condition, so for the higher performance in the split condition, we don’t know how close it is to the one-target level of performance, and therefore we can’t calculate the magnitude of the different-hemifield advantage. Moreover, many studies used response time as a measure, which can be difficult to interpret quantitatively (Awh and Pashler 2000; A. B. Sereno and Kosslyn 1991; Dimond and Beaumont 1971).
For a proper assessment of hemifield specificity in a visual working memory task, J.-F. Delvenne (2005) used both dual-target and single-target conditions, and calculated 40% hemifield independence. Unfortunately, however, he used the discredited A’ measure of performance (Zhang and Mueller 2005) and did not space the stimuli widely enough to reduce the possibility of spatial interference (crowding). Nevertheless, the advantage was large and did not occur for non-spatial color working memory (J. Delvenne 2012). This latter finding is one example of a broader result that has emerged, that only tasks with spatial demands seem to show a substantial different-hemifield advantage (Holt and Delvenne 2015; Umemoto et al. 2010).
G. A. Alvarez, Gill, and Cavanagh (2012) studied visual search, with the stimuli to search arrayed bilaterally or unilaterally. In a standard search task, they found only a small advantage of the bilateral display. In a subset search task where participants knew the target would be located in one of several locations designated by a pre-trial cue, however, they found a large bilateral advantage. When the relevant locations were visually salient (due to a color difference) rather than requiring top-down selection, the bilateral advantage largely disappeared. These results, and those reviewed above, suggest that hemifield advantages are strongest when spatial selection is critical.
Finally, Strong and Alvarez (2020) investigated spatial working memory for stimuli that moved either within a hemifield or between hemifields. For between-hemifield movement, they found a substantial decrease in accuracy for remembering which positions of a 2x2 grid contained dots at the beginning of the trial, before the (empty) grid moved - 79% correct for between-hemifield movement, and 85% correct for within-hemifield movement. This between-hemifield cost for spatial memory was similar to the cost they found for MOT itself. No such cost was found for color or identity memory tasks.
The association found between spatial tasks and a different-hemifield advantage may reflect a large-scale difference in how the brain processes spatial versus identity information. Famously, the dorsal stream is more concerned with spatial information than is the ventral stream, which is more involved in object recognition (Goodale and Milner 1992). Neural responses in the dorsal pathway to parietal cortex are largely contralateral (M. I. Sereno, Pitzalis, and Martinez 2001), although as we have seen above, this may depend on having stimuli in both hemifields. Contralateral dominance is also found for other brain areas thought to contribute to a “saliency map” (Fecteau and Munoz 2006), such as the frontal eye fields (Hagler Jr and Sereno 2006), the superior colliculus (Schneider and Kastner 2005), and the pulvinar (Cotton and Smith 2007). In contrast, identity-related processing seems to involve more bilateral neural responses and connectivity between hemispheres (M. R. Cohen and Maunsell 2011; Hemond, Kanwisher, and Op de Beeck 2007).
The multiple identity tracking (MIT) task, which is discussed further in Section 10, combines the location-updating demand of multiple object tracking with an additional requirement to maintain knowledge of what features belong to each of the objects. Across four MIT experiments, Hudson, Howe, and Little (2012) consistently found partial hemifield independence for this task, ranging from 26 to 37% (my calculations are here) with a paradigm that yielded 65% independence for MOT. This is consistent with the suggestion of the findings listed above that spatial selection and/or location updating processes are much more hemisphere-specific than processes that require maintenance of non-spatial features.
Putting it all together, spatial selection appears to occur at a hemifield-specific stage, with other features subsequently updated and linked in at a field-wide stage.
9.6 Hemispheric differences
So MOT and spatial selection seem to be limited by processing that happens concurrently and independently in the two hemispheres. Are the two hemispheres doing exactly the same thing?
Any functional differences between the left and right cerebral hemispheres if often attenuated at the behavioral level by the cross-hemisphere integration that can occur. But tracking processes, or at least the processes that underlie tracking capacity, work independently in the two hemispheres, and thus there is a higher potential to show hemifield differences than for other tasks. As it turns out, however, while some differences have been observed, they do not seem to be large.
In each of four experiments conducted for a 2014 paper, my colleagues and I found either a trend for or a statistically significant advantage for targets in the right hemifield (Figure A2) (Alex O. Holcombe, Chen, and Howe 2014). This was also found by Strong and Alvarez (2020). Interestingly, there was some non-significant evidence that this was greater in their one-target condition than in their two-target condition. Figure 4 of Battelli et al. (2001), like Holcombe et al. (2014), shows better tracking performance in the RVF than the left only when tracking one target, although this finding was also not statistically significant.
The right hemifield advantage, if it replicates, could be explained by the idea that stimuli presented to the right hemifield are processed by both hemispheres to a greater degree than are stimuli presented to the left hemifield. This is a leading explanation of why left neglect is more common than right neglect — the right hemisphere is thought to mediate attention to both hemifields (Mesulam 1999), such that the right hemifield is doubly-processed. However, while Strong and Alvarez (2020) did find a right hemifield advantage in their MOT experiments, they found a left visual field advantage for spatial working memory experiments, even though spatial working memory is also thought to be mediated by parietal cortex. Most strikingly, Matthews and Welch (2015) found a large advantage for temporal order judgments and simultaneity judgments for stimuli presented to the left hemifield, which may reflect specialization by the right parietal cortex (Battelli et al. 2003).
The situation becomes more complex when one considers that subtle interactions between the two hemispheres seem to affect attention in each hemifield, as highlighted above in “The Underlying Mechanisms”. A recent finding by Edwards, Berestova, and Battelli (2021) illustrates this. Participants were trained on MOT in a hemifield for 30 min, and afterwards there was little to no accuracy change in the trained hemifield, but a significant performance improvement was found in the untrained hemifield. The reason for this is not clear, but could reflect “fatigue” by the hemisphere contralateral to the trained hemifield and an associated reduction of its inhibition of the other hemisphere. Alternatively, the mechanism could be potentiation (an increase in gain) of the untrained hemisphere as a result of the deprivation, which may be the reason why depriving an eye results in increased cortical activity when that eye is stimulated later (Lunghi, Burr, and Morrone 2011).