By the middle of the twentieth century, thinkers began to appreciate that we know ourselves very little. We are not at the center of ourselves, but instead—like the Earth in the Milky Way, and the Milky Way in the universe—far out on a distant edge, hearing little of what is transpiring.
* * *
Freud’s intuition about the unconscious brain was spot-on, but he lived decades before the modern blossoming of neuroscience. We can now peer into the human cranium at many levels, from electrical spikes in single cells to patterns of activation that traverse the vast territories of the brain. Our modern technology has shaped and focused our picture of the inner cosmos, and in the following chapters we will travel together into its unexpected territories.
How is it possible to get angry at yourself: who, exactly, is mad at whom? Why do rocks appear to climb upward after you stare at a waterfall? Why did Supreme Court Justice William Douglas claim that he was able to play football and go hiking, when everyone could see that he was paralyzed after a stroke? Why was Topsy the elephant electrocuted by Thomas Edison in 1916? Why do people love to store their money in Christmas accounts that earn no interest? If the drunk Mel Gibson is an anti-Semite and the sober Mel Gibson is authentically apologetic, is there a real Mel Gibson? What do Ulysses and the subprime mortgage meltdown have in common? Why do strippers make more money at certain times of month? Why are people whose name begins with J more likely to marry other people whose name begins with J? Why are we so tempted to tell a secret? Are some marriage partners more likely to cheat? Why do patients on Parkinson’s medications become compulsive gamblers? Why did Charles Whitman, a high-IQ bank teller and former Eagle Scout, suddenly decide to shoot forty-eight people from the University of Texas Tower in Austin?
What does all this have to do with the behind-the-scenes operations of the brain?
As we are about to see, everything.
The Testimony of the Senses: What Is Experience Really Like?
DECONSTRUCTING EXPERIENCE
One afternoon in the late 1800s, the physicist and philosopher Ernst Mach took a careful look at some uniformly colored strips of paper placed next to each other. Being interested in questions of perception, he was given pause by something: the strips did not look quite right. Something was amiss. He separated the strips, looked at them individually, and then put them back together. He finally realized what was going on: although each strip in isolation was uniform in color, when they were placed side by side each appeared to have a gradient of shading: slightly lighter on the left side, and slightly darker on the right. (To prove to yourself that each strip in the figure is in fact uniform in brightness, cover up all but one.)1
Mach bands.
Now that you are aware of this illusion of “Mach bands,” you’ll notice it elsewhere—for example, at the corner where two walls meet, the lighting differences often make it appear that the paint is lighter or darker right next to the corner. Presumably, even though the perceptual fact was in front of you this entire time, you have missed it until now. In the same way, Renaissance painters noticed at some point that distant mountains appeared to be tinted a bit blue—and once this was called out, they began to paint them that way. But the entire history of art up to that point had missed it entirely, even though the data was unhidden in front of them. Why do we fail to perceive these obvious things? Are we really such poor observers of our own experiences?
Yes. We are astoundingly poor observers. And our introspection is useless on these issues: we believe we’re seeing the world just fine until it’s called to our attention that we’re not. We will go through a process of learning to observe our experience, just as Mach carefully observed the shading of the strips. What is our conscious experience really like, and what is it not like?
* * *
Intuition suggests that you open your eyes and voilà: there’s the world, with all its beautiful reds and golds, dogs and taxicabs, bustling cities and floriferous landscapes. Vision appears effortless and, with minor exceptions, accurate. There is little important difference, it might seem, between your eyes and a high-resolution digital video camera. For that matter, your ears seem like compact microphones that accurately record the sounds of the world, and your fingertips appear to detect the three-dimensional shape of objects in the outside world. What intuition suggests is dead wrong. So let’s see what’s really happening.
Consider what happens when you move your arm. Your brain depends on thousands of nerve fibers registering states of contraction and stretching—and yet you perceive no hint of that lightning storm of neural activity. You are simply aware that your limb moved and that it is somewhere else now. Sir Charles Sherrington, an early neuroscience pioneer, spent some time fretting about this fact during the middle of the last century. He was awestruck by the lack of awareness about the vast mechanics under the surface. After all, despite his considerable expertise with nerves, muscles, and tendons, he noted that when he went to pick up a piece of paper, “I have no awareness of the muscles as such at all.… I execute the movement rightly and without difficulty.”2 He reasoned that if he were not a neuroscientist it would not have occurred to him to suspect the existence of nerves, muscles, and tendons. This intrigued Sherrington, and he finally inferred that his experience of moving his arm was “a mental product.… derived from elements which are not experienced as such and yet … the mind uses them in producing the percept.” In other words, the storm of nerve and muscle activity is registered by the brain, but what is served up to your awareness is something quite different.
To understand this, let’s return to the framework of consciousness as a national newspaper. The job of a headline is to give a tightly compressed summary. In the same manner, consciousness is a way of projecting all the activity in your nervous system into a simpler form. The billions of specialized mechanisms operate below the radar—some collecting sensory data, some sending out motor programs, and the majority doing the main tasks of the neural workforce: combining information, making predictions about what is coming next, making decisions about what to do now. In the face of this complexity, consciousness gives you a summary that is useful for the larger picture, useful at the scale of apples and rivers and humans with whom you might be able to mate.
OPENING YOUR EYES
The act of “seeing” appears so natural that it is difficult to appreciate the vastly sophisticated machinery underlying the process. It may come as a surprise that about one-third of the human brain is devoted to vision. The brain has to perform an enormous amount of work to unambiguously interpret the billions of photons streaming into the eyes. Strictly speaking, all visual scenes are ambiguous: for example, the image to the right can be caused by the Tower of Pisa at a distance of five hundred yards, or a toy model of the tower at arm’s length: both cast the identical image on your eyes. Your brain goes through a good deal of trouble to disambiguate the information hitting your eyes by taking context into account, making assumptions, and using tricks that we’ll learn about in a moment. But all this doesn’t happen effortlessly, as demonstrated by patients who surgically recover their eyesight after decades of blindness: they do not suddenly see the world, but instead must learn to see again.3 At first the world is a buzzing, jangling barrage of shapes and colors, and even when the optics of their eyes are perfectly functional, their brain must learn how to interpret the data coming in.
For those of us with a lifetime of sight, the best way to appreciate the fact that vision is a construction is by noticing how often our visual systems get it wrong. Visual illusions exist at the edges of what our system has evolved to handle, and as such they serve as a powerful window into the brain.4
There is some difficulty in rigorously defining “illusion,” as there is a sense in which all of vision is an illusion. The resolution in your peripheral vision is roughly equivalent to looking through a frosted shower door, and yet you enjoy the illusion of seeing the periphery clearly. This is because everywhere you aim your central vision appears
to be in sharp focus. To drive this point home, try this demonstration: have a friend hold a handful of colored markers or highlighters out to his side. Keep your gaze fixed on his nose, and now try to name the order of the colors in his hand. The results are surprising: even if you’re able to report that there are some colors in your periphery, you won’t be able to accurately determine their order. Your peripheral vision is far worse than you would have ever intuited, because under typical circumstances your brain leverages the eye muscles to point your high-resolution central vision directly toward the things you’re interested in. Wherever you cast your eyes appears to be in sharp focus, and therefore you assume the whole visual world is in focus.*
That’s just the beginning. Consider the fact that we are not aware of the boundaries of our visual field. Stare at a point on the wall directly in front of you, stretch your arm out, and wiggle your fingers. Now move your hand slowly back toward your ear. At some point you can no longer see your fingers. Now move it forward again and you can see them. You’re crossing the edge of your visual field. Again, because you can always aim your eyes wherever you’re interested, you’re normally not the least bit aware that there are boundaries beyond which you have no vision. It is interesting to consider that the majority of human beings live their whole lives unaware that they are only seeing a limited cone of vision at any moment.
As we dive further into vision, it becomes clear that your brain can serve up totally convincing perceptions if you simply put the right keys in the right locks. Take the perception of depth. Your two eyes are set a few inches apart, and as a result they receive slightly different images of the world. Demonstrate this to yourself by taking two photographs from a few inches apart, and then putting them side by side. Now cross your eyes so that the two photos merge into a third, and a picture will emerge in depth. You will genuinely experience the depth; you can’t shake the perception. The impossible notion of depth arising from a flat image divulges the mechanical, automatic nature of the computations in the visual system: feed it the right inputs and it will construct a rich world for you.
Cross your eyes: the two images feed your brain the illusory signal of depth.
One of the most pervasive mistakes is to believe that our visual system gives a faithful representation of what is “out there” in the same way that a movie camera would. Some simple demonstrations can quickly disabuse you of this notion. In the figure below, two pictures are shown.
Change blindness.
What is the difference between them? Difficult to tell, isn’t it? In a dynamic version of this test, the two images are alternated (say, each image shown for half a second, with a tenth of a second blank period in between). And it turns out we are blind to shockingly large changes in the scene. A large box might be present in one photo and not the other, or a jeep, or an airplane engine—and the difference goes unseen. Our attention slowly crawls the scene, analyzing interesting landmarks until it finally detects what is changing.** Once the brain has latched onto the appropriate object, the change is easy to see—but this happens only after exhaustive inspection. This “change blindness” highlights the importance of attention: to see an object change, you must attend to it.5
You are not seeing the world in the rich detail that you implicitly believed you were; in fact, you are not aware of most of what hits your eyes. Imagine you’re watching a short film with a single actor in it. He is cooking an omelet. The camera cuts to a different angle as the actor continues his cooking. Surely you would notice if the actor changed into a different person, right? Two-thirds of observers don’t.6
In one astonishing demonstration of change blindness, random pedestrians in a courtyard were stopped by an experimenter and asked for directions. At some point, as the unsuspecting subject was in the middle of explaining the directions, workmen carrying a door walked rudely right between the two people. Unbeknownst to the subject, the experimenter was stealthily replaced by a confederate who had been hiding behind the door as it was carried: after the door passed, a new person was standing there. The majority of subjects continued giving directions without noticing that the person was not the same as the original one they were talking with.7 In other words, they were only encoding small amounts of the information hitting their eyes. The rest was assumption.
Neuroscientists weren’t the first to discover that placing your eyes on something is no guarantee of seeing it. Magicians figured this out long ago, and perfected ways of leveraging this knowledge.8 By directing your attention, magicians perform sleight of hand in full view. Their actions should give away the game—but they can rest assured that your brain processes only small bits of the visual scene, not everything that hits your retinas.
This fact helps to explain the colossal number of traffic accidents in which drivers hit pedestrians in plain view, collide with cars directly in front of them, and even intersect unluckily with trains. In many of these cases, the eyes are in the right place, but the brain isn’t seeing the stimuli. Vision is more than looking. This also explains why you probably missed the fact that the word “of” is printed twice in the triangle above.
The lessons here are simple, but they are not obvious, even to brain scientists. For decades, vision researchers barked up the wrong tree by trying to figure out how the visual brain reconstructed a full three-dimensional representation of the outside world. Only slowly did it become clear that the brain doesn’t actually use a 3-D model—instead, it builds up something like a 2½-D sketch at best.9 The brain doesn’t need a full model of the world because it merely needs to figure out, on the fly, where to look, and when.10 For example, your brain doesn’t need to encode all the details of the coffee shop you’re in; it only needs to know how and where to search when it wants something in particular. Your internal model has some general idea that you’re in a coffee shop, that there are people to your left, a wall to your right, and that there are several items on the table. When your partner asks, “How many lumps of sugar are left?” your attentional systems interrogate the details of the bowl, assimilating new data into your internal model. Even though the sugar bowl has been in your visual field the entire time, there was no real detail there for your brain. It needed to do extra work to fill in the finer points of the picture.
Similarly, we often know one feature about a stimulus while simultaneously being unable to answer others. Say I were to ask you to look at the following and tell me what it is composed of: ||||||||||||. You would correctly tell me it is composed of vertical lines. If I were to ask you how many lines, however, you would be stuck for a while. You can see that there are lines, but you cannot tell me how many without considerable effort. You can know some things about a scene without knowing other aspects of it, and you become aware of what you’re missing only when you’re asked the question.
What is the position of your tongue in your mouth? Once you are asked the question you can answer it—but presumably you were not aware of the answer until you asked yourself. The brain generally does not need to know most things; it merely knows how to go out and retrieve the data. It computes on a need-to-know basis. You do not continuously track the position of your tongue in consciousness, because that knowledge is useful only in rare circumstances.
In fact, we are not conscious of much of anything until we ask ourselves about it. What does your left shoe feel like on your foot right now? What pitch is the hum of the air conditioner in the background? As we saw with change blindness, we are unaware of most of what should be obvious to our senses; it is only after deploying our attentional resources onto small bits of the scene that we become aware of what we were missing. Before we engage our concentration, we are typically not aware that we are not aware of those details. So not only is our perception of the world a construction that does not accurately represent the outside, but we additionally have the false impression of a full, rich picture when in fact we see only what we need to know, and no more.
The manner in which the brain interrogates the world t
o gather more details was investigated in 1967 by the Russian psychologist Alfred Yarbus. He measured the exact locations that people were looking at by using an eye tracker, and asked his subjects to gaze at Ilya Repin’s painting An Unexpected Visitor (below).11 The subjects’ task was simple: examine the painting. Or, in a different condition, surmise what the people in the painting had been doing just before the “unexpected visitor” came in. Or answer a question about how wealthy the people were. Or their ages. Or how long the unexpected visitor had been away.
Six records of eye movements from the same subject. Each record lasted three minutes.
1) Free examination. Before subsequent recordings, the subject was asked to: 2) estimate the material circumstances of the family;
3) give the ages of the people; 4) surmise what the family had been doing before the arrival of the “unexpected visitor”;
5) remember the clothes worn by the people; 6) estimate how long the “unexpected visitor” had been away from the family. From Yarbus, 1967.
The results were remarkable. Depending on what was being asked, the eyes moved in totally different patterns, sampling the picture in a manner that was maximally informative for the question at hand. When asked about the ages of the people, the eyes went to the faces. When asked about their wealth, the focus danced around the clothes and material possessions.