Think about what this means: brains reach out into the world and actively extract the type of information they need. The brain does not need to see everything at once about An Unexpected Visitor, and it does not need to store everything internally; it only needs to know where to go to find the information. As your eyes interrogate the world, they are like agents on a mission, optimizing their strategy for the data. Even though they are “your” eyes, you have little idea what duty they’re on. Like a black ops mission, the eyes operate below the radar, too fast for your clunky consciousness to keep up with.
For a powerful illustration of the limits of introspection, consider the eye movements you are making right now while reading this book. Your eyes are jumping from spot to spot. To appreciate how rapid, deliberate, and precise these eye movements are, just observe someone else while they read. Yet we have no awareness of this active examination of the page. Instead it seems as though ideas simply flow into the head from a stable world.
* * *
Because vision appears so effortless, we are like fish challenged to understand water: since the fish has never experienced anything else, it is almost impossible for it to see or conceive of the water. But a bubble rising past the inquisitive fish can offer a critical clue. Like bubbles, visual illusions can call our attention to what we normally take for granted—and in this way they are critical tools for understanding the mechanisms running behind the scenes in the brain.
You’ve doubtless seen a drawing of a cube like the one to the right. This cube is an example of a “multistable” stimulus—that is, an image that flips back and forth between different perceptions. Pick what you perceive as the “front” face of the cube. Staring at the picture for a moment, you’ll notice that sometimes the front face appears to become the back face, and the orientation of the cube changes. If you keep watching, it will switch back again, alternating between these two perceptions of the cube’s orientation. There’s a striking point here: nothing has changed on the page, so the change has to be taking place in your brain. Vision is active, not passive. There is more than one way for the visual system to interpret the stimulus, and so it flips back and forth between the possibilities. The same manner of reversals can be seen in the face–vase illusion below: sometimes you perceive the faces, and sometimes the vase, even though nothing has changed on the page. You simply can’t see both at once.
There are even more striking demonstrations of this principle of active vision. Perceptual switching happens if we present one image to your left eye (say, a cow) and a different image to your right eye (say, an airplane). You don’t see both at the same time, nor do you see a fusion of the two images—instead, you see one, then the other, then back again.12 Your visual system is arbitrating a battle between the conflicting information, and you see not what is really out there, but instead only a moment-by-moment version of which perception is winning over the other. Even though the outside world has not changed, your brain dynamically presents different interpretations.
More than actively interpreting what is out there, the brain often goes beyond the call of duty to make things up. Consider the example of the retina, the specialized sheet of photoreceptor cells at the back of the eye. In 1668, the French philosopher and mathematician Edme Mariotte stumbled on something quite unexpected: there is a sizable patch in the retina where the photoreceptors are missing.13 This missing patch surprised Mariotte because the visual field appears continuous: there is no corresponding gaping hole of vision where the photoreceptors are missing.
Or isn’t there? As Mariotte delved more deeply into this issue, he realized that there is a hole in our vision—what has come to be known as the “blind spot” in each eye. To demonstrate this to yourself, close your left eye and keep your right eye fixed on the plus sign.
Slowly move the page closer to and farther from your face until the black dot disappears (probably when the page is about twelve inches away). You can no longer see the dot because it is sitting in your blind spot.
Don’t assume that your blind spot is small. It’s huge. Imagine the diameter of the moon in the night sky. You can fit seventeen moons into your blind spot.
So why hadn’t anyone noticed this hole in vision before Mariotte? How could brilliant minds like Michelangelo, Shakespeare, and Galileo have lived and died without ever detecting this basic fact of vision? One reason is because there are two eyes and the blind spots are in different, nonoverlapping locations; this means that with both eyes open you have full coverage of the scene. But more significantly, no one had noticed because the brain “fills in” the missing information from the blind spot. Notice what you see in the location of the dot when it’s in your blind spot. When the dot disappears, you do not perceive a hole of whiteness or blackness in its place; instead your brain invents a patch of the background pattern. Your brain, with no information from that particular spot in visual space, fills in with the patterns around it.
You’re not perceiving what’s out there. You’re perceiving whatever your brain tells you.
* * *
By the mid-1800s, the German physicist and physician Hermann von Helmholtz (1821–1894) had begun to entertain the suspicion that the trickle of data moving from the eyes to the brain is too small to really account for the rich experience of vision. He concluded that the brain must make assumptions about the incoming data, and that these assumptions are based on our previous experience.14 In other words, given a little information, your brain uses its best guesses to turn it into something larger.
Consider this: based on your previous experience, your brain assumes that visual scenes are illuminated by a light source from above.15 So a flat circle with shading that is lighter at the top and darker at the bottom will be seen as bulging out; one with shading in the opposite direction will be perceived to be dimpling in. Rotating the figure ninety degrees will remove the illusion, making it clear that these are merely flat, shaded circles—but when the figure is turned right side up again, one cannot help but feel an illusory sense of depth.
As a result of the brain’s notions about lighting sources, it makes unconscious assumptions about shadows as well: if a square casts a shadow and the shadow suddenly moves, you will believe the square has moved in depth.16
Take a look at the figure below: the square hasn’t moved at all; the dark square representing its shadow has merely been drawn in a slightly different place. This could have happened because the overhead lighting source suddenly shifted position—but because of your previous experience with the slow-moving sun and fixed electrical lighting, your perception automatically gives preference to the likelier explanation: the object has moved toward you.
Helmholtz called this concept of vision “unconscious inference,” where inference refers to the idea that the brain conjectures what might be out there, and unconscious reminds us that we have no awareness of the process. We have no access to the rapid and automatic machinery that gathers and estimates the statistics of the world. We’re merely the beneficiaries riding on top of the machinery, enjoying the play of light and shadows.
HOW CAN ROCKS DRIFT UPWARD WITHOUT CHANGING POSITION?
When we begin to look closely at that machinery, we find a complex system of specialized cells and circuits in the part of your brain called the visual cortex. There is a division of labor among these circuits: some are specialized for color, some for motion, some for edges, and others for scores of different attributes. These circuits are densely interconnected, and they come to conclusions as a group. When necessary, they serve up a headline for what we might call the Consciousness Post. The headline reports only that a bus is coming or that someone has flashed a flirtatious smile—but it does not cite the varied sources. Sometimes it is tempting to think that seeing is easy despite the complicated neural machinery that underlies it. To the contrary, it is easy because of the complicated neural machinery.
When we take a close look at the machinery, we find that vision can be deconstructed into parts. Sta
re at a waterfall for a few minutes; after shifting your gaze, stationary objects such as the nearby rocks will briefly appear to crawl upward.17 Strangely, there is no change in their position over time, even though their movement is clear. Here the imbalanced activity of your motion detectors (usually upward-signaling neurons are balanced in a push–pull relationship with downward-signaling neurons) allows you to see what is impossible in the outside world: motion without position change. This illusion—known as the motion aftereffect or the waterfall illusion—has enjoyed a rich history of study dating back to Aristotle. The illusion illustrates that vision is the product of different modules: in this case, some parts of the visual system insist (incorrectly) that the rocks are moving, while other parts insist that the rocks are not, in fact, changing position. As the philosopher Daniel Dennett has argued, the naïve introspector usually relies on the bad metaphor of the television screen,18 where moving-while-staying-still cannot happen. But the visual world of the brain is nothing like a television screen, and motion with no change in position is a conclusion it sometimes lands upon.
Motion can be seen even when there is no change in position. (a) High-contrast figures like these stimulate motion detectors, giving the impression of constant movement around the rings. (b) Similarly, the zigzag wheels here appear to turn slowly.
There are many illusions of motion with no change of position. The figure below demonstrates that static images can appear to move if they happen to tickle motion detectors in the right way. These illusions exist because the exact shading in the pictures stimulates motion detectors in the visual system—and the activity of these receptors is equivalent to the perception of motion. If your motion detectors declare that something is moving out there, the conscious you believes it without question. And not merely believes it but experiences it.
A striking example of this principle comes from a woman who in 1978 suffered carbon monoxide poisoning.19 Fortunately, she lived; unfortunately, she suffered irreversible brain damage to parts of her visual system—specifically, the regions involved in representing motion. Because the rest of her visual system was intact, she was able to see stationary objects with no problem. She could tell you there was a ball over there and a telephone over here. But she could no longer see motion. If she stood on a sidewalk trying to cross the street, she could see the red truck over there, and then here a moment later, and finally over there, past her, another moment later—but the truck had no sense of movement to it. If she tried to pour water out of a pitcher, she would see a tilted pitcher, then a gleaming column of water hanging from the pitcher, and finally a puddle of water around the glass as it overflowed—but she couldn’t see the liquid move. Her life was a series of snapshots. Just as with the waterfall effect, her condition of motion blindness tells us that position and motion are separable in the brain. Motion is “painted on” our views of the world, just as it is erroneously painted on the images above.
A physicist thinks about motion as change in position through time. But the brain has its own logic, and this is why thinking about motion like a physicist rather than like a neuroscientist will lead to wrong predictions about how people operate. Consider baseball outfielders catching fly balls. How do they decide where to run to intercept the ball? Probably their brains represent where the ball is from moment to moment: now it’s over there, now it’s a little closer, now it’s even closer. Right? Wrong.
So perhaps the outfielder’s brain calculates the ball’s velocity, right? Wrong.
Acceleration? Wrong.
Scientist and baseball fan Mike McBeath set out to understand the hidden neural computations behind catching fly balls.20 He discovered that outfielders use an unconscious program that tells them not where to end up but simply how to keep running. They move in such a way that the parabolic path of the ball always progresses in a straight line from their point of view. If the ball’s path looks like its deviating from a straight line, they modify their running path.
This simple program makes the strange prediction that the outfielders will not dash directly to the landing point of the ball but will instead take a peculiarly curved running path to get there. And that’s exactly what players do, as verified by McBeath and his colleagues by aerial video.21 And because this running strategy gives no information about where the point of intersection will be, only how to keep moving to get there, the program explains why outfielders crash into walls while chasing uncatchable fly balls.
So we see that the system does not need to explicitly represent position, velocity, or acceleration in order for the player to succeed in catching or interception. This is probably not what a physicist would have predicted. And this drives home the point that introspection has little meaningful insight into what is happening behind the scenes. Outfielding greats such as Ryan Braun and Matt Kemp have no idea that they’re running these programs; they simply enjoy the consequences and cash the resulting paychecks.
LEARNING TO SEE
When Mike May was three years old, a chemical explosion rendered him completely blind. This did not stop him from becoming the best blind downhill speed skier in the world, as well as a businessman and family man. Then, forty-three years after the explosion robbed him of his vision, he heard about a new surgical development that might be able to restore it. Although he was successful in his life as a blind man, he decided to undergo the surgery.
After the operation, the bandages were removed from around his eyes. Accompanied by a photographer, Mike sat on a chair while his two children were brought in. This was a big moment. It would be the first time he would ever gaze into their faces with his newly cleared eyes. In the resulting photograph, Mike has a pleasant but awkward smile on his face as his children beam at him.
The scene was supposed to be touching, but it wasn’t. There was a problem. Mike’s eyes were now working perfectly, but he stared with utter puzzlement at the objects in front of him. His brain didn’t know what to make of the barrage of inputs. He wasn’t experiencing his sons’ faces; he was experiencing only uninterpretable sensations of edges and colors and lights. Although his eyes were functioning, he didn’t have vision.22
And this is because the brain has to learn how to see. The strange electrical storms inside the pitch-black skull get turned into conscious summaries after a long haul of figuring out how objects in the world match up across the senses. Consider the experience of walking down a hallway. Mike knew from a lifetime of moving down corridors that walls remain parallel, at arm’s length, the whole way down. So when his vision was restored, the concept of converging perspective lines was beyond his capacity to understand. It made no sense to his brain.
Similarly, when I was a child I met a blind woman and was amazed at how intimately she knew the layout of her rooms and furniture. I asked her if she would be able to draw out the blueprints with higher accuracy than most sighted people. Her response surprised me: she said she would not be able to draw the blueprints at all, because she didn’t understand how sighted people converted three dimensions (the room) into two dimensions (a flat piece of paper). The idea simply didn’t make sense to her.23
Vision does not simply exist when a person confronts the world with clear eyes. Instead, an interpretation of the electrochemical signals streaming along the optic nerves has to be trained up. Mike’s brain didn’t understand how his own movements changed the sensory consequences. For example, when he moves his head to the left, the scene shifts to the right. The brains of sighted people have come to expect such things and know how to ignore them. But Mike’s brain was flummoxed at these strange relationships. And this illustrates a key point: the conscious experience of vision occurs only when there is accurate prediction of sensory consequences,24 a point to which we will return shortly. So although vision seems like a rendition of something that’s objectively out there, it doesn’t come for free. It has to be learned.
After moving around for several weeks, staring at things, kicking chairs, examining silverware, rubbing his wife?
??s face, Mike came to have the experience of sight as we experience it. He now experiences vision the same way you do. He just appreciates it more.
* * *
Mike’s story shows that the brain can take a torrent of input and learn to make sense of it. But does this imply the bizarre prediction that you can substitute one sense for another? In other words, if you took a data stream from a video camera and converted it into an input to a different sense—taste or touch, say—would you eventually be able to see the world that way? Incredibly, the answer is yes, and the consequences run deep, as we are about to see.
SEEING WITH THE BRAIN
In the 1960s, the neuroscientist Paul Bach-y-Rita at the University of Wisconsin began chewing on the problem of how to give vision to the blind.25 His father had recently had a miraculous recovery from a stroke, and Paul found himself enchanted by the potential for dynamically reconfiguring the brain.
A question grew in his mind: could the brain substitute one sense for another? Bach-y-Rita decided to try presenting a tactile “display” to blind people.26 Here’s the idea: attach a video camera to someone’s forehead and convert the incoming video information into an array of tiny vibrators attached to their back. Imagine putting this device on and walking around a room blindfolded. At first you’d feel a bizarre pattern of vibrations on the small of your back. Although the vibrations would change in strict relation to your own movements, it would be quite difficult to figure out what was going on. As you hit your shin against the coffee table, you’d think, “This really is nothing like vision.”