Or, how about the following:
The Genomes of Ten Leading Pathogens
The Floor Plans of Leading Skyscrapers
The Layout of U.S. Nuclear Reactors
The Hundred Top Vulnerabilities of Modern Society
The Top Ten Vulnerabilities of the Internet
Personal Health Information on One Hundred Million Americans
The Customer Lists of Top Pornography Sites
Anyone posting the first item above is almost certain to get a quick visit from the FBI, as did Nate Ciccolo, a fifteen-year-old high school student, in March 2000. For a school science project he built a papier-mâché model of an atomic bomb that turned out to be disturbingly accurate. In the ensuing media storm Ciccolo told ABC News, “Someone just sort of mentioned, you know, you can go on the Internet now and get information. And I, sort of, wasn’t exactly up to date on things. Try it. I went on there and a couple of clicks and I was right there.”3
Of course Ciccolo didn’t possess the key ingredient, plutonium, nor did he have any intention of acquiring it, but the report created shock waves in the media, not to mention among the authorities who worry about nuclear proliferation. Ciccolo had reported finding 563 Web pages on atomic-bomb designs, and the publicity resulted in an urgent effort to remove them. Unfortunately, trying to get rid of information on the Internet is akin to trying to sweep back the ocean with a broom. Some of the sites continue to be easily accessible today. I won’t provide any URLs in this book, but they are not hard to find.
Although the article titles above are fictitious, one can find extensive information on the Internet about all of these topics.4 The Web is an extraordinary research tool. In my own experience, research that used to require a half day at the library can now be accomplished typically in a couple of minutes or less. This has enormous and obvious benefits for advancing beneficial technologies, but it can also empower those whose values are inimical to the mainstream of society. So are we in danger? The answer is clearly yes. How much danger, and what to do about it, are the subjects of this chapter.
My urgent concern with this issue dates back at least a couple of decades. When I wrote The Age of Intelligent Machines in the mid-1980s, I was deeply concerned with the ability of then-emerging genetic engineering to enable those skilled in the art and with access to fairly widely available equipment to modify bacterial and viral pathogens to create new diseases.5 In destructive or merely careless hands these engineered pathogens could potentially combine a high degree of communicability, stealthiness, and destructiveness.
Such efforts were not easy to carry out in the 1980s but were nonetheless feasible. We now know that bioweapons programs in the Soviet Union and elsewhere were doing exactly this.6 At the time I made a conscious decision to not talk about this specter in my book, feeling that I did not want to give the wrong people any destructive ideas. I didn’t want to turn on the radio one day and hear about a disaster, with the perpetrators saying that they got the idea from Ray Kurzweil.
Partly as a result of this decision I faced some reasonable criticism that the book emphasized the benefits of future technology while ignoring its pitfalls. When I wrote The Age of Spiritual Machines in 1997–1998, therefore, I attempted to account for both promise and peril.7 There had been sufficient public attention by that time (for example, the 1995 movie Outbreak, which portrays the terror and panic from the release of a new viral pathogen) that I felt comfortable to begin to address the issue publicly.
In September 1998, having just completed the manuscript, I ran into Bill Joy, an esteemed and longtime colleague in the high-technology world, in a bar in Lake Tahoe. Although I had long admired Joy for his work in pioneering the leading software language for interactive Web systems (Java) and having cofounded Sun Microsystems, my focus at this brief get-together was not on Joy but rather on the third person sitting in our small booth, John Searle. Searle, the eminent philosopher from the University of California at Berkeley, had built a career of defending the deep mysteries of human consciousness from apparent attack by materialists such as Ray Kurzweil (a characterization I reject in the next chapter).
Searle and I had just finished debating the issue of whether a machine could be conscious during the closing session of George Gilder’s Telecosm conference. The session was entitled “Spiritual Machines” and was devoted to a discussion of the philosophical implications of my upcoming book. I had given Joy a preliminary manuscript and tried to bring him up to speed on the debate about consciousness that Searle and I were having.
As it turned out, Joy was interested in a completely different issue, specifically the impending dangers to human civilization from three emerging technologies I had presented in the book: genetics, nanotechnology, and robotics (GNR, as discussed earlier). My discussion of the downsides of future technology alarmed Joy, as he would later relate in his now-famous cover story for Wired, “Why the Future Doesn’t Need Us.”8 In the article Joy describes how he asked his friends in the scientific and technology community whether the projections I was making were credible and was dismayed to discover how close these capabilities were to realization.
Joy’s article focused entirely on the downside scenarios and created a firestorm. Here was one of the technology world’s leading figures addressing new and dire emerging dangers from future technology. It was reminiscent of the attention that George Soros, the currency arbitrageur and archcapitalist, received when he made vaguely critical comments about the excesses of unrestrained capitalism, although the Joy controversy became far more intense. The New York Times reported there were about ten thousand articles commenting on and discussing Joy’s article, more than any other in the history of commentary on technology issues. My attempt to relax in a Lake Tahoe lounge thus ended up fostering two long-term debates, as my dialogue with John Searle has also continued to this day.
Despite my being the origin of Joy’s concern, my reputation as a “technology optimist” has remained intact, and Joy and I have been invited to a variety of forums to debate the peril and promise, respectively, of future technologies. Although I am expected to take up the “promise” side of the debate, I often end up spending most of my time defending his position on the feasibility of these dangers.
Many people have interpreted Joy’s article as an advocacy of broad relinquishment, not of all technological developments, but of the “dangerous ones” like nanotechnology. Joy, who is now working as a venture capitalist with the legendary Silicon Valley firm of Kleiner, Perkins, Caufield & Byers, investing in technologies such as nanotechnology applied to renewable energy and other natural resources, says that broad relinquishment is a misinterpretation of his position and was never his intent. In a recent private e-mail communication, he says the emphasis should be on his call to “limit development of the technologies that are too dangerous” (see the epigraph at the beginning of this chapter), not on complete prohibition. He suggests, for example, a prohibition against self-replicating nanotechnology, which is similar to the guidelines advocated by the Foresight Institute, founded by nanotechnology pioneer Eric Drexler and Christine Peterson. Overall, this is a reasonable guideline, although I believe there will need to be two exceptions, which I discuss below (see p. 411).
As another example, Joy advocates not publishing the gene sequences of pathogens on the Internet, which I also agree with. He would like to see scientists adopt regulations along these lines voluntarily and internationally, and he points out that “if we wait until after a catastrophe, we may end up with more severe and damaging regulations.” He says he hopes that “we will do such regulation lightly, so that we can get most of the benefits.”
Others, such as Bill McKibben, the environmentalist who was one of the first to warn against global warming, have advocated relinquishment of broad areas such as biotechnology and nanotechnology, or even of all technology. As I discuss in greater detail below (see p. 410), relinquishing broad fields would be impossible to achieve without essentially relin
quishing all technical development. That in turn would require a Brave New World style of totalitarian government, banning all technology development. Not only would such a solution be inconsistent with our democratic values, but it would actually make the dangers worse by driving the technology underground, where only the least responsible practitioners (for example, rogue states) would have most of the expertise.
Intertwined Benefits . . .
It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way.
—CHARLES DICKENS, A TALE OF TWO CITIES
It’s like arguing in favor of the plough. You know some people are going to argue against it, but you also know it’s going to exist.
—JAMES HUGHES, SECRETARY OF THE TRANSHUMANIST ASSOCIATION AND SOCIOLOGIST AT TRINITY COLLEGE, IN A DEBATE, “SHOULD HUMANS WELCOME OR RESIST BECOMING POSTHUMAN?”
Technology has always been a mixed blessing, bringing us benefits such as longer and healthier lifespans, freedom from physical and mental drudgery, and many novel creative possibilities on the one hand, while introducing new dangers. Technology empowers both our creative and destructive natures.
Substantial portions of our species have already experienced alleviation of the poverty, disease, hard labor, and misfortune that have characterized much of human history. Many of us now have the opportunity to gain satisfaction and meaning from our work, rather than merely toiling to survive. We have ever more powerful tools to express ourselves. With the Web now reaching deeply into less developed regions of the world, we will see major strides in the availability of high-quality education and medical knowledge. We can share culture, art, and humankind’s exponentially expanding knowledge base worldwide. I mentioned the World Bank’s report on the worldwide reduction in poverty in chapter 2 and discuss that further in the next chapter.
We’ve gone from about twenty democracies in the world after World War II to more than one hundred today largely through the influence of decentralized electronic communication. The biggest wave of democratization, including the fall of the Iron Curtain, occurred during the 1990s with the growth of the Internet and related technologies. There is, of course, a great deal more to accomplish in each of these areas.
Bioengineering is in the early stages of making enormous strides in reversing disease and aging processes. Ubiquitous N and R are two to three decades away and will continue an exponential expansion of these benefits. As I reviewed in earlier chapters, these technologies will create extraordinary wealth, thereby overcoming poverty and enabling us to provide for all of our material needs by transforming inexpensive raw materials and information into any type of product.
We will spend increasing portions of our time in virtual environments and will be able to have any type of desired experience with anyone, real or simulated, in virtual reality. Nanotechnology will bring a similar ability to morph the physical world to our needs and desires. Lingering problems from our waning industrial age will be overcome. We will be able to reverse remaining environmental destruction. Nanoengineered fuel cells and solar cells will provide clean energy. Nanobots in our physical bodies will destroy pathogens, remove debris such as misformed proteins and protofibrils, repair DNA, and reverse aging. We will be able to redesign all of the systems in our bodies and brains to be far more capable and durable.
Most significant will be the merger of biological and nonbiological intelligence, although nonbiological intelligence will quickly come to predominate. There will be a vast expansion of the concept of what it means to be human. We will greatly enhance our ability to create and appreciate all forms of knowledge from science to the arts, while extending our ability to relate to our environment and one another.
On the other hand . . .
. . . and Dangers
“Plants” with “leaves” no more efficient than today’s solar cells could out-compete real plants, crowding the biosphere with an inedible foliage. Tough omnivorous “bacteria” could out-compete real bacteria: They could spread like blowing pollen, replicated swiftly, and reduce the biosphere to dust in a matter of days. Dangerous replicators could easily be too tough, small, and rapidly spreading to stop—at least if we make no preparation. We have trouble enough controlling viruses and fruit flies.
—ERIC DREXLER
As well as its many remarkable accomplishments, the twentieth century saw technology’s awesome ability to amplify our destructive nature, from Stalin’s tanks to Hitler’s trains. The tragic event of September 11, 2001, is another example of technologies (jets and buildings) taken over by people with agendas of destruction. We still live today with a sufficient number of nuclear weapons (not all of which are accounted for) to end all mammalian life on the planet.
Since the 1980s the means and knowledge have existed in a routine college bioengineering lab to create unfriendly pathogens potentially more dangerous than nuclear weapons.9 In a war-game simulation conducted at Johns Hopkins University called “Dark Winter,” it was estimated that an intentional introduction of conventional smallpox in three U.S. cities could result in one million deaths. If the virus were bioengineered to defeat the existing smallpox vaccine, the results could be far worse.10 The reality of this specter was made clear by a 2001 experiment in Australia in which the mousepox virus was inadvertently modified with genes that altered the immune-system response. The mousepox vaccine was powerless to stop this altered virus.11 These dangers resonate in our historical memories. Bubonic plague killed one third of the European population. More recently the 1918 flu killed twenty million people worldwide.12
Will such threats prevent the ongoing acceleration of the power, efficiency, and intelligence of complex systems (such as humans and our technology)? The past record of complexity increase on this planet has shown a smooth acceleration, even through a long history of catastrophes, both internally generated and externally imposed. This is true of both biological evolution (which faced calamities such as encounters with large asteroids and meteors) and human history (which has been punctuated by an ongoing series of major wars).
However, I believe we can take some encouragement from the effectiveness of the world’s response to the SARS (severe acute respiratory syndrome) virus. Although the possibility of an even more virulent return of SARS remains uncertain as of the writing of this book, it appears that containment measures have been relatively successful and have prevented this tragic outbreak from becoming a true catastrophe. Part of the response involved ancient, low-tech tools such as quarantine and face masks.
However, this approach would not have worked without advanced tools that have only recently become available. Researchers were able to sequence the DNA of the SARS virus within thirty-one days of the outbreak—compared to fifteen years for HIV. That enabled the rapid development of an effective test so that carriers could quickly be identified. Moreover, instantaneous global communication facilitated a coordinated response worldwide, a feat not possible when viruses ravaged the world in ancient times.
As technology accelerates toward the full realization of GNR, we will see the same intertwined potentials: a feast of creativity resulting from human intelligence expanded manyfold, combined with many grave new dangers. A quintessential concern that has received considerable attention is unrestrained nanobot replication. Nanobot technology requires trillions of such intelligently designed devices to be useful. To scale up to such levels it will be necessary to enable them to self-replicate, essentially the same approach used in the biological world (that’s how one fertilized egg cell becomes the trillions of cells in a human). And in the same way that biological self-replication gone awry (that is, cancer) results in biological destruction, a defect in the mechanism curtailing
nanobot self-replication—the so-called gray-goo scenario—would endanger all physical entities, biological or otherwise.
Living creatures—including humans—would be the primary victims of an exponentially spreading nanobot attack. The principal designs for nanobot construction use carbon as a primary building block. Because of carbon’s unique ability to form four-way bonds, it is an ideal building block for molecular assemblies. Carbon molecules can form straight chains, zigzags, rings, nanotubes (hexagonal arrays formed in tubes), sheets, buckyballs (arrays of hexagons and pentagons formed into spheres), and a variety of other shapes. Because biology has made the same use of carbon, pathological nanobots would find the Earth’s biomass an ideal source of this primary ingredient. Biological entities can also provide stored energy in the form of glucose and ATP.13 Useful trace elements such as oxygen, sulfur, iron, calcium, and others are also available in the biomass.
How long would it take an out-of-control replicating nanobot to destroy the Earth’s biomass? The biomass has on the order of 1045 carbon atoms.14 A reasonable estimate of the number of carbon atoms in a single replicating nanobot is about 106. (Note that this analysis is not very sensitive to the accuracy of these figures, only to the approximate order of magnitude.) This malevolent nanobot would need to create on the order of 1039 copies of itself to replace the biomass, which could be accomplished with 130 replications (each of which would potentially double the destroyed biomass). Rob Freitas has estimated a minimum replication time of approximately one hundred seconds, so 130 replication cycles would require about three and a half hours.15 However, the actual rate of destruction would be slower because biomass is not “efficiently” laid out. The limiting factor would be the actual movement of the front of destruction. Nanobots cannot travel very quickly because of their small size. It’s likely to take weeks for such a destructive process to circle the globe.