But even for medicine's most influential studies, the evidence sometimes remains surprisingly narrow. Of those forty-five super-cited studies that Ioannidis focused on, eleven had never been retested. Perhaps worse, Ioannidis found that even when a research error is outed, it typically persists for years or even decades. He looked at three prominent health studies from the 1980s and 1990s that were each later soundly refuted and discovered that researchers continued to cite the original results as correct more often than as flawed—in one case for at least twelve years after the results were discredited.
Doctors may notice that their patients don't seem to fare as well with certain treatments as the literature would lead them to expect, but the field is appropriately conditioned to subjugate such anecdotal evidence to study findings. Yet much, perhaps even most, of what doctors do has never been formally put to the test in credible studies, given that the need to do so became obvious to the field only in the 1990s, leaving it playing catch-up with a century or more of non-evidence-based medicine, and contributing to Ioannidis's shockingly high estimate of the degree to which medical knowledge is flawed. That we're not routinely made seriously ill by this shortfall, he argues, is due largely to the fact that most medical interventions and advice don't address life-and-death situations but rather aim to leave us marginally healthier or less unhealthy, so we usually neither gain nor risk all that much.
Medical research is not especially plagued with wrongness. Other meta-research experts have confirmed that similar issues distort research in all fields of science, from physics to economics (where the highly regarded economists J. Bradford DeLong and Kevin Lang once showed how a remarkably consistent paucity of strong evidence in published economics studies made it unlikely that any of them were right). And needless to say, things only get worse when it comes to the pop expertise that endlessly spews at us from diet, relationship, investment, and parenting gurus and pundits. But we expect more of scientists, and especially of medical scientists, given that we believe we are staking our lives on their results. The public hardly recognizes how bad a bet this is. The medical community itself might still be largely oblivious to the scope of the problem, if Ioannidis hadn't forced a confrontation when he published his studies in 2005.
Ioannidis initially thought the community might come out fighting. Instead, it seemed relieved, as if it had been guiltily waiting for someone to blow the whistle, and eager to hear more. David Gorski, a surgeon and researcher at Detroit's Barbara Ann Karmanos Cancer Institute, noted in his prominent medical blog that when he presented Ioannidis's paper on highly cited research at a professional meeting, "not a single one of my surgical colleagues was the least bit surprised or disturbed by its findings." Ioannidis offers a theory for the relatively calm reception. "I think that people didn't feel I was only trying to provoke them, because I showed that it was a community problem, instead of pointing fingers at individual examples of bad research," he says. In a sense, he gave scientists an opportunity to cluck about the wrongness without having to acknowledge that they themselves succumb to it—it was something everyone else did.
To say that Ioannidis's work has been embraced would be an understatement. His PLoS Medicine paper is the most downloaded in the journal's history, and it's not even Ioannidis's most-cited work—that would be a paper he published in Nature Genetics on the problems with gene-link studies. Other researchers are eager to work with him: he has published papers with 1,328 different coauthors at 538 institutions in forty-three countries, he says. Last year he received, by his estimate, invitations to speak at a thousand conferences and institutions around the world, and he was accepting an average of about five invitations a month until a case last year of excessive-travel-induced vertigo led him to cut back. Even so, in the weeks before I visited him he had addressed an AIDS conference in San Francisco, the European Society for Clinical Investigation, Harvard's School of Public Health, and the medical schools at Stanford and Tufts.
The irony of his having achieved this sort of success by accusing the medical-research community of chasing after success is not lost on him, and he notes that it ought to raise the question of whether he himself might be pumping up his findings. "If I did a study and the results showed that in fact there wasn't really much bias in research, would I be willing to publish it?" he asks. "That would create a real psychological conflict for me." But his bigger worry, he says, is that while his fellow researchers seem to be getting the message, he hasn't necessarily forced anyone to do a better job. He fears he won't in the end have done much to improve anyone's health. "There may not be fierce objections to what I'm saying," he explains. "But it's difficult to change the way that everyday doctors, patients, and healthy people think and behave."
As helter-skelter as the University of Ioannina Medical School campus looks, the hospital abutting it looks reassuringly stolid. Athina Tatsioni has offered to take me on a tour of the facility, but we make it only as far as the entrance when she is greeted—accosted, really—by a worried-looking older woman. Tatsioni, normally a bit reserved, is warm and animated with the woman, and the two have a brief but intense conversation before embracing and saying goodbye. Tatsioni explains to me that the woman and her husband were patients of hers years ago; now the husband has been admitted to the hospital with abdominal pains, and Tatsioni has promised she'll stop by his room later to say hello. Recalling the appendicitis story, I prod a bit, and she confesses she plans to do her own exam. She needs to be circumspect, though, so she won't appear to be second-guessing the other doctors.
Tatsioni doesn't so much fear that someone will carve out the man's healthy appendix. Rather, she's concerned that, like many patients, he'll end up with prescriptions for multiple drugs that will do little to help him and may well harm him. "Usually what happens is that the doctor will ask for a suite of biochemical tests—liver fat, pancreas function, and so on," she tells me. "The tests could turn up something, but they're probably irrelevant. Just having a good talk with the patient and getting a close history is much more likely to tell me what's wrong." Of course, the doctors have all been trained to order these tests, she notes, and doing so is a lot quicker than a long bedside chat. They're also trained to ply the patient with whatever drugs might help whack any errant test numbers back into line. What they're not trained to do is to go back and look at the research papers that helped make these drugs the standard of care. "When you look the papers up, you often find the drugs didn't even work better than a placebo. And no one tested how they worked in combination with the other drugs," she says. "Just taking the patient off everything can improve their health right away." But not only is checking out the research another time-consuming task, patients often don't even like it when they're taken off their drugs, she explains; they find their prescriptions reassuring.
Later, Ioannidis tells me he makes a point of having several clinicians on his team. "Researchers and physicians often don't understand each other; they speak different languages," he says. Knowing that some of his researchers are spending more than half their time seeing patients makes him feel the team is better positioned to bridge that gap; their experience informs the team's research with firsthand knowledge and helps the team shape its papers in a way more likely to hit home with physicians. It's not that he envisions doctors making all their decisions based solely on solid evidence—there's simply too much complexity in patient treatment to pin down every situation with a great study. "Doctors need to rely on instinct and judgment to make choices," he says. "But these choices should be as informed as possible by the evidence. And if the evidence isn't good, doctors should know that, too. And so should patients."
In fact, the question of whether the problems with medical research should be broadcast to the public is a sticky one in the meta-research community. Already feeling that they're fighting to keep patients from turning to alternative medical treatments such as homeopathy, or misdiagnosing themselves on the Internet, or simply neglecting medical treatment altogether, m
any researchers and physicians aren't eager to provide even more reason to be skeptical of what doctors do—not to mention how public disenchantment with medicine could affect research funding. Ioannidis dismisses these concerns. "If we don't tell the public about these problems, then we're no better than nonscientists who falsely claim they can heal," he says. "If the drugs don't work and we're not sure how to treat something, why should we claim differently? Some fear that there may be less funding because we stop claiming we can prove we have miraculous treatments. But if we can't really provide those miracles, how long will we be able to fool the public anyway? The scientific enterprise is probably the most fantastic achievement in human history, but that doesn't mean we have a right to overstate what we're accomplishing."
We could solve much of the wrongness problem, Ioannidis says, if the world simply stopped expecting scientists to be right. That's because being wrong in science is fine, and even necessary—as long as scientists recognize that they blew it, report their mistake openly instead of disguising it as a success, and then move on to the next thing, until they come up with the very occasional genuine breakthrough. But as long as careers remain contingent on producing a stream of research that's dressed up to seem more right than it is, scientists will keep delivering exactly that.
"Science is a noble endeavor, but it's also a low-yield endeavor," he says. "I'm not sure that more than a very small percentage of medical research is ever likely to lead to major improvements in clinical outcomes and quality of life. We should be very comfortable with that fact."
Letting Go
Atul Gawande
FROM The New Yorker
SARA THOMAS MONOPOLI was pregnant with her first child when her doctors learned that she was going to die. It started with a cough and a pain in her back. Then a chest X-ray showed that her left lung had collapsed, and her chest was filled with fluid. A sample of the fluid was drawn off with a long needle and sent for testing. Instead of an infection, as everyone had expected, it was lung cancer, and it had already spread to the lining of her chest. Her pregnancy was thirty-nine weeks along, and the obstetrician who had ordered the test broke the news to her as she sat with her husband and her parents. The obstetrician didn't get into the prognosis—she would bring in an oncologist for that—but Sara was stunned. Her mother, who had lost her best friend to lung cancer, began crying.
The doctors wanted to start treatment right away, and that meant inducing labor to get the baby out. For the moment, though, Sara and her husband, Rich, sat by themselves on a quiet terrace off the labor floor. It was a warm Monday in June 2007. She took Rich's hands, and they tried to absorb what they had heard. Monopoli was thirty-four. She had never smoked or lived with anyone who had. She exercised. She ate well. The diagnosis was bewildering. "This is going to be okay," Rich told her. "We're going to work through this. It's going to be hard, yes. But we'll figure it out. We can find the right treatment." For the moment, though, they had a baby to think about.
"So Sara and I looked at each other," Rich recalled, "and we said, 'We don't have cancer on Tuesday. It's a cancer-free day. We're having a baby. It's exciting. And we're going to enjoy our baby.'" On Tuesday, at 8:55 P.M., Vivian Monopoli, seven pounds nine ounces, was born. She had wavy brown hair like her mom, and she was perfectly healthy.
The next day Sara underwent blood tests and body scans. Dr. Paul Marcoux, an oncologist, met with her and her family to discuss the findings. He explained that she had a non-small-cell lung cancer that had started in her left lung. Nothing she had done had brought this on. More than 15 percent of lung cancers—more than people realize—occur in nonsmokers. Hers was advanced, having metastasized to multiple lymph nodes in her chest and its lining. The cancer was inoperable. But there were chemotherapy options, notably a relatively new drug called Tarceva, which targets a gene mutation commonly found in lung cancers of female nonsmokers. Eighty-five percent respond to this drug, and, Marcoux said, "some of these responses can be long-term."
Words like "respond" and "long-term" provide a reassuring gloss on a dire reality. There is no cure for lung cancer at this stage. Even with chemotherapy, the median survival is about a year. But it seemed harsh and pointless to confront Sara and Rich with this now. Vivian was in a bassinet by the bed. They were working hard to be optimistic. As Sara and Rich later told the social worker who was sent to see them, they did not want to focus on survival statistics. They wanted to focus on "aggressively managing" this diagnosis.
Sara was started on the Tarceva, which produced an itchy, acne-like facial rash and numbing tiredness. She also underwent a surgical procedure to drain the fluid around her lung; when the fluid kept coming back, a thoracic surgeon eventually placed a small, permanent tube in her chest, which she could drain whenever fluid accumulated and interfered with her breathing. Three weeks after the delivery, she was admitted to the hospital with severe shortness of breath from a pulmonary embolism—a blood clot in an artery to the lungs, which is dangerous but not uncommon in cancer patients. She was started on a blood thinner. Then test results showed that her tumor cells did not have the mutation that Tarceva targets. When Marcoux told Sara that the drug wasn't going to work, she had an almost violent physical reaction to the news, bolting to the bathroom in mid-discussion with a sudden bout of diarrhea.
Dr. Marcoux recommended a different, more standard chemotherapy, with two drugs called carboplatin and paclitaxel. But the paclitaxel triggered an extreme, nearly overwhelming allergic response, so he switched her to a regimen of carboplatin plus gemcitabine. Response rates, he said, were still very good for patients on this therapy.
She spent the remainder of the summer at home, with Vivian and her husband and her parents, who had moved in to help. She loved being a mother. Between chemotherapy cycles, she began trying to get her life back.
Then in October, a CT scan showed that the tumor deposits in her left lung and chest and lymph nodes had grown substantially. The chemotherapy had failed. She was switched to a drug called pemetrexed. Studies found that it could produce markedly longer survival in some patients. In reality, however, only a small percentage of patients gained very much. On average, the drug extended survival by only two months—from eleven months to thirteen months—and that was in patients who, unlike Sara, had responded to first-line chemotherapy.
She worked hard to take the setbacks and side effects in stride. She was upbeat by nature, and she managed to maintain her optimism. Little by little, however, she grew sicker—increasingly exhausted and short of breath. By November she didn't have the wind to walk the length of the hallway from the parking garage to Marcoux's office; Rich had to push her in a wheelchair.
A few days before Thanksgiving, she had another CT scan, which showed that the pemetrexed—her third drug regimen—wasn't working, either. The lung cancer had spread: from the left chest to the right; to the liver; to the lining of her abdomen; and to her spine. Time was running out.
This is the moment in Sara's story that poses a fundamental question for everyone living in the era of modern medicine: What do we want Sara and her doctors to do now? Or, to put it another way, if you were the one who had metastatic cancer—or, for that matter, a similarly advanced case of emphysema or congestive heart failure—what would you want your doctors to do?
The issue has become pressing in recent years, for reasons of expense. The soaring cost of health care is the greatest threat to the country's long-term solvency, and the terminally ill account for a lot of it. Twenty-five percent of all Medicare spending is for the 5 percent of patients who are in their final year of life, and most of that money goes for care in their last couple of months that is of little apparent benefit.
Spending on a disease like cancer tends to follow a particular pattern. There are high initial costs as the cancer is treated, and then, if all goes well, these costs taper off. Medical spending for a breast-cancer survivor, for instance, averaged an estimated $54,000 in 2003, the vast majority of it for the initial diagnost
ic testing, surgery, and, where necessary, radiation and chemotherapy. For a patient with a fatal version of the disease, though, the cost curve is U-shaped, rising again toward the end—to an average of $63,000 during the last six months of life with an incurable breast cancer. Our medical system is excellent at trying to stave off death with $8,000-a-month chemotherapy, $3,000-a-day intensive care, $5,000-an-hour surgery. But ultimately, death comes, and no one is good at knowing when to stop.
The subject seems to reach national awareness mainly as a question of who should "win" when the expensive decisions are made: the insurers and the taxpayers footing the bill or the patient battling for his or her life. Budget hawks urge us to face the fact that we can't afford everything. Demagogues shout about rationing and death panels. Market purists blame the existence of insurance: if patients and families paid the bills themselves, those expensive therapies would all come down in price. But they're debating the wrong question. The failure of our system of medical care for people facing the end of their life runs much deeper. To see this, you have to get close enough to grapple with the way decisions about care are actually made.
Recently, while seeing a patient in an intensive-care unit at my hospital, I stopped to talk with the critical-care physician on duty, someone I'd known since college. "I'm running a warehouse for the dying," she said bleakly. Out of the ten patients in her unit, she said, only two were likely to leave the hospital for any length of time. More typical was an almost eighty-year-old woman at the end of her life, with irreversible congestive heart failure, who was in the ICU for the second time in three weeks, drugged to oblivion and tubed in most natural orifices and a few artificial ones. Or the seventy-year-old with a cancer that had metastasized to her lungs and bone, and a fungal pneumonia that arises only in the final phase of the illness. She had chosen to forgo treatment, but her oncologist pushed her to change her mind, and she was put on a ventilator and antibiotics. Another woman, in her eighties, with end-stage respiratory and kidney failure, had been in the unit for two weeks. Her husband had died after a long illness with a feeding tube and a tracheotomy, and she had mentioned that she didn't want to die that way. But her children couldn't let her go and asked to proceed with the placement of various devices: a permanent tracheotomy, a feeding tube, and a dialysis catheter. So now she just lay there tethered to her pumps, drifting in and out of consciousness.