Dr. Alice Freundlich, Harvard University, Department of Biochemistry.
Dr. Habib Ndina, University of Virginia Medical School.
Dr. Stuart Thornton, University of Maryland, College Park, Genomics Department.
Dr. Francesca Taolini, Massachusetts Institute of Technology, Center for Biocomputational Studies.
Dr. Jerome Frenkel, University of Pennsylvania, Department of Genomics.
Dr. Yao Lee, Cambridge University (visiting GWU’s Department of Microbiology).
Frank made his usual introductory remarks and then said, “We’ve got a lot of them to go through this time. I’m sorry it’s so many, but that’s what we’ve received. I’m sure we’ll hack our way through them all if we keep on track. Let’s start with the fifteen-minutes-per-jacket drill, and see if we can get twelve or even fourteen done before lunch. Sound good?”
Everyone nodded and tapped away, calling up the first one.
“Oh, and before we start, let’s have everyone give me their conflict-of-interest forms, please. I have to remind you that as referees here, you have a conflict if you’re the applying principal investigator’s thesis advisor or advisee, an employee of the same institution as the P.I. or a co-P.I., a collaborator within the last four years of the P.I. or a co-P.I., an applicant for employment in any department at the submitting institution, a recipient of an honorarium or other pay from the submitting institution within the last year, someone with a close personal relationship to the P.I. or a co-P.I., a shareholder in a company participating in the proposal, or someone who would otherwise gain or lose financially if the proposal were awarded or declined.
“Everybody got that? Okay, hand those forms down to me, then. We’ll have a couple of people step outside for some of the proposals today, but mostly we’re clear as far as I know, is that right?”
“I’ll be leaving for the Esterhaus proposal, as I told you,” Stuart Thornton said.
Then they started the group evaluations. This was the heart of their task for that day and the next—also the heart of NSF’s method, indeed of science more generally. Peer-review; a jury of fellow experts. Frank clicked the first proposal’s page onto his screen. “Seven reviewers, forty-four jackets. Let’s start with EIA-02 18599, ‘Electromagnetic and Informational Processes in Molecular Polymers.’ Habib, you’re the lead on this?”
Habib Ndina nodded and opened with a description of the proposal. “They want to immobilize cytoskeletal networks on biochips, and explore whether tubulin can be used as bits in protein logic gates. They intend to do this by measuring the electric dipole moment, and what the P.I. calls the predicted kink-solitonic electric dipole moment flip waves.”
“Predicted by whom?”
“By the P.I.” Habib smiled. “He also states that this will be a method to test out the theories of the so-called quantum brain.”
“Hmm.” People read past the abstract.
“What are you thinking?” Frank said after a while. “I see Habib has given it a ‘Good,’ Stuart a ‘Fair,’ and Alice a ‘Very Good.’ ”
This represented the middle range of their scale, which ran Poor, Fair, Good, Very Good, and Excellent.
Habib replied first. “I’m not so sure that you can get these biochips to array in neural nets. I saw Inouye try something like that at MIT, and they got stuck at the level of chip viability.”
“Hmm.”
The others chimed in with questions and opinions. At the end of fifteen minutes, Frank stopped the discussion and asked them to mark their final judgments in the two categories they used: intellectual merit and broader impacts.
Frank summed up the responses. “Four ‘Goods,’ two ‘Very Goods’ and a ‘Fair.’ Okay, let’s move on. But tell you what, I’m going to start the big board right now.”
He had a whiteboard in the corner next to him, and a pile of Post-it pads on the table. He drew three zones on the whiteboard with marker, and wrote at the top “Fund,” “Fund If Possible,” and “Don’t Fund.”
“I’ll put this one in the ‘Fund If Possible’ column for now, although naturally it may get bumped.” He stuck the proposal’s Post-it in the middle zone. “We’ll move these around as the day progresses and we get a sense of the range.”
Then they began the next one. “Okay. ‘Efficient Decoherence Control Algorithms for Computing Genome Construction.’ ”
This jacket Frank had assigned to Stuart Thornton.
Thornton started by shaking his head. “This one’s gotten two ‘Goods’ and two ‘Fairs,’ and it wasn’t very impressive to me either. It may be a candidate for limited discussion. It doesn’t really exhibit a grasp of the difficulties involved with codon tampering, and I think it replicates the work being done in Seattle by Johnson’s lab. The applicant seems to have been too busy with the broader impacts component to fully acquaint himself with the literature. Besides which, it won’t work.”
People laughed shortly at this extra measure of disdain, which was palpable, and to those who didn’t know Thornton, a little surprising. But Frank had seen Stuart Thornton on panels before. He was the kind of scientist who habitually displayed an ultrapure devotion to the scientific method, in the form of a relentless skepticism about everything. No study was designed tightly enough, no data were clean enough. To Frank it seemed obvious that it was really a kind of insecurity, part of the gestural set of a beta male convincing the group he was tough enough to be an alpha male, and maybe already was.
The problem with these gestures was that in science one’s intellectual power was like the muscle mass of an Australopithecus—there for all to see. You couldn’t fake it. No matter how much you ruffed your fur or exposed your teeth, in the end your intellectual strength was discernible in what you said and how insightful it was. Mere skepticism was like baring teeth; anyone could do it. For that reason Thornton was a bad choice for a panel, because while people could see his attitude and try to discount it, he set a tone that was hard to shake off. If there was an always defector in the group, one had to be less generous oneself in order not to become a sap.
That was why Frank had invited him.
Thornton went on, “The basic problem is at the level of their understanding of an algorithm. An algorithm is not just a simple sequence of mathematical operations that can each be performed in turn. It’s a matter of designing a grammar that will adjust the operations at each stage, depending on what the results are from the stage before. There’s a very specific encoding math that makes that work. They don’t have that here, as far as I can tell.”
The others nodded and tapped in notes at their consoles. Soon enough they were on to the next proposal, with the previous one posted under “Don’t Fund.”
Now Frank could predict with some confidence how the rest of the day would go. A depressed norm had been set, and even though the third reporter, Alice Freundlich from Harvard, subtly rebuked Thornton by talking about how well-designed her first jacket was, she did so in a less generous context, and was not overly enthusiastic. “They think that the evolutionary processes of gene conservation can be mapped by cascade studies, and they want to model it with big computer array simulations. They claim they’ll be able to identify genes prone to mutation.”
Habib Ndina shook his head. He too was a habitual skeptic, although from a much deeper well of intelligence than Thornton’s; he wasn’t just making a display, he was thinking. “Isn’t the genome’s past pretty much mapped by now?” he complained. “Do we really need more about evolutionary history?”
“Well, maybe not. Broader impacts might suffer there.”
And so the day proceeded, and, with some subliminal prompting from Frank (“Are you sure they have the lab space?” “Do you think that’s really true, though?” “How would that work?” “How could that work?”), the full Shooting Gallery Syndrome slowly emerged. The panelists very slightly lost contact with their sense of the proposals as human efforts performed under a deadline, and started to compare them to some perfect mo
del of scientific practice. In that light, of course, all the candidates were wanting. They all had feet of clay and their proposals all became clay pigeons, cast into the air for the group to take potshots at. New jacket tossed up: bang! bang! bang!
“This one’s toast,” someone said at one point.
Of course a few people in such a situation would stay anchored, and begin to shake their heads or wrinkle their noses, or even protest the mood, humorously or otherwise. But Frank had avoided inviting any of the real stalwarts he knew, and Alice Freundlich did no more than keep things pleasant. The impulse in a group toward piling on was so strong that it often took on extraordinary momentum. On the savannah it would have meant an expulsion and a hungry night out. Or some poor guy torn limb from limb.
Frank didn’t need to tip things that far. Nothing explicit, nothing heavy. He was only the facilitator. He did not express an obvious opinion on the substance of the proposals at any point. He watched the clock, ran down the list, asked if everybody had said what they wanted to say when there were three minutes left out of the fifteen; made sure everyone got their scores into the system at the end of the discussion period. “That’s an ‘Excellent’ and five ‘Very Goods.’ Alice do you have your scores on this one?”
Meanwhile the discussions got tougher and tougher. “I don’t know what she could have been thinking with this one, it’s absurd!”
“Let me start by suggesting limited discussion.”
Frank began subtly to apply the brakes. He didn’t want them to think he was a bad panel manager.
Nevertheless, the attack mood gained momentum. Baboons descending on wounded prey; it was almost Pavlovian, a food-rewarded joy in destruction that did not bode well for the species. The pleasure taken in wrecking anything meticulous. Frank had seen it many times: a carpenter doing demolition with a sledgehammer, a vet who went duck-hunting on weekends … It was unfortunate, given their current overextended moment in planetary history, but nevertheless real. As a species they were therefore probably doomed. And so the only real adaptive strategy, for the individual, was to do one’s best to secure one’s own position. And sometimes that meant a little strategic defection.
Near the end of the day it was Thornton’s turn again. Finally they had come to the proposal from Yann Pierzinski. People were getting tired.
Frank said, “Okay, almost done here. Let’s finish them off, shall we? Two more to go. Stu, we’re to you again, on ‘Mathematical and Algorithmic Analysis of Palindromic Codons as Predictors of a Gene’s Protein Expression.’ Mandel and Pierzinski, Caltech.”
Thornton shook his head wearily. “I see it’s got a couple of ‘Very Goods’ from people, but I give it a ‘Fair.’ It’s a nice thought, but it seems to be promising too much. I mean, predicting the proteome from the genome would be enough in itself, but then understanding how the genome evolved, building error-tolerant biocomputers—it’s like a list of the big unsolved problems.”
Francesca Taolini asked him what he thought of the algorithm that the proposal hoped to develop.
“It’s too sketchy to be sure! That’s really what he’s hoping to find, as far as I can tell. There would be a final toolbox with a software environment and language, then a gene grammar to make sense of palindromes in particular, he seems to think those are important, but I think they’re just redundancy and repair sequences, that’s why the palindromic structure. They’re like the reinforcement at the bottom of a zipper. To think that he could use this to predict all the proteins that a particular gene would produce!”
“But if you could, you would see what proteins you would get without needing to do microassays and use crystallography to see what came up,” Francesca pointed out. “That would be very useful. I thought the line he was following had potential, myself. I know people working on something like this, and it would be good to have more people on it, it’s a broad front. That’s why I gave it a ‘Very Good,’ and I’d still recommend we fund it.” She kept her eyes on her screen.
“Well yeah,” Thornton said crossly, “but where would he get the biosensors that would tell him if he was right or not? There’s no controls.”
“That would be someone else’s problem. If the predictions were turning out good you wouldn’t have to test all of them, that would be the point.”
Frank waited a beat. “Anyone else?” he said in a neutral tone.
Pritchard and Yao Lee joined in. Lee obviously thought it was a good idea, in theory. He started describing it as a kind of cookbook with evolving recipes, and Frank ventured to say, “How would that work?”
“Well, by successive iterations of the operation, you know. It would be to get you started, suggest directions to try.”
“Look,” Francesca interjected, “eventually we’re going to have to tackle this issue, because until we do, the mechanics of gene expression are just a black box. It’s a very valid line of inquiry.”
“Habib?” Frank asked.
“It would be nice, I guess, if he could make it work. It’s not so easy. It would be like a roll of the dice to support it.”
Before Francesca could collect herself and start again, Frank said, “Well, we could go round and round on that, but we’re out of time on this one, and it’s late. Those of you who haven’t done it yet, write down your scores, and let’s finish with one more from Alice before we go to dinner.”
Hunger made them nod and tap away at their consoles, and then they were on to the last one for the day, “Ribozymes as Molecular Logic Gates.” When they were done with that, Frank stuck its Post-it on the whiteboard with the rest. Each little square of paper had its proposal’s averaged scores written on it. It was a tight scale; the difference between 4.63 and 4.70 could matter a great deal. They had already put three proposals in the “Fund” column, two in the “Fund If Possible,” and six in the “Do Not Fund.” The rest were stuck to the bottom of the board, waiting to be sorted out the following day. Pierzinski’s was among those.
That evening the group went out for dinner at Tara, a good nearby Thai restaurant with a wall-sized fish tank. The conversation was animated and wide-ranging, the mood getting better as the meal wore on. Afterward a few of them went to the hotel bar; the rest retreated to their rooms. At eight the next morning they were back in the conference room doing everything over again, working their way through the proposals with an increasing efficiency. Thornton recused himself for a discussion of a proposal from someone at his university, and the mood in the room noticeably lightened; even when he returned they held to this. They were learning each other’s predilections, and sometimes jetted off into discussions of theory that were very interesting even though they were only a few minutes long. Some of the proposals brought up interesting problems, and several strong ones in a row made them aware of just how amazing contemporary work in bioinformatics was, and what some of the potential benefits for human health might be, if all this were to come together and make a robust biotechnology. The shadow of a good future drove the group toward more generous strategies. The second day went better. The scores were, on average, higher.
“My Lord,” Alice said at one point, looking at the whiteboard. “There are going to be some very good proposals that we’re not going to be able to fund.”
Everyone nodded. It was a common feeling at the end of a panel.
“I sometimes wonder what would happen if we could fund about ninety percent of all the applications. You know, only reject the limited-discussions. Fund everything else.”
“It might speed things up.”
“Might cause a revolution.”
“Now back to reality,” Frank suggested. “Last jacket here.”
When they had all tapped in their grading of the forty-fourth jacket, Frank quickly crunched the numbers on his general spreadsheet, sorting the applicants into a hierarchy from one to forty-four, with a lot of ties.
He printed out the results, including the funding each proposal was asking for, then called the group back to order. They
started moving the unsorted Post-its up into one or another of the three columns.
Pierzinski’s proposal had ended up ranked fourteenth out of the forty-four. It wouldn’t have been that high if it weren’t for Francesca. Now she urged them to fund it; but because it was in fourteenth place, the group decided it should be put in “Fund If Possible,” with a bullet.
Frank moved its Post-it on the whiteboard up into the “Fund If Possible” column, keeping his face perfectly blank. There were eight in “Fund If Possible,” six in “Fund,” twelve in “Do Not Fund.” Eighteen to go, therefore, but the arithmetic of the situation would doom most of these to the “Do Not Fund” column, with a few stuck into the “Fund If Possible” as faint hopes.
Later it would be Frank’s job to fill out a Form Seven for every proposal, summarizing the key aspects of the discussion, acknowledging outlier reviews that were more than one full place off the average, and explaining any “Excellents” awarded to nonfunded reviews; this was part of keeping the process transparent to the applicants, and making sure that nothing untoward happened. The panel was advisory only, NSF had the right to overule it, but in the great majority of cases the panel’s judgments would stand—that was the whole point—that was scientific objectivity, at least in this part of the process.
In a way it was funny. Solicit seven intensely subjective and sometimes contradictory opinions; quantify them; average them; and that was objectivity. A numerical grading that you could point to on a graph. Ridiculous, of course. But it was the best they could do. Indeed, what other choice did they have? No algorithm could make these kinds of decisions. The only computer powerful enough to do it was one made up of a networked array of human brains—that is to say, a panel. Beyond that they could not reach.
So they discussed the proposals one last time, their scientific potential and also their educational and benefit-to-society aspects, the “broader impacts” rubric, usually spelled out rather vaguely in the proposals, and unpopular with research purists. But as Frank put it now, “NSF isn’t here just to do science but also to promote science, and that means all these other criteria. What it will add to society.” What Anna will do with it, he almost said.