Peter Seibel - Coders at Work - Reflections on the craft of programming

Norvig:I see tests more as a way of correcting errors rather than as a way of design. This extreme approach of saying, “Well, the first thing you do is write a test that says I get the right answer at the end,” and then you run it and see that it fails, and then you say, “What do I need next?”—that doesn’t seem like the right way to design something to me.

It seems like only if it was so simple that the solution was preordained would that make sense. I think you have to think about it first. You have to say, “What are the pieces? How can I write tests for pieces until I know what some of them are?” And then, once you’ve done that, then it is good discipline to have tests for each of those pieces and to understand well how they interact with each other and the boundary cases and so on. Those should all have tests. But I don’t think you drive the whole design by saying, “This test has failed.”

The other thing I don’t like is a lot of the things we run up against at Google don’t fit this simple Boolean model of test. You look at these test suites and they have assertEqualand assertNotEqualand assertTrueand so on. And that’s useful but we also want to have assertAsFastAsPossibleand assert over this large database of possible queries we get results whose score is precision value of such and such and recall value of such and such and we’d like to optimize that. And they don’t have these kinds of statistical or continuous values that you’re trying to optimize, rather than just having a Boolean “Is this right or wrong?”

Seibel:But ultimately all of those can get converted into Booleans—run a bunch of queries and capture all those values and see if they’re all within the tolerances that you want.

Norvig:You could. But you can tell, just from the methods that the test suites give you, that they aren’t set up to do that, they haven’t thought about that as a possibility. I’m surprised at how much this type of approach is accepted at Google—when I was at Junglee I remember having to teach the QA team about it. We were doing this shopping search and saying, “We want a test where on this query we want to get 80 percent right answers.” And so they’re saying, “Right! So if it’s a wrong answer it’s a bug, right?” And I said, “No, it’s OK to have one wrong answer as long at it’s not 80 percent.” So they say, “So a wrong answer’s not a bug?” It was like those were the only two possibilities. There wasn’t an idea that it’s more of a trade-off.

Seibel:But you are still a believer in unit tests. How should programmers think about testing?

Norvig:They should write lots of tests. They should think about different conditions. And I think you want to have more complex regression tests as well as the unit tests. And think about failure modes—I remember one of the great lessons I got about programming was when I showed up at the airport at Heathrow, and there was a power failure and none of the computers were working. But my plane was on time.

Somehow they had gotten print-outs of all the flights. I don’t know where—there must have been some computer off-site. I don’t know whether they printed them that morning or if they had a procedure of always printing them the night before and sending them over and every day when there is power they just throw them out. But somehow they were there and the people at the gates had a procedure for using the paper backup rather than using the computer system.

I thought that was a great lesson in software design. I think most programmers don’t think about, “How well does my program work when there’s no power?”

Seibel:How does Google work when there’s no power?

Norvig:Google does not work very well without power. But we have backup power and multiple data centers. And we do think in terms of, “How well does my piece work when the server it’s connecting to is down or when there are other sorts of failures?” Or, “I’m running my program on a thousand machines; what happens when one of them dies?” How does that computation get restarted somewhere else?

Seibel:Knuth has an essay about developing TeX where he talks about flipping over to this pure, destructive QA personality and doing his darnedest to break his own code. Do you think most developers are good at that?

Norvig:No. And I had an example of that in my spelling corrector. I had introduced a bug in the code that measured how well I was doing and simultaneously had made some minor change in the real code. I ran it and I got back a much better score for how well it was doing. And I believed it! If it had been a much worse score I would have never said, “Oh, this minor change to the real function must have made it much worse.” But I was willing to believe this minor change made the score much better rather than being skeptical and saying, “Nah, couldn’t have made that much difference, there must be something else wrong.”

Seibel:How do you avoid over-generalization and building more than you need and consequently wasting resources that way?

Norvig:It’s a battle. There are lots of battles around that. And, I’m probably not the best person to ask because I still like having elegant solutions rather than practical solutions. So I have to sort of fight with myself and say, “In my day job I can’t afford to think that way.” I have to say, “We’re out here to provide the solution that makes the most sense and if there’s a perfect solution out there, probably we can’t afford to do it.” We have to give up on that and say, “We’re just going to do what’s the most important now.” And I have to instill that upon myself and on the people I work with. There’s some saying in German about the perfect being the enemy of the good; I forget exactly where it comes from—every practical engineer has to learn that lesson.

Seibel:Why is it so tempting to solve a problem we don’t really have?

Norvig:You want to be clever and you want closure; you want to complete something and move on to something else. I think people are built to only handle a certain amount of stuff and you want to say, “This is completely done; I can put it out of my mind and then I can go on.” But you have to calculate, well, what’s the return on investment for solving it completely? There’s always this sort of S -shaped curve and by the time you get up to 80 or 90 percent completion, you’re starting to get diminishing returns. There are 100 other things you could be doing that are just at the bottom of the curve where you get much better returns. And at some point you have to say, “Enough is enough, let’s stop and go do something where we get a better return.”

Seibel:And how can programmers learn to better recognize where they are on that curve?

Norvig:I think you set the right environment, where it’s results-oriented. And I think people can train themselves. You want to optimize, but left to yourself you optimize your own sense of comfort and that’s different from what you really should be optimizing—some people would say return on investment for the company, others would say satisfaction of your customers. You have to think how much is it going to benefit the customer if I go from 95 percent to 100 percent on this feature vs. working on these ten other features that are at 0 percent.

At Google, I think it’s easy because we have this “launch early and often” philosophy. And because of the way the company is, for a number of reasons: one, most of our products we don’t charge any money for so it’s easy to say, well, go ahead and ship it; how much could they complain? The other one is we’re not stamping CDs and putting them in a box so if there’s something that’s not complete today or even if it has a bug, it’s not a disaster. Most of the software is on our servers so we can fix it tomorrow and everybody gets the update instantly. We don’t have this nightmare of installing updates. So it makes it easier for us to say, “We’re just going to launch things and get some feedback from the users and fix the stuff that needs to be fixed and don’t worry about the other stuff.”