Peter Seibel - Coders at Work - Reflections on the craft of programming

Seibel:If you’re working on the design of a big system what are the tools you use—do you sit down with a pad of graph paper or a UML drawing tool?

Norvig:I never liked any of these UML-type tools. I always thought, “If you can’t do it in the language itself that’s a weakness of the language.” I think a lot of what you’re doing, you’re dealing at a higher level. At Google a lot of what we do is figuring out how to break things up and parallelize them. We’re going to necessarily run this on multiple machines but we’ve got so many users and, for many applications, so much data; how’s that going to work? So we’re thinking more at the level of machines and racks of machines rather than at the level of functions and interactions. Once you get that straightened out, then you can start diving into individual functions and methods.

Seibel:And so that level of description is just prose?

Norvig:Yeah, mostly. Sometimes people draw pictures. They’ll say, “We’ll have this server here that will be serving these kinds of requests and then it’s connected to this server and then we’ll use these various tools for storage and big distributed hash tables and other types of things. We’ll choose these three tools off the shelf and then we’ll argue about whether we have to build a new one; which of these existing ones works or do we need something else?”

Seibel:And how do you evaluate that kind of design?

Norvig:You show it to the people who’ve done it before. They say, “Oh, it looks like you’ll need a cache here—it’s going to be too slow but there should be a lot of repeat requests, so if you install a cache of this size here that should help a lot.” You have a design review where people look over it and say whether they think it makes sense and then you start building it and testing it.

Seibel:And you guys have formal design reviews? You worked at NASA where they had a very formal design review.

Norvig:Nothing formal like NASA. The stakes are lower for us because, as I say, it’s easy for us to have a failure and recover from that. At NASA usually the first failure is fatal so they were much more careful. We don’t worry about that much. It’s more of a consultation, I think, rather than a review.

There are people who officially read design documents and comment on them. You go through that and get your design approved. But it’s still much

Norvig:I think it was something that had always been done and so people accept it. Well, I shouldn’t say that completely. Some people it takes a while to get used to it. One of the typical failure cases is a new hire comes in and they’re not used to doing this kind of thing so they just start an experimental branch and they have all their code in there and you keep on telling them, “Gee, you don’t have any check-ins yet.” And they say, “Yeah, yeah, yeah, I’m just cleaning it up—I’ll check it in tomorrow.” And then another week goes by and another week goes by and eventually they have this one gigantic check-in. And then it’s a problem that too much time has gone by, it’s hard to evaluate it all at once, and some of the things they’re comparing against have changed out from underneath them. Then they see what a headache it is and they learn not to do that.

Seibel:So that’s on the coding side. Are there skills that the reviewers develop?

Norvig:There certainly are people who are known for being better reviewers than others. There’s a trade-off of when you submit a review—do you try to get somebody who will give you a lot of good feedback or do you try to get somebody who will just say “OK” as quickly as possible?

Seibel:So what makes the better reviewers better?

Norvig:Well, that they catch more things. Some of it is the trivial stuff of you indented the wrong number of spaces or whatever but some of it is, “I think this design would be cleaner if you moved this from here over to there.” So some people will do more of that and others won’t bother.

Seibel:Sort of related to that, does every good programmer turn into a good architect when they grow up? Or are there some people who are brilliant coders but only at a certain level and they should never be allowed to do bigger designs?

Norvig:I think different people have different skills. One of our best search people is by no means our best programmer, in terms of how the code looks. But if you say, “Here’s this new factor that we have—you know, how many times people click on this page after they’ve done such and such—how do we fold that into our search results?” He’ll say, “Oh, on line 427 there’s this variable alpha and you should take this new factor and raise it to the second power and multiply it by 1.5 and add it to alpha.” Then you experiment for a couple months trying different things and you find out he was right except it should have been 1.3 instead of 1.5.

Seibel:So that suggests he just has this very good mental model of how the software works.

Norvig:He understands the code perfectly. Other people can write code better but he understands all the implications of what goes where.

Seibel:Do you think those are related? It often seems that people who write the worst spaghetti code are the ones who can hold the most in their head—that’s the only way they could possibly write code like that.

Norvig:Yeah, I think that may be.

Seibel:So the reviews here are less formal here than at NASA. What are the other differences between the “engineering” and “hacker” ethos, in the best sense of both those words?

Norvig:One big difference is organizational structure and how software is accepted. Google was founded as a software company, and they went out and hired a CEO who has a PhD in computer science from Berkeley, hired a VP of Sales who has a computer engineering background—it’s throughout the whole company. At NASA they’re rocket scientists! They aren’t software guys. They say, “Software is this necessary evil. Straight line code, I can sort of understand; if it’s got a loop in it, that’s kind of iffy. Then if there’s a branch statement inside the loop, ooooh, that’s getting away from what I can solve with a differential equation in control theory.” So they’re distrustful.

Seibel:As well they should be!

Norvig:As well they should be, yeah. And they’re distrustful of innovation. So you can say, “Look at this great new prototype I have,” and they’ll say, “That’s fantastic; I’d love to fly that on my mission—as soon as it’s been proven on two other missions.” And you go to everybody else and they all say the same thing.

Don Goldin came in as NASA administrator and he said, “We’ve got to do this better, faster, cheaper. These space missions cost too much. It’d be better to run more missions and some of them would fail but overall we’d still get more done for the same amount of money.” And that was undeniably true. Unfortunately it was not politically true. It’s not OK to lose a spacecraft. Because the public understands very well, NASA lost a spacecraft. They don’t really know that there’s any difference between a $100 million spacecraft and a billion dollar spacecraft. It’s not like you get to lose ten of the $100 millions instead of one billion. So it was never quite true.

Seibel:What is the worst bug you’ve ever had to track down?

Norvig:Well, I guess the most consequential bugs I was involved with were not mine, but the ones I had to clean up after: the Mars program failures in ’98. One was foot-pounds vs. newtons. And the other was, we think, though we’re not 100 percent sure, prematurely shutting off the engines due to a software problem.