Peter Seibel - Coders at Work - Reflections on the craft of programming

Fitzpatrick:I don’t think code should be owned. I don’t think anyone really thinks that. The way it works within Google is that it’s one massive source tree, one root, and one unified build system across all of it. And so anyone can go and change anything. But there are code reviews, and directories have owners, always at least two people, just in case someone quits or is on vacation.

To check in you need three conditions met: You need someone to review it and say it looks good. You need to be certified in the language—basically you’ve proven you know the style of this language—called “readability.” And then you also need the approval above from somebody in the owner’s file in that directory. So in the case that you already are an owner of that directory and you have readability in that language, you just need someone to say, “Yeah, it looks good.” And it’s a pretty good system, because there tends to be a minimum of two, up to twenty, thirty owners. Once you work on a code base for a while, someone just adds you to owners. I think it’s a great system.

Seibel:So let’s go back in time a bit—how did LiveJournal start?

Fitzpatrick:It was just fucking around with my friends—what I wanted and what we thought would be funny. Commenting on LiveJournal was a practical joke. I was checking my LiveJournal right before I ran into class. We had just introduced friend pages and I saw something my friend wrote and it was really stupid and I wanted to make fun of him. “Oh, but I can’t reply.” So I went to class and all throughout class I was thinking, “How can I add a reply system?” I was thinking of the existing schema and how we could render it. I had a two-hour break between classes, so I add commenting and I reply something smartass and sarcastic and go to my other class. When I came back from my second class, he’s like, “What the fuck? We can comment now?”

Everything on LiveJournal was pretty much a joke. The whole security thing, like friends-only posts and private posts, was because a friend wrote that he went to a party and woke up drunk in a ditch the next day. His parents read it and were like, “What? You’re drinking?” He was like, “Brad, we need a way to lock this shit down!” I was like, “On it!” We already had friends, so we just made it so some posts are friends only and then your parents—just don’t be friends with them.

Seibel:In the early days of LiveJournal it seems your life was an endless series of late nights, sleeping late, and overall working long hours. How much of that is a necessary part of programming?

Fitzpatrick:I just thought it was the least stressful time. During the day, there’s always something coming up, like another meal is coming up, or a class, or maybe you get a phone call. There’s always some interruption. I can’t relax. If I go into work two hours before some meeting, that two hours is less productive than if I didn’t have that meeting that day or if the meeting was the first thing in the morning. Knowing that I have nothing coming up to bug me, I’m so much more relaxed.

At night I feel like this is my time and I’m stealing this time because everyone else is sleeping. There’s no noise and no interruptions, and I can do whatever. I still stay up late sometimes. I did it this weekend; I was up quite a bit working on different things. But that screws me up for days sleepwise. I did that mostly when I had to in college, because I had some project, and I was also doing LiveJournal on the side. The only time to do it was at night and also all our server maintenance had to be at night. And then in the summer, just because why not? There’s no reason to wake up early in the morning to go to a class or anything, so might as well work at night.

Seibel:What about the length and intensity? I’m sure you’ve done the 80-, 100-, 120-hour weeks. Is that necessary? Under what circumstances is that really necessary and when is it just a macho thing that we do?

Fitzpatrick:In my case, I’m not sure it was either necessary or a macho thing. I was having fun and it was what I wanted to be doing. Sometimes things were breaking, but even when they weren’t breaking, I was still doing it just because I was working on a new feature that I really wanted to see happen.

Seibel:Have you ever been in a situation where you really had to estimate how long things were going to take?

Fitzpatrick:Once I got to Six Apart. I guess that was my first experience, three and a half years ago. We had started doing migration—we’d have a customer and they’d say, “Can you move this data?” That requires adding this support for this code and testing, and pushing it out. I was terrible at it. I probably still am terrible at it, because I always forget a factor, like the bullshit multiplier of having to deal with interruptions and the fact that I’m never going to get away from maintaining a dozen projects on the side.

I think I’m getting better, but fortunately they don’t ask for that too often. And now when I actually do get a deadline for something, I’m like, “Yay! A deadline!” and I get so excited that the adrenaline kicks in, and I work, and I finish the damn thing. Nothing with Google is really a deadline. With Google it’s like, “What do you think about launching this? How does that feel?” It’s rare that there’s some real deadline. Most of them, we think it’d be nice to launch on this date and so everyone tries really hard. But you’re only letting down other people that want to see it launch by that day if you don’t finish something. And most of the things I work on are very “When it’s done, it’s done.”

Seibel:When you were hiring programmers at LiveJournal, did you manage them?

Fitzpatrick:Well, I kind of assumed that none of them would need managing; that they would just all be self-driven like me. That was a learning experience in HR, that some people just do what they’re told and don’t really have a passion for excellence. They’re just like, “Done. Next assignment,” Or they don’t tell you and just browse the Web. So I had a couple of painful experiences. But I think after a year or two of that, I learned that people are different.

Some are purists. They would just do abstraction on abstraction on abstractions. They would go really slowly and are very religious about their style. They’re like, “I’m an artisan programmer.” And I was like, “Your code doesn’t run. It’s not efficient and it doesn’t look like any of the other code that you’re interacting with.”

Seibel:Did you figure out how to make good use of people like that?

Fitzpatrick:One person, I tried dozens of different things. I think he might’ve been ten years older than me. I don’t know how much, because I never ask that—I was afraid of legal hiring questions. But I got the feeling that he didn’t want to work for some young punk. I was like 22. That one eventually didn’t work out. That was the only person I let go.

Other people I eventually figured out what motivated them. One guy was really good at tinkering and getting a prototype working. He wrote sysadmin Perl. He could wire stuff together, write shell scripts, and write really bad Perl and really bad C, but kind of get it working. Then we would be like, “Holy crap, you researched all this stuff, you got all these components talking to each other?”

We were setting up a voice bridge to LiveJournal so you record something and post it to LiveJournal. There were just so many moving parts involved. I thought it was painful as hell. He loved it. He figured it all out and got it working. Then we just rewrote it all. And we figured out that was the way it worked with him. He figured out the interface and we would fix it all up. Once I figured out that was his role, we got along great.