Peter Seibel - Coders at Work - Reflections on the craft of programming

Seibel:When did you start working with other people?

Fitzpatrick:It was pretty much towards the end of college when I started hiring other people, and especially once I moved back to Portland after college.

Then the early employees were customer support, so they didn’t write any code. Then slowly I started hiring programmers. The first person I hired was a friend of mine from online. His name is Brad Whitaker and we both had websites called BradleyLand or BradleyWorld, so we found each other’s websites. I was a couple of years ahead of him web programming–wise, or maybe a year, and he was asking me, “Hey, how do you that,” whether it was HTML, or frames, or CGI, or Perl stuff. So then I started getting a bunch of contract projects and I would give the ones I didn’t want to him. And then we had a project that was too big for either of us so we told the guy, “It’s going to take two people to do this project.” And he flew us out to Pennsylvania. Pittsburgh? I don’t know the East coast at all; I’m a Westcoast guy. Philadelphia? The cheesesteak place.

Seibel:Philadelphia.

Fitzpatrick:Yeah, and we met for the first time at some cheapo hotel and it felt like I knew him already. He was like, “Hey, what up?” He came in and took a piss in my hotel bathroom without even closing the door as I’m standing right there. I’m like, “Alright. You’re comfortable.” It was like we knew each other for four or five years, even though we had never met. We started working on this stuff together.

He moved up into my spare bedroom and we basically moved all the stuff out of my kitchen, set up a bunch of tables, and worked on computers. We would wake up around 10 or 11 and work until noon, and watch some TV—sit around in our boxers and watch TV, and hack, and stay up until 3 or 4 in the morning just working nonstop. Then another friend of mine moved down for the summer from UW. This was after my freshman year in college and then there were three of us working there. The third friend was living downtown. He would come on the light rail in the morning and skateboard over to my house. He would sit outside on Wi-Fi, just hacking until we woke up, opened the door, and let him in.

Once there were three of us, it was a little crowded in my house, so I was like, “Oh, OK, let’s get an office.” So we got an office and we were like, “Oh, we have all this space! Let’s hire more people.” We slowly got up to 12 over the next couple of years, and LiveJournal got more and more popular, and then more stressful too, because I was dealing with HR. Or my mom was dealing with HR and my mom was fighting with me because she worked for me. I had to make rules for my mom, like, “If you call me, it has to be personal or business; whatever one you start with, that’s how you end it. You can’t switch from work to personal or personal to work.” I just started hanging up on her if she switched. Then she’d call back and I’m like, “Nope, you lost.” So that was really stressful. She was really happy when I sold it, and she could stop working for me and we could stop fighting.

Seibel:Was your company still doing contract work or was it all LiveJournal at that point?

Fitzpatrick:It was pretty much all LiveJournal. We were also trying to start a photo hosting-service, which Flickr beat us to. Ours was probably overdesigned: beautifully abstract and plugged into everything. But each new infrastructure thing we did for LiveJournal, we were like, “How is this going to work on FotoBilder?” so we started building everything abstract. Memcached was abstract because there was no reason to tie it into LiveJournal. Then we built a file system like GFS, and we built a job queue. So we kept building all these infrastructure components for scaling that would work for either of our products, but also because the less intertwined spaghetti dependency-wise, the easier it is to maintain something. Even if it’s a little bit more work, if you can cut some dependencies, it was great, so we started building all that generic infrastructure.

Seibel:I’m curious about the process that you went through of scaling LiveJournal, in terms of where you started and how you learned the lessons you needed to learn along the way.

Fitzpatrick:So it started on one shared Unix box with other customers and pretty much killed that.

Seibel:Running as CGIs?

Fitzpatrick:Yes. Yeah. I think it was probably a literal CGI, fork up the whole world and die. There was a guy assigned to me at this ISP. I was having problems with my server dying all the time. I’m like, “I paid my $10.00 a month. Why isn’t it working?” So he would say, “Oh, do this.” Pretty soon I was learning Unix and learning what was actually going on.

Then I converted to FastGCI. Then I tuned Apache and turned off reverse DNS lookups. All these steps you go through. Finally, I was I/O-bound or CPU-bound. Then I got my own dedicated server, but it was still just one and it was dying and I was out of capacity. I had originally opened it up for my friends and I just left the signup page alive. Then they invited their friends who invited their friends—it was never really supposed to be a public site. It just had an open signup page on accident. So then I put something up on the LiveJournal news page and I said, “Help. We need to buy servers.”

I think that raised maybe six or seven thousand dollars or something to buy these two big Dells and put them in Speakeasy in downtown Seattle. Somebody recommend some servers, Dells, these huge 6U things, like ninety pounds each. The logical split was the database server and the web server. That was the only division I knew because I was running a MySQL process and an Apache process.

That worked well for a while. The web servers spoke directly to the world and had two network cards and had a little crossover cable to the database server. Then the web server got overloaded, but that was still fairly easy. At this point I got 1U servers. Then we had three web servers and one database server. At that point, I started playing with three or four HTTP load balancers—mod_backhand and mod_proxy and Squid and hated them all. That started my hate for HTTP load balancers.

The next thing to fall over was the database, and that’s when I was like, “Oh, shit.” The web servers scale out so nicely. They’re all stateless. You just throw more of them and spread load. So that was a long stressful time. “Well, I can optimize queries for a while,” but that only gives you another week until it’s loaded again. So at some point, I started thinking about what does an individual request need.

That’s when—I thought I was the first person in the world to think of this—I was like, we’ll shard it out—partition it. So I wrote up design doc with pictures saying how our code would work. “We’ll have our master database just for metadata about global things that are low traffic and all the per-blog and per-comment stuff will be partitioned onto a per-user database cluster. These user IDs are on this database partition.” Obvious in retrospect—it’s what everyone does. Then there was a big effort to port the code while the service was still running.

Seibel:Was there a red-flag day where you just flipped everything over?

Fitzpatrick:No. Every user had a flag basically saying what cluster number they were on. If it was zero, they were on the master; if it was nonzero, they were partitioned out. Then there was a “Your Account Is Locked” version number. So it would lock and try to migrate the data and then retry if you’d done some mutation in the meantime—basically, wait ’til we’ve done a migration where you hadn’t done any write on the master, and then pivot and say, “OK, now you’re over there.”