Peter Seibel - Coders at Work - Reflections on the craft of programming

Seibel:When do you think was the last time that you programmed?

Allen:Oh, it was quite a while ago. I kind of stopped when C came out. That was a big blow. We were making so much good progress on optimizations and transformations. We were getting rid of just one nice problem after another. When C came out, at one of the SIGPLAN compiler conferences, there was a debate between Steve Johnson from Bell Labs, who was supporting C, and one of our people, Bill Harrison, who was working on a project that I had at that time supporting automatic optimization.

The nubbin of the debate was Steve’s defense of not having to build optimizers anymore because the programmer would take care of it. That it was really a programmer’s issue. The motivation for the design of C was three problems they couldn’t solve in the high-level languages: One of them was interrupt handling. Another was scheduling resources, taking over the machine and scheduling a process that was in the queue. And a third one was allocating memory. And you couldn’t do that from a high-level language. So that was the excuse for C.

Seibel:Do you think C is a reasonable language if they had restricted its use to operating-system kernels?

Allen:Oh, yeah. That would have been fine. And, in fact, you need to have something like that, something where experts can really fine-tune without big bottlenecks because those are key problems to solve.

By 1960, we had a long list of amazing languages: Lisp, APL, Fortran, COBOL, Algol 60. These are higher-level than C. We have seriously regressed, since C developed. C has destroyed our ability to advance the state of the art in automatic optimization, automatic parallelization, automatic mapping of a high-level language to the machine. This is one of the reasons compilers are… basically not taught much anymore in the colleges and universities.

Seibel:Surely there are still courses on building a compiler?

Allen:Not in lots of schools. It’s shocking. there are still conferences going on, and people doing good algorithms, good work, but the payoff for that is, in my opinion, quite minimal. Because languages like C totally overspecify the solution of problems. Those kinds of languages are what is destroying computer science as a study.

Seibel:But most newer languages these days are higher-level than C. Things like Java and C# and Python and Ruby.

Allen:But they still overspecify. The core thing is that it specifies location of data. If you look at these other languages, they stayed away from specifying the location of data and how to move it, where to put it in the machine. It was ultimately about its value at any point.

Seibel:But very few languages other than C and C++ have raw pointers anymore. Java has garbage collection and the data moves around. Would you say that’s still overspecified?

Allen:Yes. I believe that there’s an opportunity to do what we have done with computation in the optimization world with data. We don’t manage data very well. We don’t have good ways of managing data automatically—establishing locality of data that’s going to be used together.

There are lots of threads of research now which are very exciting. But I think what’s missing is the bigger, bolder concepts. A lot of this is happening within a space that is bounded by what exists already or the current thinking. It’s not going to change overnight by any means—there are millions of lines of code out there. But we do need to start trying to break the boundaries of, “This’ll be done here and that’ll be done there.”

Seibel:Your career has been largely in high-performance computing. Yet by 2019, or whatever, we’re supposed to have 1,000 cores in a notebook computer. Does that mean high-performance computing and everyday computing will merge? Or will high-performance computing always be out there doing things in a sufficiently different way?

Allen:Well, it kind of depends on where one is on the scale. To go to the petaflop, which is our current goal in high-performance computing—I don’t know how that’s going to go. Certainly the game in performance is going to be at the multicore because it’s driven by reducing energy and lots of good things and solving some problems with the basic physics.

And there’s a competitive element that’s just going to drive it. But harnessing those multicores pushes the problem out of the hardware space into the software space. And that is where we’re not prepared to make any progress as far as I can see. To harness these multicores—I think that’s where the new language levels are going to have to break in. We should do an end-to-end look at it. But it’s going to take some very new thinking.

I think these first 50 years—60 years maybe; ENIAC was in ’44, ’43—we’ve built up not only a wonderful, amazing legacy—just astounding—but also some artifacts that we need to get rid of. It’ll take a very long time to replace these and I think it’s a little hard to predict how that will evolve. But it’ll evolve very fast if we can get some new thinking going in the right places. We know how to do computations on a lot of stuff. We don’t know how to deliver the data to the computation elements in the machine.

Seibel:Can you give a simple example of what you mean by bringing the data to the computation in contrast to what we know how to do now?

Allen:To me it means taking over the management of the data. Basically, how we do it now is by reference—it’s moved by hardware, or by the underlying operating systems and support systems. And often the references are at an element level.

Seibel:You mean in the sense that you can have a pointer into the middle of a struct or an array?

Allen:Yeah, into an element of it. And that brings, depending on the protocols of the hardware and of the architecture itself, the value to where it can be used as part of the computation.

But another way to do that would be to organize locations of data in their relative positions as a target of optimization. The other part of it is that very often what is good for one computation is poor for another. One organization, even of simple things like matrices, is bad when you’re actually accessing it in a different way. So it’s a combination of the order of the accessing against the location. It may require some architectural work, and hardware work, but I think that one can do this if we put some of the referencing, addressing capabilities back out in the hardware itself. There are machines where one has the ability, at the point data comes into the memory, to do quite a lot of transformations. Mapping can happen there.

Computation speed is what we measure, mostly, in high-performance computing so we go through all kinds of things to increase that speed. Feeding that computational unit is one of the big issues that we face, but we never made it a first-order problem to solve. We leave it to the hardware.

Seibel:In your Turning Award lecture you said something along the lines of, “We’re at a crossroads here and we might miss it. We might go down the wrong path and then be going down it for quite a while.”

Allen:Yeah.

Seibel:So the right path, in your view, is getting back to that kind of work in automatic parallelization?

Allen:Right, but it has to be done against a higher-level language than we have now.

Seibel:And the wrong path is finding better ways for people to express parallelism explicitly?

Allen:Well, I think we would eventually realize that we’ve created more of a mess than we had. But we do need higher-level languages and there certainly are domain-specific languages, and ways of developing things, which are really quite wonderful.