Vikram Chandra - Geek Sublime - The Beauty of Code, the Code of Beauty

You may have to use a dozen tools and websites to handle the various logistical aspects of software development, and soon the triumph starts to wear a little thin. Add another dozen software libraries and frameworks that you may use internally in your programs — again, each one comes bristling with its own eccentricities, bugs, and books — and weariness sets in. Each tool and preconstructed library solves a problem that you must otherwise solve yourself, but each solution is a separate body of knowledge you must maintain. A user named jdietrich wrote in a discussion on Hacker News:

My biggest gripe with modern programming is the sheer volume of arbitrary stuff I need to know. My current project has so far required me to know about Python, Django, Google App Engine and its datastore, XHTML, CSS, JQuery, Javascript, JSON, and a clutch of XML schema, APIs and the like …

Back in ye olden days, most programming tasks I performed felt quite natural and painless, just a quiet little chat between me and the compiler. Sometimes longwinded, sometimes repetitive, but I just sat and thought and typed and software happened. The work I do these days feels more like being a dogsbody at the tower of babel. I just don’t seem to feel fluent in anything much any more. 23

And every year, the new technologies arrive in a cloud of acronyms and cute names: MongoDB, HTML5, PaaS, CoffeeScript, TPL, Rx. One must keep up. On programmers.stackexchange.com, one hapless coder wrote:

I was humbled at a job interview yesterday almost to the point of a beat-down and realized that although I know what I know, my skills are pretty old and I’m getting to where I don’t know what I don’t know, which for a tech guy is a bad thing.

I don’t know if I can keep current just doing my day to day job, so I need to make sure I at least know what’s out there.

Are there well known blogs I should be keeping up with for software development? 24

The best — or at least the most ambitious — programmers read blogs and books, attend conferences to keep up with the state of the art, learn a new language every year or two. When you begin programming, one of the attractions is the certainty that you will never run out of things to learn. But after a few years of working in a corporate cubicle under exploitive managers, after one deadline too many, after family and age and a tiring body, learning the ins and outs of the latest library can feel like another desperate sprint on a nonstop treadmill. There is a reigning cult of overwork in the industry — the legend of the rock-star programmer usually has him coding sixteen hours a day, while simultaneously contributing to open-source projects, blogging, conferencing, and somehow managing to run a start-up — and this ideal has led many an aspirant to burnout, complete with techie thousand-yard-stare, clinical depression, outbursts of anger, and total disinterest in programming. This trough of disillusionment is so deep that for many, the only way to emerge from it is to leave the industry altogether, which rewards a few with fame and dazzling amounts of money, but treats the many as disposable cogs in its software production machine. The endless cycle of idea and action, endless invention, endless experiment, all this knowledge of motion takes its toll, leaves behind a trail of casualties.

Butler Lampson’s hope that millions of ordinary people would write “non-trivial programs” and thus become poets of logic has proved elusive. From the sixties onwards, numerous technologists have promised that their new programming languages would make programmers redundant, that “managers [could] now do their own programming; engineers [could] now do their own programming.” 25Advertisements touted the magical abilities of “automatic programming systems.” But Knuth’s “Software is hard” dictum still remains true, and business users have found that getting custom software out of IT departments requires large budgets and lots of patience. This is because programmers — at their best — try to build software out of elegant code that is modular, secure, and legible, which takes time and money. Instead of waiting, mere mortals often hack something together in the programs they already have available on their machines. This is why, according to the economics blogger James Kwak, “Microsoft Excel is one of the greatest, most powerful, most important software applications of all time.” 26Much of the planet’s business data is stored in Excel, and its intuitive interface allows non-programmers access to some very powerful capabilities. Executives and marketers and secretaries write formulae and macros to extract information when they need it, and are therefore able to take action in a timely fashion. The trouble is that in Excel there

is no way to trace where your data come from, there’s no audit trail (so you can overtype numbers and not know it), and there’s no easy way to test spreadsheets … The biggest problem is that anyone can create Excel spreadsheets — badly. Because it’s so easy to use, the creation of even important spreadsheets is not restricted to people who understand programming and do it in a methodical, well-documented way. 27

Sloppy Excel-wrangling can lead to some very bad decisions, as in the “London Whale” trading disaster of 2012, which caused the financial services firm JPMorgan Chase a loss of approximately six billion dollars; the company’s internal investigation listed as one of the contributing factors a financial modeling process which required cutting and pasting data through a series of spreadsheets. One of these spreadsheets contained a formula dividing by the sum of some numbers instead of their average. 28

The day that millions will dash off beautiful programs — as easily as with a pencil — still remains distant. The “lovely gems and brilliant coups” of coding remain hidden and largely incomprehensible to outsiders. But the beauty that programmers pursue leads to their own happiness, and — not incidentally — to the robustness of the systems they create, so the aesthetics of code impact your life more than you know.

For example, one of the problems that have always plagued programmers is the “maintenance of state.” Suppose you have a hospital that sends out invoices for services provided, accepts payments, and also sends out reminders for overdue payments. On Tuesday evening, Ted creates an invoice for a patient, but then leaves the office for an early dinner; there is now an “Invoice” object in the system. This object has its “InvoiceNumber” field set to 56847, and its “Status” field set to “Created.” All of these current settings together constitute this invoice’s “state.” The next morning, Ted comes in and adds a couple of line items to this invoice. Those inserted line items and a new “Status” setting of “Edited” along with all the other data fields are now the invoice’s state. After a coffee break, Ted deletes the second line item and adds two more. He has changed the invoice’s state again. Notice that we’ve already lost some information — from now on, we can’t ever work out that Ted once inserted and deleted a line item. If you wanted to track historical changes to the invoice, you would have to build a whole complex system to store various versions.

Things get even more complicated in our brave new world of networked systems. Ted and his colleagues can’t keep up with the work, so an offshored staff is hired to help, and the invoice records are now stored on a central server in Idaho. On Thursday afternoon, Ted begins to add more line items to invoice 56847, but then is called away by a supervisor. Now Ramesh in Hyderabad signs on and begins to work on the same invoice. How should the program deal with this? Should it allow Ramesh to make changes to invoice 56847? But maybe he’ll put in duplicate line items that Ted has already begun working on. He may overwrite information — change the “Status” field to “Sent”—and thereby introduce inconsistencies into the system. You could lock the entire invoice record for 56847 on a first come, first served basis, and tell Ramesh he can’t access this invoice because someone else is editing it. But what if Ted decides to go to lunch, leaving 56847 open on his terminal? Do you maintain the lock for two hours?