Harding, Luke - WikiLeaks - Inside Julian Assange's War on Secrecy

The discreet office, well away from the daily news operation, had become a multinational war room, with reporters flown in from Islamabad, New York, and eventually Berlin to analyse hundreds of thousands of leaked military field reports. They jostled with London-based computer experts and website specialists. A shredder was installed alongside the bank of six computer screens, and the air of security was intensified by the stern notice stuck on the door: “Project Room. Private & Confidential. No Unauthorised Access.”

Nick Davies was so fixated by secrecy that he initially even refused to tell the Guardian ’s head of news, deputy editor Ian Katz, about the project. He was dismayed to discover how quickly word spread that he was involved in a top-secret story. Another colleague, Richard Norton-Taylor, the Guardian ’s veteran security editor, soon asked Davies about his “scoop”. Davies refused to tell him. A couple of hours later Norton-Taylor encountered Davies again, and teased him gleefully: “I know all your secrets!” A newspaper office is a bad place in which to try and keep the lid on things for very long.

The paper’s staff did do their best, however. Declan Walsh, the Guardian ’s Pakistan-based correspondent, was recalled in conditions of great secrecy. Meeting round a table in the editor’s office, the Guardian ’s team chewed over the technical difficulties. David Leigh was cantankerous: “It’s like panning for tiny grains of gold in a mountain of data,” he complained. “How are we ever going to find if there are any stories in it?” The answer to that question set the Guardian ’s old hands on a steep learning curve as they got to grips with modern methods.

First they discovered, embarrassingly, that their first download, the Afghan spreadsheet, did not contain 60,000 entries, as they had spent several days believing. It contained far more. But the paper’s early version of Excel software had simply stopped reading after recording 60,000 rows. The real total of hour-by-hour field reports – the war logs – amounted to 92,201 rows of data. The next problem was greater still. It transpired that a spreadsheet of such enormous size was impossibly slow to manipulate, although it could theoretically be sorted and filtered to yield reams of statistics and different types of military event. The Iraq war logs release dumped another 391,000 records into their laps, which quadrupled the data problems.

Harold Frayman, the technical expert, solved those problems: he improvised at speed a full-scale database. Like Google, or sophisticated news search engines such as LexisNexis, the Frayman database could be searched by date, by key word, or by any phrase put between quotation marks. Declan Walsh recalls: “When I first got access to the database, it felt like being a kid in a candy shop. My first impulse was to search for ‘Osama bin Laden’, the man who had started the war. Several of us furiously inputted the name to see what it would produce (not much, as it turned out).” Leigh, too, began to cheer up: “Now this data is beginning to speak to me!” he said.

Leigh was introduced to another Guardian specialist, Alastair Dant: “Alastair’s our data visualiser,” he was told. Leigh: “I didn’t know such a job existed.” He was soon brought up to speed. The WikiLeaks project was producing new types of data. Now they needed to be mined with new kinds of journalism. Dant explained that he could convert the statistics of the thousands of bomb explosions recorded in the Afghan war logs into a bespoke moving graphic display. He could use the same basic template with which the Guardian had formerly developed a popular interactive map of the Glastonbury festival. That had been a nice bit of fun for music fans. The viewer had been able to move a pointer over a map of the festival field, and up came the artists playing at that spot, at that particular time.

Now, with Afghanistan, the viewer would be able similarly to press a button, but this time a much more chilling display would start to run. It would reveal, day by day and year by year, the failure of the US army to contain the insurgents in Afghanistan, as literally thousands of “improvised explosive devices” blossomed all around the country’s road system. The viewer could see how the vast majority of the roadside bombs were slaughtering ordinary civilians rather than military opponents, and how the assaults ebbed and flowed with changes in political developments. It was a rendering that made at least something comprehensible, in an otherwise scrappy and ill-reported war.

The key online expert proved to be Simon Rogers, the Guardian ’s data editor. “You’re good with spreadsheets, aren’t you?” he was asked. “This is one hell of a spreadsheet,” he said. After working on those spreadsheets, he concluded: “Sometimes people talk about the internet killing journalism. The WikiLeaks story was a combination of the two: traditional journalistic skills and the power of the technology, harnessed to tell an amazing story. In future, data journalism may not seem amazing and new; for now it is. The world has changed and it is data that has changed it.”

One obvious opportunity was to obtain genuine statistics of casualties for the first time. The US military had asserted, disingenuously, that at least as far as civilians and “enemies” were concerned, there were no figures available. In fact, the journalists could now see that the war logs contained highly detailed categories that were supposed to be filled in for every military event, breaking them down into US and allies, local Iraqi and Afghan forces, civilians and enemy combatants, and classing them in each case as either killed or wounded. But it wasn’t so simple. Rogers and his reporter colleagues had to grapple with the realities on the military ground: those realities made apparently enticing data sets into dirty and unreliable statistics.

At its simplest, a person listed as “wounded” at the time might have actually died later. More sweepingly, the casualty boxes were sometimes not filled in at all. The reporters felt sympathy with exhausted soldiers, after a day of fighting, being confronted with forms to input that required the filling in of no fewer than 30 fields of bureaucratic information. Some units were more meticulous than others. Early years of the wars saw sketchier information gathering than later, when systems were better organised. When there was heavy urban fighting, or when bodies were carried away, casualties were hard to count. Some units had a penchant for writing down improbably large numbers of purported “enemy killed in action”. Sometimes, more sinisterly, civilians who were killed were recorded as “enemy”. That avoided awkward questions for the troops. All the figures were in any event too low, because some months and years were missing. So were details from the special forces, who operated outside the normal army chains of command. And many of the clashes involving British, German and other “allies” were apparently not recorded on the US army database.

So it was a tricky task to produce statistics that could be claimed to have real value. That highlighted once again the inescapable limitations of the purist WikiLeaks ideology. The material that resided in leaked documents, no matter how voluminous, was not “the truth”. It was often just a signpost pointing to some of the truth, requiring careful interpretation.

Assange himself eventually flew into London from Stockholm late one night in July 2010. He arrived in the Guardian office with nothing but his backpack and a shy smile, like one of the Lost Boys out of Peter Pan . “Have you anywhere to stay?” asked Leigh. “No,” he said. “Have you had anything to eat?” Again the answer was no. Leigh walked him down the road to the brasserie which was still open at St Pancras station and presented him with the menu. Assange ate 12 oysters and a piece of cheese, and then went to stay the night at Leigh’s flat in nearby Bloomsbury.