Luke Harding - The Snowden Files

Asked how he might explain the discrepancy, one Google executive called it a ‘conundrum’. He dismissed the PRISM slides as a piece of flimsy ‘internal marketing’. He added: ‘There is no back-door way of giving data to the NSA. It’s all through the front door. They send us court orders. We are obliged by law to follow them.’

But in October 2013 it emerged there was indeed a back door – just one that the companies involved knew nothing about. The Washington Post revealed that the NSA was secretly tapping data from Yahoo and Google. The method was ingenious: ‘on British territory’, the agency had hacked into the private fibre-optic links that inter-connect Yahoo and Google’s own data centres around the world.

The NSA codename for this tapping operation is MUSCULAR. It appears to be the British who are doing the actual hacking on the US’s behalf. (One MUSCULAR slide says ‘Operational July 2009’, and adds: ‘Large international access located in the United Kingdom.’)

The firms go to great lengths to keep their customers’ data safe. However, they transfer their information between data centres situated in Europe and America, along leased private internet cables protected by company-specific protocols. It was these cables that the NSA had managed to hack, as they transit the UK. Curiosity focused on Level 3, reported to have been hired as a cable operator by Yahoo and Google: Level 3 is named in the top-secret British documents as an ‘intercept partner’ with the codename LITTLE. The Colorado-based corporation’s response is to say it complies with legal requests in the countries where it operates.

An NSA analyst drew a child-like sketch explaining how the program works; it shows two regions marked ‘Public Internet’ and ‘Google Cloud’. There is a smiley face at the interface where the NSA hacks data. The sketch provoked a thousand Twitter parodies. ‘With so many of these slides you get the feeling people inside the NSA are bragging about their programs,’ ProPublica’s Jeff Larson says. ‘They are saying: ‘We can break encryption! We can grab protocols!”

A document from the NSA’s acquisitions directorate reports that thanks to its back-door access the agency can break into hundreds of millions of user accounts. The data is sent back to the NSA’s Fort Meade headquarters and stored. The volumes are remarkable. In a 30-day period in late 2012, 181,280,466 new records were funnelled back to the Puzzle Palace, including metadata.

Google and Yahoo reacted with apoplexy to the tapping disclosures. Google’s chief legal officer David Drummond said he was outraged at the lengths to which the US government had gone to ‘intercept data from our private fibre networks’. Yahoo repeated that it had no knowledge of the NSA’s back-door cyber-theft.

By the autumn of 2013 all the tech companies said they were scrambling to defend their systems from this kind of NSA snooping. They stood some chance of success. For the NSA’s power to suck up the world’s communications is not quite as awesome as Snowden has made it seem. Tapping into global flows of data is one thing: being able actually to read them is quite another. Particularly if they start to be encrypted.

On 23 October 1642, two armies clashed in the English fields north of Oxford. One belonged to King Charles, the other to Parliament. The battle of Edge Hill was the first in the bloody English civil war. The fight was messy. Parliament forces fired their cannons; the royalists led a cavalry charge; inexperienced soldiers on both sides ran away. Some were keener on looting than defeating the enemy. Neither side really won. The war dragged on for another four years.

Two centuries later, on 21 July 1861, another skirmish took place. This time the Union Army was fighting the Confederates, in the first major land encounter of the American civil war. The location was Bull Run, a tributary of the Potomac in Virginia. The Northern forces expected a quick victory. Instead, the Confederate army launched a ferocious counter-attack. Brigadier General Irvin McDowell and his Union soldiers fled in the direction of Washington DC. The battle revealed there would be no easy knockout.

Many years later, American and British spies were mulling over names for two top-secret programs. Their new battles were electronic rather than territorial. It was the growing practice of encryption that was their enemy. The names they chose for their new battles were BULLRUN and EDGEHILL. Did the emphasis on civil wars have a special significance? Certainly, the spies were now about to declare war on their own domestic corporations.

Cryptography was first used in ancient Egypt and Mesopotamia. The aim, then as now, was to protect secrets. During the first and second world wars, military cryptography and cryptanalysis – the ability to decrypt coded information on enemy movements – played a key role. But it was largely the preserve of embattled nation states. Typically, those interested in codes were the British mathematicians working in secret to defeat the Nazis at wartime Bletchley Park, and the Soviets subsequently.

By the 1970s, however, encryption software such as Pretty Good Privacy (or PGP) was available to private individuals, as well as commercial organisations. Encryption thus posed an obvious challenge to western intelligence agencies, anxious to continue reading their adversaries’ messages. The Clinton administration responded by trying to insert a back door into commercial encryption systems. This would let the NSA in. The attempt met with political defeat. A bipartisan group of senators and tech executives argued this would be bad for the Valley. Plus it would violate the fourth amendment.

By 2000, as encryption was increasingly employed by service providers and individuals in everyday online communications, the NSA was spending billions of dollars finding ways to get round it. Its encrypted targets included web searches, internet chats, emails, personal data, phone calls, even banking and medical records. The challenge was to convert ‘ciphertext’ – what encrypted data looks like in its raw form: that is, mathematical nonsense – into ‘cleartext’.

In 2010 a British GCHQ document warned that over time the allies’ capacities could degrade as ‘information flows change’ and ‘widespread encryption becomes more commonplace’.

At first, the eavesdroppers seemed to face defeat, or at least stalemate. One of the leaked documents from 2006 shows that, at that date, the agency had only broken the encryption of one foreign state’s nuclear ministry, a single travel reservation system, and three foreign airlines.

It was not until 2010 that the NSA made dramatic progress, thanks to BULLRUN and EDGEHILL. It used super-computers to crack algorithms, encryption’s basic building blocks. (Algorithms generate the key which can encrypt and decrypt messages. The longer the key, the better the encryption.)

But most importantly, the Snowden files show that the NSA cheated. Despite the political defeat on back doors, the agency simply went ahead and secretly introduced ‘trapdoors’ into commercial encryption software used by millions of people. It collaborated with developers and technology companies to insert deliberate, exploitable flaws into both hardware and software. Sometimes this co-operation was voluntary; sometimes bullying legal orders enforced it. The NSA, if necessary, would steal encryption keys, almost certainly by hacking into servers where the keys were kept.

Unsurprisingly, the NSA and GCHQ were keen to keep details of these most shadowy of programs under wraps. A 2010 document from Snowden shows just how restricted knowledge was of BULLRUN – and how effective it was. The PowerPoint was used to brief British staff in Cheltenham on the NSA’s recent breakthroughs, as a result of which decrypted internet traffic was suddenly streaming across the desks of analysts.