Luke Harding - The Snowden Files

It says: ‘For the past decade the NSA has led an aggressive, multi-pronged effort to break widely used internet encryption technologies. Cryptanalytic capabilities are now coming online. Vast amount of encrypted internet data which up to till now have been discarded are now exploitable.’

The slide says ‘major new processing systems’ must be put in place ‘to capitalise on this opportunity’. GCHQ staff previously kept in the dark about BULLRUN were astonished by the NSA’s formidable new capabilities. One internal British memo reports: ‘Those not already briefed were gobsmacked.’

Snowden’s first batch of published files did not disclose details of which companies work with the NSA on counter-encryption. Or which commercial products may have back doors. But the files do give some idea of BULLRUN’s massive dimensions. A budget report for the entire US intelligence community says that 2013 funding for the program was $254.9m. (PRISM, by contrast, costs just $20m annually.) Since 2009, the agency has splashed more than $800m on ‘SIGINT [signals intelligence] enabling’. The program ‘actively engages US and foreign IT industries to covertly influence and/or overtly leverage their commercial products’ designs’, the report says.

The joy of the program, the NSA says, is that ordinary citizens have no idea that their everyday encrypted communications are now hackable. When the NSA inserts ‘design changes’ into commercial encryption systems, the 178-page report for the fiscal year notes, ‘To the consumer and other adversaries… the systems’ security remains intact.’

James Clapper, the director of national intelligence, stresses the importance of crypto. ‘We are investing in groundbreaking cryptanalytic capabilities to defeat adversarial cryptography and exploit internet traffic,’ he writes.

The agency is not lacking in ambition. The files show the NSA is breaking the encryption systems of 4G phones. It targets online protocols used in secure banking and business transactions, such as HTTPS and Secure Sockets Layer (SSL). It wants to ‘shape’ the worldwide encryption marketplace. Soon it expects to get access to ‘data flowing through a hub for a major communications provider’ and to a ‘major internet peer-to-peer voice and text communications system’. That sounds like Skype.

Meanwhile, the British were pressing on with their own parallel EDGEHILL project. One file shows that the British spies have succeeded in breaking into three internet providers and 30 types of Virtual Private Networks (VPN) used by businesses to access their systems remotely. By 2015 it hoped to have penetrated 15 internet companies and 300 VPNs.

The spy agencies insist that their ability to defeat encryption is essential to their mission, and that without it they would be unable to track terrorists or gather valuable foreign intelligence. The problem, as the New York Times points out, is that the NSA’s anti-encryption stealth campaign may have disastrous unwanted consequences.

By inserting deliberate weaknesses into encryption systems, the agency has made those systems exploitable. Not just by government agencies, who may be acting with good intentions, but by anybody who can get hold of encryption keys – such as hackers or hostile intelligence agencies. Paradoxically, in its quest to make Americans more secure, the NSA has made American communications less secure; it has undermined the safety of the entire internet.

The main US agency for setting security norms in cyberspace is the National Institute of Standards and Technology (NIST). It appears the NSA has corrupted this, too. A Snowden document reveals that in 2006 the NSA put a back door into one of the institute’s main encryption standards. (The standard generates random prime numbers used to encode text.) The agency then encouraged another international standards body – and the rest of the world – to adopt it, boasting: ‘Eventually the NSA became the sole editor.’

Both US and UK agencies have also devoted considerable efforts to cracking Tor, the popular tool to protect online anonymity. Ironically, the US government is one of Tor’s biggest backers. The State Department and the Department of Defense – which houses the NSA – provide around 60 per cent of its funding. The reason is simple: journalists, activists and campaigners in authoritarian countries such as Iran use Tor to protect themselves from political reprisals and online censorship.

Thus far, however, the NSA and GCHQ have been unable to de-anonymise most Tor traffic. Instead, the agencies have attacked web browsers such as Firefox, which allows them control over a target’s end computer. They have also developed the ability to ‘stain’ some traffic as it bounces around the Tor system.

Despite their best endeavours, the truth appears to be that NSA and GCHQ have not yet won cryptography’s new civil war. With the right training and some technical expertise, corporations and individuals (as well, no doubt, as terrorists and paedophiles) are still successfully using cryptography to protect their privacy.

In a Q&A with Guardian readers while in hiding in Hong Kong, Snowden himself said: ‘Encryption works. Properly implemented strong crypto systems are one of the few things that you can rely on.’

And he should know.

‘We always imagine eternity as something beyond our conception, something vast. But why must it be vast? Instead of all that, what if it’s one little room, like a bath house in the country, black and grimy and spiders in every corner, and that’s all eternity is?’

Ed Snowden went underground after hastily checking out of the Mira Hotel in Hong Kong. His local legal team, barrister Robert Tibbo and solicitor Jonathan Man, knew where he was. So did someone else. Snowden had a mystery guardian angel – a well-connected Hong Kong resident. The American’s interest in China was long-standing, dating back to his time with the CIA in Geneva and his support for the Free Tibet movement.

The precise details are murky. But it appears this benefactor invited Snowden to stay with one of his friends. Another lawyer, Albert Ho, says that Snowden shifted between several homes, staying in at least one house in the New Territories area, close to the border with mainland China. He was lost in a densely packed metropolis of seven million people.

Tibbo, a human rights lawyer, was used to dealing with clients in bad situations. A Canadian by nationality, with a pleasant manner, a smart blazer and a receding hairline, Tibbo represented the vulnerable and the downtrodden – Sri Lankans facing deportation, Pakistanis wrongly denied asylum, abused refugees.

One of his cases dated back to the darkest chapter of the Tony Blair era. In 2004, the Libyan Islamist Sami al-Saadi arrived in Hong Kong with his wife and family. He thought he was travelling back to the UK, his old home. Instead, MI6, working closely with Muammar Gaddafi’s intelligence services, bundled him on a plane back to Tripoli. There, Saadi was interrogated, tortured and imprisoned. Shortly afterwards, Blair, the then British prime minister, struck a deal with the Libyan dictator. MI6’s discreditable role in the affair emerged after Gaddafi’s 2011 fall.

Like Saadi, Snowden was another client whom, he feared, western intelligence services would render and then imprison in a dark, damp hole. Tibbo and Snowden first met after he slipped out of the Mira Hotel. The lawyer refuses to talk about the details, citing client confidentiality. But he evidently considered Snowden to be bright, a rational actor who was making his own conscience-driven choices. And a young man in a whole pile of trouble. Over the next two weeks Tibbo would juggle his regular case-load while working on Snowden’s behalf, often through the night.