Luke Harding - The Snowden Files

The landing points of these submarine cables are so important that the American Department of Homeland Security lists them as critical American national infrastructure (according to leaked US diplomatic messages). In this new world of internet-driven communications, Britain’s position on the eastern edge of the Atlantic makes it a hub. As much as 25 per cent of the world’s current internet traffic crosses British territory via the cables, en route between the US, Europe, Africa and all points east. Much of the remaining traffic has landing or departure points in the US. So between them Britain and the US play host to most of the planet’s burgeoning data flows.

Unsurprisingly, given their history, both countries’ spy agencies wanted to exploit their good luck and tap into all these submarine cables in order to eavesdrop. As technology changed, the two organisations had successively intercepted radio traffic, then microwave beams and ultimately satellite links. It was logical to seek now to break into the floods of internet and phone data which were travelling by the latest fibre-optic systems.

Postwar Britain originally won its place in the so-called ‘Five Eyes’ electronic spying team, along with Australia, Canada and New Zealand, by handing over access to a network of listening stations across the globe in Cyprus, Ceylon, Hong Kong, South Africa, Diego Garcia, Ascension Island and such Middle East client states as Oman. But with the loss of empire, some of that advantage evaporated.

Britain also gave the US two satellite stations of its own on British soil – Menwith Hill (known as ‘MHS’), on the southern edge of the Yorkshire Dales, and Croughton, which handles CIA communications. But the Brits constantly had their hands held out for cash. As one GCHQ chief, Sir David Omand, was heard to say optimistically: ‘We have the brains: they have the money.’

Thanks to Snowden we know to what extent, at least partially. In the period 2009 to 2012 the US government paid GCHQ at least £100m. In 2009 the NSA gave GCHQ £22.9m. The following year the NSA’s payments rose to £39.9m. This included £4m to support GCHQ’s work for NATO forces in Afghanistan, and £17.2m for ‘mastering the internet’. The NSA paid a further £15.5m towards redevelopments at GCHQ Bude. The gesture ‘protected (GCHQ’s core) budget’, at a time of austerity by David Cameron’s coalition. In 2011/2012 the NSA gave another £34.7m to GCHQ.

British officials sniff that the sums are tiny. ‘In a 60-year alliance it is entirely unsurprising that there are joint projects in which resources and expertise are pooled,’ a Cabinet Office spokesman says. But the cash gives the NSA further leverage. In one 2010 document, GCHQ acknowledges that Fort Meade had ‘raised a number of issues with regards to meeting NSA’s minimum expectations’. It said GCHQ ‘still remains short of the full NSA ask’.

Lurking always is the spectre of US displeasure. One internal paper warns: ‘The NSA ask is not static and retaining “equability” will remain a challenge in the near future.’ The UK’s biggest fear, says another, is that ‘US perceptions of the… partnership diminish, leading to loss of access and/or reduction in investment… to the UK.’

In other words, the British needed to keep up and demonstrate their worth. They were only a tenth of the size of their US partners. If they fell behind technically, the mighty NSA might cease intelligence-sharing, and Britain’s ability to punch above its weight in the world could end ignominiously.

It was against this background that the GCHQ director in charge of ‘mastering the internet’ wrote a pitch for a new British project on 19 May 2009. He asserted that the agency had been struggling with changes in technology: ‘It’s becoming increasingly difficult for GCHQ to acquire the rich source of traffic needed to enable our support to partners within HMG [Her Majesty’s government], the armed forces and overseas.’

But a breakthrough was in sight, he said. Experiments had been taking place for two years at Bude, and had been crowned with success.

The problem was not so much to tap into the internet pipes – both the US and the UK could do that. It was to find a method of reading and analysing the torrents of data within the tapped cables, as they rushed past at speeds of at least 10 gigabytes per second.

GCHQ’s achievement was to be able to build a gigantic computerised internet buffer. The buffer could store traffic. Analysts and data miners would then be able retrospectively to sort through this vast pool of digital material. Full content, such as email messages, could be kept available for three days, and the less bulky metadata, such as email contacts and subject lines, for as much as 30 days. Uninteresting material such as peer-to-peer downloads of movies would be filtered out.

From the residue the spy agencies would, with luck, glean usable intelligence about targets of interest. The system was analogous to a gargantuan catch-up TV service where you could go back and watch any broadcast you’d previously missed.

At Bude, several key transatlantic fibre-optic cables made landfall close by. They could therefore be tapped into relatively cheaply and the data diverted the short distance to RPC-1 – a new ‘Regional Processing Centre’ secretly constructed on-site by a consortium of private firms, led by Lockheed Martin with BAE Systems’ subsidiary Detica and software company Logica. The process of furtive extraction had its own acronym: SSE, for special source exploitation.

By March 2010, analysts from the NSA had been allowed some preliminary access to the Bude project, initially codenamed TINT, then christened TEMPORA. It was described as a ‘joint GCHQ/NSA research initiative’. It uniquely ‘allows retrospective analysis’ of internet traffic.

Soon GCHQ was boasting of major achievements. ‘We are starting to “master the internet”. And our current capability is quite impressive.’ One document spoke of 2 billion users of the internet worldwide, with over 400 million regular users of Facebook, and a 600 per cent increase in mobile phone traffic from the year before. The agency believed it was on top of these developments. The report claimed the UK now had the ‘biggest internet access in Five Eyes’.

Not everything was perfect. The memo noted that American service providers were moving to Malaysia and India, with the NSA ‘buying up real estate in these places’ in a scramble to keep up. ‘We won’t see this traffic crossing the UK. Oh dear,’ the author said, suggesting Britain should follow suit and ‘buy facilities overseas’.

But the general tone of GCHQ’s 2010–2011 mid-year review was cheery. It stated that in one 24-hour period the agency had been able to process and store ‘more than 39 billion events’, ‘increasing our capability to produce unique intelligence from our targets’ use of the internet’. Apparently this meant GCHQ had managed to collect 39 billion pieces of information in a single day.

The NSA was impressed with British efforts. In a 2011 ‘Joint Collaboration Activity’ report it said that the UK now ‘produced larger amounts of metadata than the NSA’. By May 2012 it was reported that a second internet buffering centre had been constructed at Cheltenham, within the vast circular state-of-the-art headquarters complex its 6,000 staff generally referred to as ‘the doughnut’. A third overseas processing centre was also successfully organised and built at a location in the Middle East. The whole program was capable of collecting ‘a lot of data!’ Using TEMPORA, more than ‘300 GCHQ and 250 NSA analysts’ now had access to ‘huge amounts of data to support the target discovery mission’.

Snowden’s files show just how closely British and US intelligence personnel work alongside each other. While working for the CIA in Geneva, Snowden himself visited Croughton, the CIA communications base 30 miles north of Oxford in rustic Northamptonshire. Writing as TheTrueHOOHA, Snowden said he was struck by the large number of sheep grazing nearby in green fields – a classic English scene.